<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Hadoop Noob - Medium]]></title>
        <description><![CDATA[Elephant trainers - Medium]]></description>
        <link>https://medium.com/hadoop-noob?source=rss----39bc3bbb9264---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Hadoop Noob - Medium</title>
            <link>https://medium.com/hadoop-noob?source=rss----39bc3bbb9264---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:00:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/hadoop-noob" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[fluentd out_forward plugin is stuck longer than expected]]></title>
            <link>https://medium.com/hadoop-noob/fluentd-out-forward-plugin-is-stuck-longer-than-expected-f20b2a2501db?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/f20b2a2501db</guid>
            <category><![CDATA[computer-networking]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <category><![CDATA[fluentd]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Mon, 18 Feb 2019 16:01:01 GMT</pubDate>
            <atom:updated>2019-02-18T16:01:00.941Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I am working log ingestion project which we use fluentd to collect and stream the log to Kinesis for further data processing. We use a very standard fluentd cluster setup described <a href="https://docs.fluentd.org/v1.0/articles/high-availability#network-topology">here</a>. I will not debate whether the setup is optimal or not. Just try to give some context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*xjsqk0xaO5FyRc1lbCaKug.png" /></figure><p>In this setup, fluentd on the nodes which produce logs act as log forwarders. Aggregators will centralize the logs and then push the logs to other places, e.g. Kinesis.</p><p>Our log forwarder still on 0.12 version. In the log forwarder, it uses <a href="https://docs.fluentd.org/v0.12/articles/out_forward">out_forward plugin</a> to forward the data. Configuration looks something like:</p><pre>&lt;match pattern&gt;<br>  @type forward<br>  expire_dns_cache 0<br>  dns_round_robin true<br>  heartbeat_type tcp<br>  heartbeat_interval 1</pre><pre>  &lt;server&gt;<br>    host my-log-aggregator.example.com<br>    port 24224<br>  &lt;/server&gt;<br>&lt;/match&gt;</pre><p>As you can see, we set <a href="https://docs.fluentd.org/v0.12/articles/out_forward#dns_round_robin">dns_round_robin</a> to true because we put several log aggregators behind the dns. We hope each aggregator behind the dns could get a fair change to ingest the data. In other words, we could have a client-side load balancing. If you are familiar with fluentd, you know we can also achieve client-side load balancing through multiple <a href="https://docs.fluentd.org/v0.12/articles/out_forward#%3Cserver%3E-(at-least-one-is-required)">&lt;server&gt; &lt;/server&gt; section</a>. We didn’t do that because we have hundreds of instances, if we ever want to add/remove log aggregators to the aggregator cluster, we need to changes config on all the log forwarders and then restart them. By using the dns_round_robin, we just need to update the dns. That’s why we also set <a href="https://docs.fluentd.org/v0.12/articles/out_forward#expire_dns_cache">expire_dns_cache</a> to 0 so we should get updated dns after TTL.</p><p>For load balancing, this setup works well. We do see traffic on the forwarder splits to all aggregators evenly.</p><p>We also noticed one issue. Let’s say we have 20 aggregators sitting behind the dns. I stop one of them but do not remove it from the dns. So basically 20 ip addresses behind the dns but only 19 have aggregator listening on it. I would expect we will have 19/20 chance to hit a good ip. Data on the log forwarder should be moving although performance is slightly impacted. But in reality, we found right after we stop one aggregator, the log forwarders begin to have trouble sending data. And it doesn’t seem to recover until we restart the stopped aggregator.</p><p>To debug this. I ssh to one log forwarder and use strace to monitor the network activities.</p><pre>strace -p $PID -e trace=network -f</pre><p>The output looks similar down below. You can see all the system calls related to network</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*KUepy6SaxxRVUEBqKRPAVw.png" /></figure><p>If you are familiar with fluentd, you probably know out_forward plugin uses two threads. One for heartbeat; another one actually sends data to the destination. How can you tell which thread’s functionality? I usually just check them one by one. Each thread follows different patterns. For example, the heartbeat thread will run every 1 second to resolve the dns first and then ping one ip behind it. If you change the heartbeat_interval to 5, it should run every 5 seconds. For the thread actually sends the data, if you remove the -e trace=network flag, you can observe the buffer data path in the strace output.</p><p>By examing both threads, strace shows the threads are stuck for near 2 minutes. After 2 minutes, it moves a little bit and then stuck for another 2 minutes. I figured out that the stuck system call is <strong>connect</strong> (just <strong>man connect</strong> on your laptop). After I looked up on the man page, I believe it could accept a timeout parameter. Great, Let’s find out where we can pass the timeout value.</p><p>After I researched a little bit on the <a href="https://github.com/gaohao/fluentd/blob/v0.12.43/lib/fluent/plugin/out_forward.rb">out_forward plugin</a> source code, I think the issue is related to this <a href="https://github.com/gaohao/fluentd/blob/v0.12.43/lib/fluent/plugin/out_forward.rb#L378">line of code</a>.</p><pre>def connect(node)<br>    # TODO unix socket?<br>    <strong>TCPSocket.new(node.resolved_host, node.port)</strong><br>end</pre><p>The above method is called every time out_forward plugin tries to do heartbeat or forward data. Because no matter it is heartbeat or forwarding data, it always starts with initializing a new socket. So when it tries to initialize a TCP connection, <strong>connect system call (not the connect method) </strong>is get called. Then it waits there for 2 minutes.</p><p>If we look at the <a href="https://ruby-doc.org/stdlib-2.1.0/libdoc/socket/rdoc/TCPSocket.html">TCPSocket.new</a>, we cannot see where we can set the timeout. There is also nothing about the default timeout value. If we open irb and type:</p><pre><strong>require &#39;socket&#39;<br>TCPSocket.new(INVALID_IP, 8888)</strong></pre><p>Make sure INVALID_IP a none exist ip address. You may see a different timeout than my 2 minutes. I think it depends on the OS setting. <strong>Although I didn’t figure out where it is configured in the OS. </strong>Let me know in the comment if you know.</p><p>It seems we cannot use TCPSocket api, we need something else to bypass it. Luckily since we use ruby 2.0 above, there is a new api <a href="https://docs.ruby-lang.org/en/2.2.0/Socket.html#method-c-tcp">Socket.tcp</a> which actually can accept a timeout.</p><p>So <a href="https://github.com/gaohao/fluentd/commit/0735a419944e1837a07c742d9b39d2dd7d7fa77e">here is what I got</a>. Very simple. I tested the code change in a scenario where I set the timout to 1s and give 1 bad ip out of 2 ips in total. The out_forward works pretty well with a slight performance decrease</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f20b2a2501db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/fluentd-out-forward-plugin-is-stuck-longer-than-expected-f20b2a2501db">fluentd out_forward plugin is stuck longer than expected</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[json vs msgpack]]></title>
            <link>https://medium.com/hadoop-noob/json-vs-msgpack-2461cf08489?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/2461cf08489</guid>
            <category><![CDATA[json]]></category>
            <category><![CDATA[flink]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Fri, 14 Sep 2018 03:01:01 GMT</pubDate>
            <atom:updated>2018-09-14T03:01:01.367Z</atom:updated>
            <content:encoded><![CDATA[<p>Which is better? It is really hard to say if we don’t give some context or constraints. Because if I could build it from scratch, I may choose neither.</p><p>So I have a cluster of Fluentd aggregators which streams data to treasure data. I need to fork the stream to kinesis. Through kinesis, I can use flink to process the data. Fluentd has built-in json and msgpack formatter. I need to make a choice on which format to use. Although there are some protobuf or avro plugins from the community, we still want to keep the data schema-less. We also try to combine with <strong>zlib</strong> since it is also built-in</p><p>After read some blogs on the internet, I decided to benchmark it myself. In my benchmark, I don’t care much about serialization since we have lots of capacity on our aggregator clusters. I care more about the <strong>deserialization speed on consumers (e.g. flink).</strong></p><p>Json data, I will use Gson and dsl-json to decode. dsl-json, we use reflection since schemaless. Msgpack data, I will use unpacker to decode.</p><pre>#################### Deserialization Benchmarks ####################<br>18667 json records read. 18667 records will be processed<br>18667 msgpack records read. 18667 records will be processed<br>########################################<br>dslJson test<br>Deserialization takes 469.865097 ms<br>########################################<br>Gson test<br>Deserialization takes 709.607553 ms<br>########################################<br>msgpack test with unpacker<br>Deserialization takes 264.010102 ms<br>########################################</pre><p>So looks like msgpack is the winner :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2461cf08489" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/json-vs-msgpack-2461cf08489">json vs msgpack</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debug Flink OOM in Docker Container]]></title>
            <link>https://medium.com/hadoop-noob/flink-docker-oom-94883af7d17?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/94883af7d17</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[flink]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Mon, 10 Sep 2018 23:11:35 GMT</pubDate>
            <atom:updated>2018-09-11T20:01:02.928Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I am planning to deploy a new Flink pipeline. I tested on my local and staging environment. When I deploy it to serve the full traffic, the TaskManagers are killed randomly. Since I have enabled externalized checkpoint, I won’t lose any data or state. But every time, it crashes, it certainly causes some delay in processing. And it really annoys me.</p><p>From the stderr, I got <strong>Container exited with status 137</strong><br>From the exit code, we can tell the container is killed by the Linux OOM killer<br>Running dmesg on the host machine, we got</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IuTLrwKNppDXbNLv91yP7g.png" /></figure><p>As I understand, OOM killer kills the process when rss (in RAM) reaches docker’s memory limit (-m flag)</p><p>Just give some context:</p><ol><li>I deploy Flink 1.3.3 under Marathon in cluster mode</li><li>production environment, I give 24GB to each TaskManager and 22GB for taskmanager.heap.mb. So 2GB for everything else</li><li>staging environment, I give 8GB to each TaskManager and 6GB for taskmanager.heap.mb</li><li><strong>In this post, I actually use 24GB to TaskManager and 19GB for taskmanager.heap.mb</strong></li><li>Java version is 8</li><li>A “<em>WARNING: No swap limit support”</em> message :)</li></ol><p>My Flink application is very heavy on memory consumption, so at first, I thought it may be related to GC or heap.<br>I turned on the JMX, I got the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ntl8-k7LI9zfsyKaOJodDQ.png" /></figure><p>One thing to notice, from the code <a href="https://github.com/apache/flink/blob/release-1.3.3/flink-dist/src/main/flink-bin/bin/taskmanager.sh#L39">here</a> <br>If we add any java.opts, the script won’t set your GC to G1GC. The above diagram is actually ParallelGC which is the Java 8 default option</p><p>Anyway I changed it back to G1GC, here is what I got</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5bw3UfVHsaG3iooQIQLDww.png" /></figure><p>All look normal to me</p><p>If it is not the heap, then what else? Since we are on Java 8, I can enable NMT by</p><pre>-XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics</pre><p>After <strong>jcmd $pid VM.native_memory</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*dLbeO1dIB8bhUvBWt8hmXA.png" /></figure><p>If we sum up everything off-heap it is actually more than 2GB on reserved. We may wonder why GC took around 800MB and Class reserved around 1GB<br><strong>GC</strong> accounts for GC native structures. It grows with heap size linearly when the heap size is big (for example, more than 500MB) <br><strong>Class</strong> is for the Metaspace. Before Java 8, classes are stored on PermGem which is on the heap. Now it is off-heap</p><p><strong>**Reserved for Class should not account for the rss, only committed should/could. But from my observation my long-running TaskManagers, rss is actually close to Reserved. So if anyone has an answer, do let me know :) **</strong></p><p>In a conclusion, <strong>in the production setup, I simply cut the heap a little bit and give more to non-heap</strong></p><p>Along with the research, I also came across some interesting findings I would love to share</p><ol><li><strong>taskmanager.memory.preallocate</strong>, If this configuration is set to false (<strong>which is default</strong>) cleaning up of the allocated off-heap memory happens only when the configured JVM parameter MaxDirectMemorySize is reached by triggering a full GC. But if we run Flink cluster mode, MaxDirectMemorySize by default is actually set as 8388607T which means there is no limit. Although if <strong>taskmanager.memory.off-heap</strong> is false, Flink should not use too much off-heap memory.</li><li>Some interesting JIRAs. <a href="https://issues.apache.org/jira/browse/FLINK-9904">https://issues.apache.org/jira/browse/FLINK-9904</a>, <a href="https://issues.apache.org/jira/browse/FLINK-6217">https://issues.apache.org/jira/browse/FLINK-6217</a>, <a href="https://issues.apache.org/jira/browse/FLINK-4094">https://issues.apache.org/jira/browse/FLINK-4094</a></li></ol><p>References:<br><a href="https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html">https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html</a><br><a href="https://shipilev.net/jvm-anatomy-park/12-native-memory-tracking/">https://shipilev.net/jvm-anatomy-park/12-native-memory-tracking/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94883af7d17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/flink-docker-oom-94883af7d17">Debug Flink OOM in Docker Container</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes on Apache Mesos Setup]]></title>
            <link>https://medium.com/hadoop-noob/notes-on-apache-mesos-setup-978e8f923f29?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/978e8f923f29</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[mesos]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Mon, 30 Jul 2018 23:02:01 GMT</pubDate>
            <atom:updated>2018-07-30T23:02:01.229Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I work on building a new data ingestion pipelines. I need to ingest data from kinesis and dump them on S3. Since I am familiar with flink and parquet, I decide to just use them. But before I can write and run some flink jobs, I need a cluster. I really like docker and I have lots of data framework related docker files like flink, spark, presto. So I am going to setup a mesos cluster which I can use it to play with my data frameworks :)</p><p>I don’t think I have much experience on setting up production ready mesos cluster. But I just want to take a note on what I went through. And hope this can help anyone wants to try it. Everything here</p><h3>Enviroment</h3><p>I am running all my services on top AWS.</p><p>instance type, m4.2xlarge</p><p>os: ubuntu 16.04</p><h3>Install Mesos</h3><p># In case the host name is not the ip</p><pre>cat &lt;&lt;EOF &gt;&gt; /etc/hosts<br>127.0.0.1 `hostname`<br>EOF</pre><p>According to <a href="https://mesosphere.com/blog/mesosphere-package-repositories/">Mesosphere’s blog</a>, we are going to install mesos and marathon. <strong>Make sure to install java 8 first</strong>. If you don’t install java 8 first, after you run sudo apt-get -y install mesos marathon chronos, it will pull java 9 automatically. Unfortunately, I have trouble to start mesos/zookeeper with java 9.</p><pre># Setup<br>sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF<br>DISTRO=$(lsb_release -is | tr &#39;[:upper:]&#39; &#39;[:lower:]&#39;)<br>CODENAME=$(lsb_release -cs)</pre><pre># Add the repository<br>echo &quot;deb <a href="http://repos.mesosphere.com/$%7BDISTRO">http://repos.mesosphere.com/${DISTRO</a>} ${CODENAME} main&quot; | <br>  sudo tee /etc/apt/sources.list.d/mesosphere.list<br>sudo apt-get -y update</pre><pre># Install Java 8. Make sure it is java 8, not java 9<br>sudo apt-get -y install openjdk-8-jre</pre><pre># Install packages<br># this is the exact version I installed<br># mesos=1.6.0-2.0.4 <br># marathon=1.6.352 <br># chronos=2.5.1-0.1.20171211074431.ubuntu1604<br># you don&#39;t needto install chronos, but marathon is very useful<br>sudo apt-get -y install mesos marathon chronos</pre><h3>Configure Mesos slave</h3><p>Now we want to configure the mesos slave. Run the following command under root</p><pre>sudo su -</pre><pre>cat &lt;&lt;EOF &gt; /etc/mesos-slave/containerizers<br>docker,mesos<br>EOF</pre><pre>cat &lt;&lt;EOF &gt; /etc/mesos-slave/image_providers<br>docker<br>EOF</pre><pre>cat &lt;&lt;EOF &gt; /etc/mesos-slave/isolation<br>filesystem/linux,docker/runtime<br>EOF</pre><pre>cat &lt;&lt;EOF &gt; /etc/mesos-slave/executor_environment_variables<br>{<br>  &quot;JAVA_HOME&quot;: &quot;/usr/lib/jvm/java-8-openjdk-amd64&quot;<br>}<br>EOF</pre><pre>#remove the default config. <strong>Assume the zookeeper ip is 10.10.10.10</strong><br>cat &lt;&lt;EOF &gt; /etc/mesos/zk<br>zk://<strong>10.10.10.10</strong>:2181/mesos<br>EOF</pre><pre>#remove the default config<br>cat &lt;&lt;EOF &gt; /etc/default/mesos-slave<br>MASTER=`cat /etc/mesos/zk`<br>EOF</pre><pre># By default, the ports offered by mesos are [31000-32000] <br>echo &quot;ports:[5500-60000]&quot; | sudo tee /etc/mesos-slave/resources</pre><pre># Configure the slave&#39;s ip<br>echo 10.10.10.14 | sudo tee /etc/mesos-slave/ip<br>cp /etc/mesos-slave/ip /etc/mesos-slave/hostname</pre><pre>echo manual | tee /etc/init/zookeeper.override<br>echo manual | tee /etc/init/mesos-master.override</pre><p>Here we configured the ports resource. Because I hardcode some ports in my dockers. For example, presto’s coordinator, I configured 9090; flink’s blobserver, I configured 50100. So the default range is too small for me.</p><p>Now before we start the mesos’s slave, we want to install docker. According to the <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">docker’s doc</a></p><pre>sudo apt-get update<br>sudo apt-get -y install \<br>    apt-transport-https \<br>    ca-certificates \<br>    curl \<br>    software-properties-common<br>curl -fsSL <a href="https://download.docker.com/linux/ubuntu/gpg">https://download.docker.com/linux/ubuntu/gpg</a> | sudo apt-key add -<br>sudo apt-key fingerprint 0EBFCD88<br>sudo add-apt-repository \<br>   &quot;deb [arch=amd64] <a href="https://download.docker.com/linux/ubuntu">https://download.docker.com/linux/ubuntu</a> \<br>   $(lsb_release -cs) \<br>   stable&quot;<br>sudo apt-get update<br>sudo apt-get -y install docker-ce</pre><p>One last thing, we need to modify the /etc/resolv.conf. We want our internal dns can be resolved by our mesos-dns. Put the your resolver’s ip at the first line of your resolv.conf. Somthing like this:</p><pre>nameserver 10.10.10.11<br># your old config below<br>nameserver 8.8.8.8</pre><h3>Configure Mesos DNS</h3><p>Nothing fancy here, I just used a <a href="https://hub.docker.com/r/mesosphere/mesos-dns/">mesos dns docker image </a>. I guess so far so good. Make sure security groups are correct, it actually takes me sometime to debug..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=978e8f923f29" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/notes-on-apache-mesos-setup-978e8f923f29">Notes on Apache Mesos Setup</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flink Forward 2018]]></title>
            <link>https://medium.com/hadoop-noob/flink-forward-2018-2d301e3cd161?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/2d301e3cd161</guid>
            <category><![CDATA[mesosphere]]></category>
            <category><![CDATA[mesos]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-flink]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Wed, 18 Apr 2018 02:30:34 GMT</pubDate>
            <atom:updated>2018-04-18T02:30:34.248Z</atom:updated>
            <content:encoded><![CDATA[<p>It is a little late to write something about Flink Forward 2018. But I have to because we, well not me, actually co-present with Mesosphere about <strong>Flink on Mesos</strong>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NiObo8oqgyII4mEzLQbMEA.jpeg" /></figure><p>If you are interested in the presentation, you could try to find the recorded video later (not published yet). I think it would be a useful for whom looking for Flink + Mesos combination.</p><p>I am going to share some quick stats about our Mesos cluster and Flink jobs</p><ol><li>50 streaming jobs in Flink. Streaming jobs read from Kafka and persist data into HDFS/Druid.</li><li>Streaming jobs handle 120k rps/sec. The biggest Kafka topic we have is around 70k rps/sec. We assign 32 cores to the Flink streaming to handle the biggest Kafka topic.</li><li>Streaming jobs process 2TB raw data (protobuf) per day and persist 200GB on HDFS (parquet)</li><li>We run a Mesos cluster with more than 200 nodes in AWS</li><li>We run Flink in Mesos both as a framework and as a Marathon app. Streaming we run one pipeline per cluster as a mesos framework. Flink in Marathon usually is used for ad-hoc batch jobs.</li></ol><p>Although, we already run Flink on Mesos in production, we have to overcome some limitations in Flink cluster management itself (not only Mesos). For example, unlike spark, when you launch your Flink application in Mesos, your driver doesn’t wait until the required resources (slots) is ready, it will just start and fail. If you are interested in more details, you can refer to <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077"><strong><em>FLIP-6</em></strong></a><strong>. </strong>I see lots of people including mesosphere are actively working on it. We will definitely see more in the next Flink Forward!</p><p>Hope this brief note could give you some confidence for Flink on Mesos. If you are more interested in Flink Forward 2018 in general, I found this <a href="https://medium.com/fintech-studios-engineering/flink-forward-2018-1e25a580239d"><strong><em>Article</em></strong></a> useful :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d301e3cd161" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/flink-forward-2018-2d301e3cd161">Flink Forward 2018</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flink Fault tolerance using externalized checkpoint]]></title>
            <link>https://medium.com/hadoop-noob/flink-externalized-checkpoint-eb86e693cfed?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/eb86e693cfed</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-flink]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Mon, 09 Apr 2018 15:01:02 GMT</pubDate>
            <atom:updated>2018-04-10T21:48:44.869Z</atom:updated>
            <content:encoded><![CDATA[<p><em>As I am writing this, Flink already on 1.4 release and 1.5 snapshot is already out. But we are still on flink 1.3.2</em></p><p>I want to talk a little bit about Flink externalized checkpoint. Flink’s checkpoint is a great feature. And externalized checkpoint make Flink pipeline easy to resume from where it stopped. In real world, there are so many reasons our Flink pipelines could fail. A code bug could break our pipeline; Task manager or Job manager could be lost due to network; Flink cluster could gone if you running it on spot instances. So if somehow our pipeline failed, it will try to recover from the last success checkpoint, job manager knows where state backend stores.</p><p>Sounds like it works really well. The tricky part is, what if the job managers are gone, all of them. In this case, if we enabled externalized checkpoint, we can use the externalized checkpoint path as a save point to recover our pipeline. <strong>BUT </strong>if we look at how the externalized checkpoint directory structured, can you tell which checkpoint belongs to which pipeline?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*kMrozlmnyn_bbuO-G6RGBA.png" /></figure><p>Our simple approach: organize the checkpoint directory well, so we can figure out which belongs to which. We assign a unique name to each of our pipelines, each pipeline only generate checkpoints under its own folder. Externalized Checkpoint file also carries a timestamp so we know which checkpoint is the latest. The new directory will look like the following:</p><pre>/checkpoint/pipeline_a/9c87f7685fe0947ab39286d0f3f1f748-1522439833113-4048<br>/checkpoint/pipeline_b/9a678059febd20b5e00cc8fd8c32f789-1522985422370-1480</pre><p>Under <strong>checkpoint</strong> folder, pipeline_a &amp; pipeline_b are unique, under the each pipeline folder, there are checkpoint files. In the file name, there are three parts, first part is the hash which already exists in current system; second part is the timestamp; third part is the checkpoint id.</p><p>In our Flink Mesos setup, we have a Flink launcher in Marathon. Flink launcher will try to figure out what’s the last success externalized checkpoint base on the above folder structure, then it uses the checkpoint as savepoint to resume the Flink job.</p><p>I know this is a small improvement/trick. But in the production, it really saves us a lot of headache. Parts of mesos agents run on spot instances to save some money, the ability to recover itself when the spot instances are gone benefits us lot.</p><p>I thought we are going to prepare something for the upcoming Flink Forward, but it turns out we don’t have time. But maybe we will go to Flink Forward this year as an audience :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb86e693cfed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/flink-externalized-checkpoint-eb86e693cfed">Flink Fault tolerance using externalized checkpoint</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Druid parquet extension on Array/List type]]></title>
            <link>https://medium.com/hadoop-noob/druid-parquet-extension-on-array-list-type-ab9ddfb150e6?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/ab9ddfb150e6</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[druid]]></category>
            <category><![CDATA[parquet]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Fri, 06 Apr 2018 02:48:48 GMT</pubDate>
            <atom:updated>2018-04-06T02:48:47.988Z</atom:updated>
            <content:encoded><![CDATA[<p>As we rolled out and stabilized our <a href="https://medium.com/hadoop-noob/a-realtime-flink-parquet-data-warehouse-df8c3bd7401"><strong>Realtime Flink Parquet Data Warehouse</strong></a>, we are considering ingest parquet data into druid directly. We follow the <a href="http://druid.io/docs/latest/development/extensions-contrib/parquet.html">guideline here</a>, everything seems working well in the beginning. When our QA team runs integration test on the dashboard which powered by druid, they found some fields are not correct. As you can tell from the title, all these fields are <a href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists"><strong>List type in parquet</strong></a>.</p><p>Let me give an example. Let’s say, we have a field which is List of String, data looks like {“Alice”, “Bob”}. When we use the druid-parquet-extension, the data shows up {“element”: “Alice”, “element”: “Bob”}. If you by chance read my <a href="https://medium.com/hadoop-noob/presto-parquet-reader-fc7c333fc0a4"><strong>previous blog about parquet</strong></a>, we know in the parquet data, the list is represented as three level structure. something like</p><pre>optional group name_tag_array(LIST) {<br>     repeated group list {<br>         optional binary element (UTF8);<br>     }<br> }</pre><p>So it looks like the extra <strong><em>element</em></strong> comes from the schema. From what we have so far, the problem happens in the parquet reader.</p><p>We took a further look at druid-parquet-extension, it says it depends on druid-avro-extension. Okay, it is not hard to guess, druid hadoop indexer actually reads parquet files from HDFS or S3 and then use Avro object to hold the data in memory.</p><p>If you confused about why Avro is needed, I suggest you read <a href="http://grepalex.com/2014/05/13/parquet-file-format-and-object-model/">this blog</a>. Quote from that blog:</p><blockquote><em>1. Storage formats</em>, which are binary representations of data. For Parquet this is contained within the <a href="https://github.com/Parquet/parquet-format">parquet-format</a> GitHub project.</blockquote><blockquote><em>2. Object model converters</em>, whose job it is to map between an external object model and Parquet’s internal data types. These converters exist in the <a href="https://github.com/Parquet/parquet-mr">parquet-mr</a> GitHub project.</blockquote><blockquote><em>3. Object models</em>, which are in-memory representations of data. <a href="http://avro.apache.org/">Avro</a>, <a href="http://thrift.apache.org/">Thrift</a>, <a href="https://code.google.com/p/protobuf/">Protocol Buffers</a>, <a href="http://hive.apache.org/">Hive</a> and <a href="http://pig.apache.org/">Pig</a> are all examples of object models. Parquet does actually supply an <a href="https://github.com/Parquet/parquet-mr/tree/master/parquet-column/src/main/java/parquet/example">example object model</a> (with <a href="https://github.com/Parquet/parquet-mr/tree/master/parquet-hadoop/src/main/java/parquet/hadoop/example">MapReduce support</a> ) , but the intention is that you’d use one of the other richer object models such as Avro.</blockquote><p><strong>Parquet is storage formats. Avro can be used as both object model and storage format</strong>. When we want to use parquet in our program (spark/flink/java/python), we need to have a object model to hold the data which could be Protobuf, POJO or Avro.</p><p><strong>In this druid case, the issue is in the object model converters.</strong></p><p>How can we prove our assumption? The easiest way is debugging it in your favourite IDE. I git cloned the druid repo and put several break points in the druid-parquet-extension modules. Here is what I got from their <a href="https://github.com/druid-io/druid/blob/master/extensions-contrib/parquet-extensions/example/wikipedia_list.parquet"><strong>parquet file in the test folder</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*nndBMrVLbiRVgonftZnlpA.png" /></figure><p>Look at the {“array”: “en”}, the extra data is <strong>array instead of element </strong>simply because in the parquet schema, it is <strong>array. </strong>The actual schema is:</p><pre>optional group language (LIST) {<br>    repeated group bag {<br>      optional binary array (UTF8);<br>    }<br>  }</pre><p>All right, to conclude, if we could fix the schema converter, it will be resolved. Let me give you a pointer to get started. The converter is <a href="https://github.com/druid-io/druid/blob/master/extensions-contrib/parquet-extensions/src/main/java/org/apache/parquet/avro/DruidParquetReadSupport.java#L98">here</a>. Let me know if anyone patches it. Me, no time for open source, need to make some money :).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab9ddfb150e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/druid-parquet-extension-on-array-list-type-ab9ddfb150e6">Druid parquet extension on Array/List type</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dremio — best Parquet viewer]]></title>
            <link>https://medium.com/hadoop-noob/dremio-best-parquet-viewer-a340f6bab003?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/a340f6bab003</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[parquet]]></category>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Tue, 13 Feb 2018 01:57:58 GMT</pubDate>
            <atom:updated>2018-02-13T01:57:58.056Z</atom:updated>
            <content:encoded><![CDATA[<p>Sorry, I trolled on the title. Probably because I expected too much on it.</p><p>Back to early last year, Dremio came to our office and did a demo. It was a very informative talk and we asked a lot of Parquet related questions since they are contributors behind the Parquet (well, they original from twitter, i think). At that time, I also learned Apache Arrow which only has less than 100 github stars at that time. Apache Arrow seems very legit to me since I am dealing with different data format (Protobuf, Avro, Parquet) and different Systems (Mapreduce, Spark, Flink, Presto) every day. A unified data layer will be a Dream.</p><p>Anyway, until very recent I saw there is a community version of Dremio so I decide to give it a try. I downloaded it from <a href="https://www.dremio.com/download/">Here</a>. I wrapped it in a docker container and put it on our Mesos cluster under Marathon.<strong> I use all default conf and 1 CPU, 4 GB memory.</strong></p><p>Now I need to choose a <strong>Data Source</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pahys4U1ISDDDtWUS0999A.png" /></figure><p>Since we don’t have Hive or Redshift (we have Presto), I just choose HDFS. All our data are partitioned by year, month, day and hour and in parquet format, I feel like it should be very easy for Dremio to dynamically create a dataset based on the partitions (partition discovery) and schemas (schema merging). Think about it, it is just like you write down:</p><p><strong>spark.read.option(“mergeSchema”, “true”).parquet(“data/test_table”)</strong></p><p>Dremio should just persist the metadata somewhere.</p><p>But it just crashed. Maybe I need give it more resources or tune the conf a little bit. I think as a Product, it should be intuitive to use or evaluate. Of course if it is an open source project, it make sense it has some glitches.</p><p>So do I still use it. Yes I do. I use it as a parquet viewer. When I want to look at a single parquet file on HDFS or S3, it works well and its UI is pretty good :). My initial look is definitely biased. I just don’t have much time to play with it. I don’t think it suits our needs right now. To me, it just has a better dashboard, not much advantage against Presto. Although its data reflections seems very legit, our current presto setup is just fast enough for our current volume (40 cores handles Terabytes easily).</p><p>I will definitely update this when I play it more! If anyone can point me a docker image (maybe a cluster mode?) I can play with, it will be awesome.</p><p>I just remembered why I didn’t just use the mac version. I downloaded it, it is not starting :(</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a340f6bab003" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/dremio-best-parquet-viewer-a340f6bab003">Dremio — best Parquet viewer</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flink Parquet Writer]]></title>
            <link>https://medium.com/hadoop-noob/flink-parquet-writer-d127f745b519?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/d127f745b519</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[parquet]]></category>
            <category><![CDATA[flink]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Wed, 08 Nov 2017 00:12:46 GMT</pubDate>
            <atom:updated>2017-11-08T00:12:45.952Z</atom:updated>
            <content:encoded><![CDATA[<p>From last post, we learned if we want to have a streaming ETL in parquet format, we need to implement a flink parquet writer. So Let’s implement the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/streaming/connectors/fs/Writer.html"><strong>Writer Interface</strong></a>.</p><h3>Writer V1:</h3><pre>public class FlinkAvroParquetWriterV1&lt;T&gt; implements Writer&lt;T&gt;<br>{<br>    private transient ParquetWriter&lt;T&gt; writer;<br>    private String schema;<br>    private CompressionCodecName compressionCodecName = CompressionCodecName.<em>GZIP</em>;<br><br>    public FlinkAvroParquetWriterV1(String schema) {<br>        this.schema = schema;<br>    }<br>    @Override<br>    public void open(FileSystem fs, Path path) throws IOException<br>    {<br>        Configuration conf = new Configuration();<br>        conf.setBoolean(<em>ADD_LIST_ELEMENT_RECORDS</em>, false);<br>        conf.setBoolean(<em>WRITE_OLD_LIST_STRUCTURE</em>, false);<br>        writer = AvroParquetWriter<br>                .&lt;T&gt;<em>builder</em>(path)<br>                .withSchema(new Schema.Parser().parse(schema))<br>                .withCompressionCodec(compressionCodecName)<br>                .withPageSize(8 * 1024)<br>                .withConf(conf)<br>                .build();<br>    }<br><br>    @Override<br>    public long flush() throws IOException<br>    {<br>        return writer.getDataSize();<br>    }<br><br>    @Override<br>    public long getPos() throws IOException<br>    {<br>        return writer.getDataSize();<br>    }<br><br>    @Override<br>    public void close() throws IOException<br>    {<br>        writer.close();<br>    }<br><br>    @Override<br>    public void write(T element) throws IOException<br>    {<br>        writer.write(element);<br>    }<br><br>    @Override<br>    public FlinkAvroParquetWriterV1&lt;T&gt; duplicate()<br>    {<br>        return new FlinkAvroParquetWriterV1&lt;&gt;(schema);<br>    }</pre><p>We return <strong>getDataSize </strong>in both getPos and flush because there is no getPos and flush method in <strong>AvroParquetWriter</strong>. So is this legit? The answer is <strong>NO</strong>. Let’s take a closer look.</p><h4>What exactly does getDataSize return?</h4><pre>public long getDataSize() {<br>    return this.lastRowGroupEndPos + this.columnStore.getBufferedSize();<br>}</pre><p>What exactly is <strong>buffer size</strong>?</p><blockquote>approximate size of the buffered encoded binary data</blockquote><p>So now we learned:</p><ul><li>getDataSize returns approximate size</li><li>getDataSize returns the size of the mix of on disk &amp; in memory data.</li></ul><h4>What does getPos do?</h4><p>Remember we want to use BucketingSink in the last post? Let’s look at <a href="https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459">where getPos get called in BucketingSink</a>. It controls when the sink closes the file. It doesn’t matter much since who cares if the file is slightly bigger or smaller than the <strong>batch size</strong>.</p><h4>What does flush do?</h4><p>We know Flink uses Checkpointing to ensure <strong>exact one</strong>. So Let’s take a look at the <a href="https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L688">snapshotState</a> function. We Learned BucketingSink stores the current file length, so when it <a href="https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L800">handleRestoredBucketState</a>, it truncates the invalid file to a valid position. So when the flush method get called, it should return the exact position of the DataStream on <strong>disk</strong></p><h4>What’s the potential solutions:</h4><ol><li>Can we just uses lastRowGroupEndPos as the DataStream position to restore the file? No. We will lose the data buffered in memory</li><li>Can we implement the flush function for AvroParquetWriter? Yes. But it is still not enough. We will cover this later.</li><li>I am not familiar with parquet. I don’t want to touch <a href="http://github.com/apache/parquet-mr">parquet-mr</a>. We all know when the writer is closed. All the data in memory will be flushed to disk. So we can simply call close() in flush(). Then how about the restore part? We can simply just do not restore. Because if the flink pipeline fails in the middle, the file will remain in-progress or pending state. The file in-progress or pending will have an underscore “_” prefix which can be used as a pattern for ignored files.</li></ol><h3>flush() Implementation</h3><p>We want to implement flush because the #3 solution works but not perfect. Imaging if your checkpoint interval is small, e.g. 10s, you will end up having tons of small files. We all know we don’t like small files in “big data” world.</p><p>In InternalParquetRecordWriter</p><pre>public long flush() throws IOException {<br>    long memSize = this.columnStore.getBufferedSize();<br>    LOG.info(String.format(&quot;Flushing %,d records to disk, mem size %,d.&quot;, new Object[]{Long.valueOf(this.recordCount), Long.valueOf(memSize)}));<br>    this.flushRowGroupToStore();<br>    this.initStore();<br>    this.recordCountForNextMemCheck = Math.min(Math.max(100L, this.recordCount / 2L), 10000L);<br>    this.lastRowGroupEndPos = this.parquetFileWriter.getPos();<br>    return this.lastRowGroupEndPos;<br>}</pre><p>In ParquetWriter</p><pre>public void sync() throws IOException {<br>    this.out.sync();<br>}</pre><h3>Writer V2</h3><pre>public class FlinkAvroParquetWriterV2&lt;T&gt; implements Writer&lt;T&gt;<br>{<br>    private transient ParquetWriter&lt;T&gt; writer;<br>    private String schema;<br>    private CompressionCodecName compressionCodecName = CompressionCodecName.<em>GZIP</em>;<br><br>    public FlinkAvroParquetWriterV2(String schema) {<br>        this.schema = schema;<br>    }<br>    @Override<br>    public void open(FileSystem fs, Path path) throws IOException<br>    {<br>        Configuration conf = new Configuration();<br>        conf.setBoolean(<em>ADD_LIST_ELEMENT_RECORDS</em>, false);<br>        conf.setBoolean(<em>WRITE_OLD_LIST_STRUCTURE</em>, false);<br>        writer = AvroParquetWriter<br>                .&lt;T&gt;<em>builder</em>(path)<br>                .withSchema(new Schema.Parser().parse(schema))<br>                .withCompressionCodec(compressionCodecName)<br>                .withPageSize(8 * 1024)<br>                .withConf(conf)<br>                .build();<br>    }<br><br>    @Override<br>    public long flush() throws IOException<br>    {<br>        return writer.flush();<br>    }<br><br>    public static void appendFooter(FileSystem fs, Path path, byte[] footer) throws IOException<br>    {<br>        FSDataOutputStream outStream = fs.append(path);<br>        outStream.write(footer);<br>        outStream.hsync();<br>        outStream.close();<br>    }<br><br>    public byte[] getFooter() throws IOException<br>    {<br>        return writer.getFooter();<br>    }<br><br>    @Override<br>    public long getPos() throws IOException<br>    {<br>        return writer.getPos();<br>    }<br><br>    @Override<br>    public void close() throws IOException<br>    {<br>        writer.close();<br>    }<br><br>    @Override<br>    public void write(T element) throws IOException<br>    {<br>        writer.write(element);<br>    }<br><br>    @Override<br>    public FlinkAvroParquetWriterV2&lt;T&gt; duplicate()<br>    {<br>        return new FlinkAvroParquetWriterV2&lt;&gt;(schema);<br>    }<br>}</pre><p>You probably noticed, two more functions <strong>getFooter &amp; appendFooter</strong>. What’s that and why we need that?</p><p>We all know parquet file<strong> </strong>stores its metadata in its footer. When we snapshot the current state, we also need to snapshot the footer because when we restore the state, we also need to put the footer at end of the parquet file.</p><p>That’s all we need to implement a Parquet Writer in Flink. In order to implement it, you need to understand how Flink checkpoiting and how parquet works. I hope I covered all the why &amp; how. If I didn’t make it clear enough, please comment down below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d127f745b519" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/flink-parquet-writer-d127f745b519">Flink Parquet Writer</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Presto In Production]]></title>
            <link>https://medium.com/hadoop-noob/presto-in-action-d706b0f6617?source=rss----39bc3bbb9264---4</link>
            <guid isPermaLink="false">https://medium.com/p/d706b0f6617</guid>
            <category><![CDATA[presto]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Hao Gao]]></dc:creator>
            <pubDate>Fri, 03 Nov 2017 16:52:43 GMT</pubDate>
            <atom:updated>2018-03-06T00:56:47.701Z</atom:updated>
            <content:encoded><![CDATA[<h3>So what’s Presto</h3><p><em>“Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.”</em></p><h3>Challenges</h3><p>The most important problem we want to solve here is the Data Visibility. We Already have data sitting on S3 and HDFS, we want to have a Minimum effort to bring Data to everyone. When we start looking, we want to find a tool to fits our needs:</p><ol><li><strong>Data exploration.</strong> We already implemented our ETL pipelines using Flink, spark. We have our warehouse in both S3 and HDFS. Before Presto, we usually open a pyspark shell and write SparkSQL. It works well for people at Data Team since we all know spark. But people don’t know spark or even programming will have no clue how to access the data. We are tired of writing scripts to export the data to Postgres so people can access to through SQL client.</li><li><strong>Troubleshooting data pipelines. </strong>Everyday, we work on bringing more data to our warehouse. Usually data flows between several systems. For example, we have a kafka topic to log all events. Then we annotate the record and splits them into different topics. Each sub topic, we will use flink to persist the data into HDFS. We try our best to test pipelines but no one writes bug free code. So when we have discrepancy in warehouse, it is hard to debug, since you need to pull the data from different data source and figure out where exactly the bug is</li></ol><h3>Presto Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-JsptDk57Gtvu6uF." /></figure><p>Above diagram shows our current Presto setup.</p><ol><li>Our data warehouse is on S3 and HDFS, we maintain external tables mapping in hive metastore. Once the new partition of the table is created on S3 or HDFS, we add the new partition to hive metastore. Because we use flink streaming to ingest the data to HDFS, we can archive very low latency on data warehouse.</li><li>Presto can bring warehouse, postgres and Kafka data together. It is much easy to troubleshoot data discrepancy because we can track the record across different systems. Presto supports pluggable connectors that provide data to query. So Whenever we add another data source, we can easily bring it into Presto.</li><li>Presto supports different kinds of clients so we can use our favorite BI tools. Because it supports JDBC and ODBC, it is very easy to connect Tableau, Zeppelin and Dbeaver to Presto. Presto also provides python driver like pyhive, so you can programmatically use presto in the code, for example, turning the Presto result into a Pandas dataframe</li></ol><h3>Scale Up/Down</h3><p>Our Presto cluster is busy at work hours and less busy at night. We give it more resources when it busy. We leverage our Mesos’s Marathon framework to achieve this. We run Presto under Marathon, if we want to scale it up, we just need to add more instances. At night, we scale it down by reducing the presto workers. Marathon will also restart the worker or coordinator if it dies.</p><p>We are also experimenting metrics based scaling. In the future, when the CPU or Memory Usage is tight, we will scale it automatically, when it is idling, we take back the resources.</p><h3>Performance</h3><p>We did some <a href="https://medium.com/hadoop-noob/benchmark-spark-sql-vs-presto-a55932accc74"><strong>benchmark</strong></a> on Presto but performance is not the most important factor. There are lots of Presto performance benchmarks available online.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d706b0f6617" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hadoop-noob/presto-in-action-d706b0f6617">Presto In Production</a> was originally published in <a href="https://medium.com/hadoop-noob">Hadoop Noob</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>