<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manu Zhang on Medium]]></title>
        <description><![CDATA[Stories by Manu Zhang on Medium]]></description>
        <link>https://medium.com/@manuzhang?source=rss-f54e395fbec------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lIRngML6L5sACJpB.jpeg</url>
            <title>Stories by Manu Zhang on Medium</title>
            <link>https://medium.com/@manuzhang?source=rss-f54e395fbec------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:41:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manuzhang/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Regression in taking LIMIT rows from a bucket table in Spark 3.1.1]]></title>
            <link>https://medium.com/@manuzhang/regression-in-taking-limit-rows-from-a-bucket-table-in-spark-3-1-1-4542a7338417?source=rss-f54e395fbec------2</link>
            <guid isPermaLink="false">https://medium.com/p/4542a7338417</guid>
            <category><![CDATA[spark-sql]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Manu Zhang]]></dc:creator>
            <pubDate>Tue, 15 Mar 2022 12:10:09 GMT</pubDate>
            <atom:updated>2022-03-15T12:10:09.602Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Cross posted from <a href="https://manuzhang.github.io/posts/2022-03-15-regression-limit-bucket-spark311/index.html">https://manuzhang.github.io/posts/2022-03-15-regression-limit-bucket-spark311/index.html</a></blockquote><p>It’s been almost three years since my <a href="https://manuzhang.github.io/posts/2019-04-18-spark-python-udf/index.html">last post on Spark Python UDF</a> and I’ve dug deeper and learned more about Spark. I’ve also kept a practice of writing down how I approached to issues at work. However, I usually find they are either too specific to our own environment or have been well documented or I was occupied with some reading. Eventually, there can be no more “excuse” and here we go again!</p><h3>Regression</h3><p>During migration of Spark applications from 2.3.1 to 3.1.1, a regression was reported for the following query, whose running time increased from 33 seconds to over 3 minutes.</p><pre>SELECT i, k<br>FROM t<br>WHERE j=&#39;z&#39;<br>LIMIT 10;</pre><p>t is a bucket table, with 10000 buckets, bucket column i and sort column j, and each bucket file is around 3 GB.</p><h3>Diagnostics</h3><p>Firstly, looking from the application UI, the query was executed quite differently. In 2.3.1, the result was returned early after 1 task reading 487.7 MB data, which is an expected behavior for LIMIT clause. In 3.1.1, however, more than 3 TB data need be read with over 200k tasks from 9 stages. It&#39;s almost a full table scan!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HILBtAd89rt9Q4vA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aMkFW-BtuvePqhJK.png" /></figure><p>Meanwhile, this driver log in 2.3.1 indicted it’s a bucketed scan which I didn’t find in 3.1.1.</p><pre>INFO FileSourceScanExec: Planning with 10000 buckets</pre><p>Okay, I remember that in bucketed scan, when dividing source data into partitions, a bucket file is a partition regardless of its size. Otherwise, the partition size is decided by spark.sql.files.maxPartitionBytes which defaults to 128 MB. Hence, for 10000 bucket files, no more than 10000 tasks are needed to find 10 rows satisifying j=&#39;z&#39; with bucketed scan. In contrast, there could be as many as 240,000(3 * 1024 * 10000 / 128) tasks with non-bucketed scan. Nonetheless, in this case Spark looked no further than the first file with bucketed scan in 2.3.1, which corresponded to at most 24 tasks with non-bucketed scan.</p><h4>Why were over 200k tasks launched in 3.1.1?</h4><blockquote>the devil is in the details</blockquote><p>After checking the codes, it turns out partitions are <a href="https://github.com/apache/spark/blob/v3.1.1/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L609">sorted and read in reverse size order</a> in non-bucketed scan. The tail of the first file, whose size is smaller than 128 MB, is left until the non-tail parts of all files have been scanned. And when the answer happened to lie in the tail of the first file here, all files need be scanned!</p><h4>And why was the bucketed scan disabled in 3.1.1?</h4><p>The behavior change was brought in by <a href="https://github.com/apache/spark/pull/29804">[SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically</a> with a new config spark.sql.sources.bucketing.autoBucketedScan.enabled. The config was later <a href="https://github.com/apache/spark/pull/30138">enabled by default except for cache query</a>. After switching off the config and always enabling bucketed scan in 3.1.1, the query is executed as fast as 2.3.1.</p><p>The rational behind the change is to disable bucketed scan when there are no benefits, e.g. no join or group by on bucket column. It looks, however, not all cases have been considered when deciding whether bucketed scan is benefitting.</p><h4>How were the tasks scheduled in 3.1.1?</h4><p>One more puzzle for me is the special incremental way the tasks were scheduled in 3.1.1. Searching for keyword “limit” in configs has led me to spark.sql.limit.scaleUpFactor whose doc says</p><blockquote>Minimal increase rate in number of partitions between attempts when executing a take on a query. Higher values lead to more partitions read. Lower values might lead to longer execution times as more jobs will be run. (Default value is 4)</blockquote><p>I tried increasing the config to 300k and the performance was a bit better with 239999 tasks immediately being scheduled in the second stage. I suppose it would much better if I could got more resources to run those tasks in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V5-r_j9lbtPyc3PK.png" /></figure><h3>Summary</h3><p>These configs are worth checking when taking LIMIT rows with filter from a bucket table.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3154e840de4911cff5f980faa492bb1f/href">https://medium.com/media/3154e840de4911cff5f980faa492bb1f/href</a></iframe><p>I also <a href="https://github.com/apache/spark/pull/35514">raised to the community that such behavior change need be put in migration guide but they didn’t agree</a>.</p><p>An important lesson I’ve learned is to resolve all the doubts before turning to other suspicious points. In this case, I got distracted by the differences in execution plans and spent a lot of time in comparing how LIMIT clause was compiled into physical plans from 2.3.1 to 3.1.1.</p><h3>References</h3><ol><li><a href="https://spark.apache.org/docs/3.1.1/configuration.html">Spark 3.1.1 Configuration</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4542a7338417" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Note about Spark Python UDF]]></title>
            <link>https://medium.com/@manuzhang/note-about-spark-python-udf-9ddc3c162f77?source=rss-f54e395fbec------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ddc3c162f77</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[udf]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Manu Zhang]]></dc:creator>
            <pubDate>Thu, 18 Apr 2019 06:44:35 GMT</pubDate>
            <atom:updated>2019-04-18T06:44:35.275Z</atom:updated>
            <content:encoded><![CDATA[<p>The other day, my colleague was developing a PySpark(2.3.1) application which reads Chinese sentences from a Hive table, tokenizes them with a Python UDF and saves the first words into another table. The codes are roughly like this.</p><pre>def <strong>tokenize</strong>(sentence):<br>    return tokenizer<strong>.</strong>tokenize(sentence)[0]</pre><pre>df <strong>=</strong> spark<strong>.</strong>sql(&quot;select sentence from db.article&quot;)<br>df <strong>=</strong> df<strong>.</strong>repartition(1000)</pre><pre>tokenize_udf <strong>=</strong> udf(tokenize, StringType())<br>df <strong>=</strong> df<strong>.</strong>withColumn(&#39;word&#39;, tokenize_udf(&#39;sentence&#39;))<br>df<strong>.</strong>filter(df<strong>.</strong>word <strong>!=</strong> &#39;&#39;)</pre><pre>df<strong>.</strong>write<strong>.</strong>format(&#39;orc&#39;)<strong>.</strong>mode(&#39;overwrite&#39;)<strong>.</strong>saveAsTable(&#39;db.words&#39;)</pre><p>The Python UDF was time-consuming so my colleague tried increasing parallelism with repartition from 99 to 1000. It didn’t work as expected and the job still stuck in those 99 tasks.</p><p>I looked into the physical plan and it turned out that the Python UDF (BatchEvalPython) had been executed twice, once in a push down filter before repartition and once after repartition.</p><pre>== Physical Plan ==<br>Execute CreateDataSourceTableAsSelectCommand CreateDataSourceTableAsSelectCommand `db`.`words`, Overwrite, [word#19]<br>+- *(3) Project [sentence#13, pythonUDF0#26 AS word#19]<br>   +- BatchEvalPython [tokenize(sentence#13)], [sentence#13, pythonUDF0#26]<br>      +- Exchange RoundRobinPartitioning(1000)<br>         +- *(2) Project [sentence#13]<br>            +- *(2) Filter NOT (pythonUDF0#25 = )<br>               +- BatchEvalPython [tokenize(sentence#13)], [sentence#13, pythonUDF0#25]<br>                  +- *(1) Filter (isnotnull(sentence#13) &amp;&amp; NOT (sentence#13 = ))<br>                     +- HiveTableScan [sentence#13], HiveTableRelation `db`.`sentence`, org.apache.hadoop.hive.ql.io.orc.OrcSerde, [sentence#13]</pre><p>I found <a href="https://issues.apache.org/jira/browse/SPARK-22541">another issue when Python UDF is used in filter</a> which led me to <a href="https://github.com/apache/spark/blob/v2.3.1/python/pyspark/sql/functions.py#L2113">an important note about Python UDF</a></p><pre>def <strong>udf</strong>(f<strong>=</strong>None, returnType<strong>=</strong>StringType()):<br>    &quot;&quot;&quot;Creates a user defined function (UDF).<br>    .. note:: The user-defined functions are considered deterministic by default. Due to<br>        optimization, duplicate invocations may be eliminated or the function may even be invoked<br>        more times than it is present in the query. If your function is not deterministic, call<br>        `asNondeterministic` on the user defined function. E.g.:</pre><p>Hence, it’s possible <strong>Python UDF is invoked more times than it is present in the query</strong>. There are more notes you should check out when your Spark application with Python UDF behaves strangely next time.</p><h3>Solution</h3><p>To increase parallelism of executing Python UDF, we can decrease the input split size as follows.</p><pre>spark.sparkContext._jsc.hadoopConfiguration().set(&quot;mapred.max.split.size&quot;, &quot;33554432&quot;)</pre><p>Thanks to this StackOverflow answer on <a href="https://stackoverflow.com/questions/28844631/how-to-set-hadoop-configuration-values-from-pyspark">How to set Hadoop configuration values from PySpark</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ddc3c162f77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NPE from Spark App that extends scala.App]]></title>
            <link>https://medium.com/@manuzhang/npe-from-spark-app-that-extends-scala-app-ef7378195850?source=rss-f54e395fbec------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef7378195850</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Manu Zhang]]></dc:creator>
            <pubDate>Sat, 02 Mar 2019 07:38:37 GMT</pubDate>
            <atom:updated>2019-03-02T07:39:34.618Z</atom:updated>
            <content:encoded><![CDATA[<p>Cross posted from <a href="https://manuzhang.github.io/2019/02/23/spark-app-npe.html">https://manuzhang.github.io/2019/02/23/spark-app-npe.html</a></p><h4>This is the log of a bizarre NullPointerException(NPE) from a Spark application that extends scala.App and a minor bug of Spark hidden for years.</h4><p>Last week, I was developing the following Spark (2.3.1) application that pushes data from Hive to a database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*tkmLWpXTfvKt_aEUOIM-0Q.png" /></figure><p>The local variables foo and bar are used to configure the DbClient on remote executors. I was expecting them to be shipped through <a href="https://spark.apache.org/docs/latest/rdd-programming-guide.html#understanding-closures-">closures</a> but somehow they weren’t.</p><p>My colleague Vincent pointed me to ClosureCleaner which logDebug <a href="https://github.com/apache/spark/blob/v2.3.1/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala#L221">all fields, methods and classes</a> in closures. I enabled driver’s debug log and the absence of foo and bar assured me that they were not captured by closure.</p><p>Then I looked into what happens in foreachPartition. The byte code reminded me that foo and bar were static members of Test$ rather than local variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B8-j7n7XgwdncGj-KpSmuQ.png" /></figure><p><strong>It suddenly struck me that the initialization of the class (</strong><strong>Test$) that extends</strong><strong>scala.App is actually delayed to its </strong><strong>main() method</strong>. That’s why foo and barwere uninitialized as confirmed by <a href="https://scala-lang.org/files/archive/api/2.11.12/#scala.App">scala.App’s doc</a>,</p><blockquote>It should be noted that this trait is implemented using the <a href="https://scala-lang.org/files/archive/api/2.11.12/scala/DelayedInit.html">DelayedInit</a> functionality, which means that fields of the object will not have been initialized before the main method has been executed.</blockquote><p>Meanwhile, I googled “Spark closure problems” which led me to an ancient Spark jira <a href="https://issues.apache.org/jira/browse/SPARK-4170">Closure problems when running Scala app that “extends App”</a>. There was a fix that would print a warning when an application extends App.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N3UBHZ8E4GA_nI1jA0y61g.png" /></figure><p>Why have I never been warned ? After some digging, I realized the above “if clause” would never be true. This is because Scala compiler generate two Java classes Test and Test$ from object Test and the mainClass (Test) I passed in is not that (Test$) extends scala.App. It’s recorded in <a href="https://issues.apache.org/jira/browse/SPARK-26977">SPARK-26977</a>.</p><p>The solution is not (<strong>never for Spark</strong>) to extend scala.App and put everything in Test’s own main method. Then foo and bar are now initialized local variables and shipped remotely through closure.</p><h3>Appendix</h3><ol><li>How to decompile the anonymous function passed to foreachPartition ?</li></ol><p>javap -c Test\$\$anonfun\$1</p><p>2. How to enable driver’s debug log ?</p><p>Prepare a log4j.properties file with</p><p>log4j.rootLogger=DEBUG</p><p>and submit a Spark application with</p><p>--driver-java-options &quot;-Dlog4j.configuration=file:/path/to/log4j.properties&quot;</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef7378195850" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A whirlwind tour of Kafka Summit SF 2018]]></title>
            <link>https://medium.com/@manuzhang/a-whirlwind-tour-of-kafka-summit-sf-2018-fec98752804c?source=rss-f54e395fbec------2</link>
            <guid isPermaLink="false">https://medium.com/p/fec98752804c</guid>
            <category><![CDATA[summary]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Manu Zhang]]></dc:creator>
            <pubDate>Sat, 10 Nov 2018 07:23:59 GMT</pubDate>
            <atom:updated>2018-11-10T07:23:59.789Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Cross posted <a href="https://manuzhang.github.io/2018/11/10/kafka-summit-sf-2018.html">https://manuzhang.github.io/2018/11/10/kafka-summit-sf-2018.html</a></blockquote><p>Kafka Summit San Francisco 2018 took place last month with +1200 attendees from +350 companies. <a href="https://www.confluent.io/resources/kafka-summit-san-francisco-2018/">All 60 talks with videos and slides side by side</a> have been posted on Confluent’s website and nicely organized in keynotes and four tracks (Pipeline, Streams, Internals and Business).</p><p>I wasn’t there but I’ve glanced through all the slides and lingered over those I found more interesting. I’d like to share my impressions and provide a whirlwind tour of the conference. Here’s an <a href="https://www.confluent.io/blog/kafka-summit-san-francisco-2018-roundup">official wrap-up</a>, by the way.</p><h3>Is Kafka a Database</h3><p>As you know, Kafka was born a message queue and has grown into a full-fledged streaming platform with <a href="https://kafka.apache.org/documentation/#connect">Kafka Connect</a>, <a href="https://kafka.apache.org/documentation/streams/">Kafka Streams</a> and <a href="https://github.com/confluentinc/ksql">KSQL</a>. What else can Kafka be ? How about a database ! Martin Kleppmann argues that <a href="https://www.confluent.io/kafka-summit-SF18/is-kafka-a-database">Kafka is a database</a> and achieves <a href="https://en.wikipedia.org/wiki/ACID_(computer_science)">ACID</a> properties as in relational databases. This is mind-boggling since relation databases don’t provide ACID at scale. Note that you can’t set up a Kafka cluster and get ACID for free but need to carefully design your Kafka topics and streaming applications as demonstrated by the author. Still, I have some doubt over whether it can only ensure eventual consistency.</p><h3>Kafka for IoT</h3><ul><li><a href="https://en.wikipedia.org/wiki/MQTT">MQTT</a> is a lightweight messaging protocol, built for tens of thousands of client connections in unstable network but it lacks in buffering, stream processing, reprocessing and scalability. Those’re what Kafka is good at and they make a perfect match to <a href="https://www.confluent.io/kafka-summit-sf18/processing-iot-data-from-end-to-end">process IoT data from end to end</a> through <a href="https://www.slideshare.net/ConfluentInc/processing-iot-data-from-end-to-end-with-mqtt-and-apache-kafka/27">Kafka Connect</a>, <a href="https://www.slideshare.net/ConfluentInc/processing-iot-data-from-end-to-end-with-mqtt-and-apache-kafka/35">MQTT Proxy</a> or <a href="https://www.slideshare.net/ConfluentInc/processing-iot-data-from-end-to-end-with-mqtt-and-apache-kafka/38">REST Proxy</a>.</li></ul><h3>Global Kafka</h3><ul><li><a href="https://www.slideshare.net/ConfluentInc/data-streaming-ecosystem-management-at-bookingcom/26">Booking manages a global Kafka cluster with brokers spread over three zones</a>. They have containerized replicator, Kafka connect and Kafka monitor on Kubernetes. It’ worth mention that they have run into issues with distributed mode of Kafka Connect so the standalone mode is used instead.</li><li><a href="https://www.slideshare.net/ConfluentInc/more-data-more-problems-scaling-kafkamirroring-pipelines-at-linkedin">Linkedin has built its own Brooklin MirrorMaker (BMM)</a> to mirror 100+ pipelines across 9 data centers with only 9 BMM clusters. In contrast, Kafka MirrorMaker (KMM) needs 100+ clusters. Moreover, KMM is difficult to operate, poor to isolate failure and unable to catch up with traffic while BMM has dynamic configuration, automatically pausing and resuming mirroring at partition level, and good throughput with almost linear scalability.</li><li>To overcome the explosion of number of KMMs, <a href="https://www.slideshare.net/ConfluentInc/beyond-messaging-enterprisescale-multicloud-intelligent-routing/27">Adobe introduces a routing KMM cluster</a> to reduce the number of directly connected data center pipelines. They also have written <a href="https://medium.com/adobetech/creating-the-adobe-experience-platform-pipeline-with-kafka-4f1057a11ef">a nice summary of their Experience Platform Pipeline built on Kafka</a>.</li></ul><h3>Streaming platform</h3><p>When Kafka Streams first came out, I was wondering why I would need another Streaming platform given the place had already been crowded with <a href="https://flink.apache.org/">Flink</a>, <a href="https://spark.apache.org/streaming/">Spark Streaming</a>, <a href="https://storm.apache.org/">Storm</a>, <a href="https://github.com/gearpump/gearpump">Gearpump</a>, etc. Today it strikes me that why I would need another Streaming platform and all the workloads to set up a cluster and maintain when I can do streaming with my Kafka already there. Futhurmore, Confluent adds <a href="https://github.com/confluentinc/ksql">KSQL</a>, a SQL interface, to relieve you of writing cumbersome codes.</p><ul><li>Thus, maybe it’s time to <a href="https://www.slideshare.net/ConfluentInc/crossing-the-streams-rethinking-stream-processing-with-kstreams-and-ksql-120334214">rethink stream processing with Kafka Streams and KSQL</a>. There is an illustrative analogy between Kafka ecosystem and Unix Pipeline.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*Lzzj85x2T9BGI7AI" /></figure><ul><li>Stream joins are not your father’s <a href="https://www.w3schools.com/sql/sql_join.asp">SQL joins</a>. How joins are implemented in Kafka Streams and when to use them ? Read this <a href="https://www.slideshare.net/ConfluentInc/zen-and-the-art-of-streaming-joinsthe-what-when-and-why">zen and the art of streaming joins</a>.</li><li>Companies like <a href="https://www.slideshare.net/ConfluentInc/data-streaming-ecosystem-management-at-bookingcom/5">Booking</a> and <a href="https://www.slideshare.net/ConfluentInc/realtime-dynamic-data-export-using-the-kafka-ecosystem/33">Braze</a> are building their streaming pipeline around Kafka Connect (data import/export) and Kafka Streams (data processing).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*XHiBxGg6jxhL9-mW" /></figure><ul><li>Change Data Capture (CDC) is a way to make use of data and schema changes of your database. <a href="https://www.slideshare.net/ConfluentInc/change-data-streaming-patterns-for-microservices-with-debezium">Deberium is a CDC platform for various databases based on Kafka</a> from Red Hat. <a href="https://www.slideshare.net/ConfluentInc/pinterests-story-of-streaming-hundreds-of-terabytes-of-pins-from-mysql-to-s3hadoop-continuously">Pinterest also shares their story of streaming hundreds of TBs of Pins from MySQL to S3</a>.</li></ul><h3>Kafka on Cloud</h3><ul><li>Intuit have deployed <a href="https://www.slideshare.net/ConfluentInc/kafka-on-kubernetesfrom-evaluation-to-production-at-intuit">Kafka on Kubernetes in production</a> with proper configurations on load balancing, security, etc.</li><li><a href="https://www.slideshare.net/ConfluentInc/change-data-streaming-patterns-for-microservices-with-debezium/28">Red Hat provide an enterprise distribution of Kafka on Kubernetes/OpenShift</a> with an open-source upstream, <a href="https://github.com/strimzi">Strimzi</a>. Typical Red Hat.</li><li>Google is not using Kafka internally but there is a demand for Kafka on Google Cloud Platform (GCP). It’s interesting to learn <a href="https://www.slideshare.net/ConfluentInc/putting-kafka-together-with-the-best-of-google-cloud-platform">Google’s perspective on Kafka and how they fit Kafka into GCP</a>.</li><li>Confluent offers some official <a href="https://www.slideshare.net/ConfluentInc/deploying-kafka-streams-applications-with-docker-and-kubernetes/33">recommendations on deploying Kafka Streams with Docker and Kubernetes</a>.</li></ul><h3>Internals</h3><p>As a developer, I love nearly every deck of the Internals track.</p><ul><li>Zalando shared their <a href="https://www.slideshare.net/ConfluentInc/war-stories-diy-kafka-120353401">War stories: DIY Kafka</a>, especially lessons learned in backing up Zookeeper and Kafka</li></ul><blockquote><em>Backups are only backups if you know how to restore them</em></blockquote><ul><li>Another war story from a Kafka veteran, Todd Palino, on <a href="https://www.slideshare.net/ConfluentInc/urp-excuse-you-the-three-metrics-you-have-to-know-120353592">monitoring Linkedin’s 5-trillion-message-per-day Kafka cluster</a>. His lesson is not to monitor and alert on everything but to keep an eye on three key metrics, under-replicated partition, request handler and request timing. I totally agree with him that</li></ul><blockquote>sleep is best in life</blockquote><ul><li><a href="https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-living-through-filesystems">ZFS makes Kafka faster and cheaper</a> through improving cache hit rates and making clever use of I/O devices.</li><li>Jason Gustafson walked you through <a href="https://www.slideshare.net/ConfluentInc/hardening-kafka-replication">failure scenarios of Kafka log replication and hardening it with TLA+ model checker</a>. ZipRecruiter is also <a href="https://www.slideshare.net/ConfluentInc/using-chaos-engineering-to-level-up-apache-kafka-skills">using Chaos Engineering to level up Kafka skills</a>.</li><li>Kafka is famous for using <a href="http://man7.org/linux/man-pages/man2/sendfile.2.html">sendfile system call</a> to accelerate data transfer. Do you know sendfile can be blocking ? I really enjoy how <a href="https://www.slideshare.net/ConfluentInc/kafka-multitenancy160-billion-daily-messages-on-one-shared-cluster-at-line/">Yuto from Line has approached the issue from hypothesis to solution</a>. (More details at the amazing <a href="https://issues.apache.org/jira/browse/KAFKA-7504">KAFKA-7504</a>). His <a href="https://www.slideshare.net/ConfluentInc/one-day-one-data-hub-100-billion-messages-kafka-at-line">analysis of long GC pause harming broker performance caused by mmap</a> from last year is a great computer system lesson as well.</li><li>Who has not secured Kafka ? If you don’t know the answer, then Stephane Maarek from DataCumulus <a href="https://www.slideshare.net/ConfluentInc/kafka-security-101-and-realworld-tips">offers some introductions and real-world tips</a>. For example,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*oemAMqZ303MG3YCW" /></figure><ul><li><a href="https://www.slideshare.net/ConfluentInc/reliable-message-delivery-with-apache-kafka">Uber keeps pushing for producer performance improvement without sacrificing at-least-once delivery</a>. They shared about their best practices such as increasing replica fetch frequency and reducing thread contention and GC pauses.</li></ul><h3>Wrap-up</h3><p>Kafka is definitely the backbone of companies’ data architectures and serves as many as billions of messages per day. More and more streaming applications are built with Kafka Connect, Kafka Streams and KSQL. Meanwhile, quite a few companies are managing and mirroring Kafka at global scale. Finally, Kafka on Kubernetes is the new fashion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fec98752804c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The hidden cost of Spark withColumn]]></title>
            <link>https://medium.com/@manuzhang/the-hidden-cost-of-spark-withcolumn-8ffea517c015?source=rss-f54e395fbec------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ffea517c015</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Manu Zhang]]></dc:creator>
            <pubDate>Mon, 13 Aug 2018 13:17:16 GMT</pubDate>
            <atom:updated>2022-03-29T00:42:51.306Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Cross posted from <a href="https://manuzhang.github.io/2018/07/11/spark-catalyst-cost.html">https://manuzhang.github.io/2018/07/11/spark-catalyst-cost.html</a>.</blockquote><p>Recently, we’ve been working on machine learning pipeline with Spark, where <a href="https://spark.apache.org/sql/">Spark SQL &amp; DataFrame</a> is used for data preprocessing and <a href="https://spark.apache.org/mllib/">MLlib</a> for training. In one use case, the data source is a very wide Hive table of ~1000 columns. The columns are stored in String so we need to cast them to Integer before they can be fed into model training.</p><p>This was what I got initially with DataFrame Scala API (2.2.0).</p><pre>df<strong>.</strong>columns<strong>.</strong>foldLeft<strong>(</strong>df<strong>)</strong> <strong>{</strong> case <strong>(</strong>df<strong>,</strong> col<strong>)</strong> =&gt;<br>  df<strong>.</strong>withColumn<strong>(</strong>col<strong>,</strong> df<strong>(</strong>col<strong>).</strong>cast<strong>(</strong>IntegerType<strong>))</strong><br><strong>}</strong></pre><p>Since DataFrame is immutable, I created a new DataFrame for each Column casted using withColumn. I didn’t think that was be a big deal since at run time all columns would be casted in one shot thanks to <a href="https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html">Spark’s Catalyst optimizer</a>.</p><blockquote>At its core, Catalyst contains a general library for representing trees and applying rules to manipulate them. On top of this framework, we have built libraries specific to relational query processing (e.g., expressions, logical query plans), and several sets of rules that handle different phases of query execution: analysis, logical optimization, physical planning, and code generation to compile parts of queries to Java bytecode</blockquote><p>To my surprise, the job stuck in submission for minutes without outputting anything. Luckily, I have a nice colleague Vincent who saved my day with the following fix.</p><pre>df<strong>.</strong>select<strong>(</strong>df<strong>.</strong>columns<strong>.</strong>map <strong>{</strong> col =&gt;<br>  df<strong>(</strong>col<strong>).</strong>cast<strong>(</strong>IntegerType<strong>)</strong><br><strong>}</strong>: _*<strong>)</strong></pre><p>He suspected that it’s expensive to call withColumn for a thousand times. Hence, he dived into its implementation and found out the above in the private method withColumns called by withColumn. In his fast version, only one new DataFrame was created.</p><p>I wondered why there was a significant cost difference and looked further into it. After turning on the debug log, I saw a lot of === Result of Batch Resolution ===s in my slow version. It suddenly struck me that Catalyst’s analysis might not be free. A thousand withColumns were actually a thousand times of analysis, which held true for all APIs on DataFrame. On the other hand, analysis of transform on Column was actually lazy.</p><p>The log led me to org.apache.spark.sql.catalyst.rules.RuleExecutor where I spotted a timeMap tracking time running specific rules. What’s more exciting, the statistics was exposed through RuleExecutor.dumpTimeSpent which I could add to compare the costs in two versions.</p><pre>df<strong>.</strong>columns<strong>.</strong>foldLeft<strong>(</strong>df<strong>)</strong> <strong>{</strong> case <strong>(</strong>df<strong>,</strong> col<strong>)</strong> =&gt;<br>  println<strong>(</strong>RuleExecutor<strong>.</strong>dumpTimeSpent<strong>())</strong><br>  df<strong>.</strong>withColumn<strong>(</strong>col<strong>,</strong> df<strong>(</strong>col<strong>).</strong>cast<strong>(</strong>IntegerType<strong>))</strong><br><strong>}</strong><br>println<strong>(</strong>RuleExecutor<strong>.</strong>dumpTimeSpent<strong>())</strong></pre><pre>df<strong>.</strong>select<strong>(</strong>df<strong>.</strong>columns<strong>.</strong>map <strong>{</strong> col =&gt;<br>  println<strong>(</strong>RuleExecutor<strong>.</strong>dumpTimeSpent<strong>())</strong><br>  df<strong>(</strong>col<strong>).</strong>cast<strong>(</strong>IntegerType<strong>)</strong><br><strong>}</strong>: _*<strong>)</strong><br>println<strong>(</strong>RuleExecutor<strong>.</strong>dumpTimeSpent<strong>())</strong></pre><p>As expected, the time spent increased for each DataFrame#withColumn while that stayed the same for Column#cast. It would take about 100ms for one round of analysis. (I have a table of time spent for all 56 rules in 100 withColumn calls in appendix if you are curious).</p><h3>Summary</h3><ol><li>The hidden cost of withColumn is Spark Catalyst’s analysis time.</li><li>The time spent in Catalyst analysis is usually negligible but it will become an issue when there is a large number of transforms on DataFrame. It’s not unusual for a model to have 1000 features which may require preprocessing.</li><li>Don’t create a new DataFrame for each transform on Column. Create one at last with DataFrame#select</li></ol><h3>Appendix</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acde6a7b8f6081e5d929c907fdd17d93/href">https://medium.com/media/acde6a7b8f6081e5d929c907fdd17d93/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fbutton.like.co%2Fin%2Fembed%2Fowenzhang1990%2Fbutton&amp;display_name=LikeCoin&amp;url=https%3A%2F%2Fbutton.like.co%2Fowenzhang1990&amp;image=https%3A%2F%2Fstorage.googleapis.com%2Flikecoin-foundation.appspot.com%2Flikecoin_store_user_owenzhang1990_main%3FGoogleAccessId%3Dfirebase-adminsdk-eyzut%2540likecoin-foundation.iam.gserviceaccount.com%26Expires%3D2430432000%26Signature%3DDJmw6Eqj1zLnBjw7Ed8Mgmohn7nV48oRLBGchhajFgeFsQqXKCz0eTKg9oHHjM4vLvIjOBc7mO8Y9RzNnvDe%252Blg9MpYdsKYzWRgOmDlWbHj6zn%252BuHqTrTeQloVz0eJ1G1zLrsxG%252BELpHTmXPeyBgFqeWqNxLGBxZ7T30VgnKvcVyJjtbDCV7bTk5NTvavnoRZFuqGMBWdUFWccNkSH7jCysZ6hdvo4Ly7Uhn0nkDQp4B6%252F8IqYj5pZ5I5LpQ3PW9f0806riwEr8FATYZntjUw1dy7fY2F5k7vpQIDxKs3Y52hFjxm7%252BGGJftrknFHIq5Tin8ZaulYVZpygrPiGIykA%253D%253D&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=like" width="485" height="212" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd679bb1e83169e58e60d117b1dfe8c0/href">https://medium.com/media/bd679bb1e83169e58e60d117b1dfe8c0/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ffea517c015" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unikernel]]></title>
            <link>https://medium.com/@manuzhang/unikernel-67b1d2aa734c?source=rss-f54e395fbec------2</link>
            <guid isPermaLink="false">https://medium.com/p/67b1d2aa734c</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[unikernel]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Manu Zhang]]></dc:creator>
            <pubDate>Thu, 28 Jan 2016 00:00:00 GMT</pubDate>
            <atom:updated>2016-02-12T15:11:56.867Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>The first time I learned about Unikernel was from <a href="https://ma.ttias.be/what-is-a-unikernel/">What is a ‘unikernel’?</a> when glancing through posts from Pocket Recommended on my shuttle bus the other day. It’s totally out of curiosity and “Recommended” usually means a lot of people are watching it. After a quick read, I didn’t get the point. Is it something <a href="http://roadtounikernels.myriabit.com/">Docker</a> ? I forgot about it as soon as I got off the shuttle bus.</p><p>Then came the news that <a href="http://blog.docker.com/2016/01/unikernel/">Unikernel Systems Joins Docker</a>. Interesting ! Meanwhile, I was looking for materials for weekly internal sharing at work. Why didn’t I share about Unikernels ? I went on to google and found a bunch of resources from <a href="https://en.wikipedia.org/wiki/Unikernel">Wikipedia</a>, <a href="https://wiki.xenproject.org/wiki/Unikernels">Xen Wiki</a> and <a href="http://unikernel.org/">unikernel.org</a>.</p><h3>Why do we need Unikernel</h3><p>At the age of cloud computing, programmers Tom and Jerry rent two Linux virtual machines (VMs) from <a href="https://aws.amazon.com/ec2">Amazon EC2</a> to host their web servers, and the two VMs end up on the same physical machine. The enabing technique is called virtualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*lYdKIShoV5Xls9lh.png" /></figure><p>(Source: <a href="http://roadtounikernels.myriabit.com/">road to unikernels</a>)</p><p>As in the above image, hardware resources are virtualized by a <a href="https://en.wikipedia.org/wiki/Hypervisor">hypervisor</a> and shared among multiple VMs. One of the most well known hypervisors, <a href="http://www.xenproject.org/">Xen</a> (used by EC2, Rackspace, etc), is developed by the <a href="https://en.wikipedia.org/wiki/University_of_Cambridge_Computer_Laboratory">University of Cambridge Computer Laboratory</a>, <strong>who conceived Unikernels</strong>.</p><blockquote><em>Despite this shift from applications running on multi-user operating systems to provisioning many instances of single-purpose VMs, there is little actual specialisation that occurs in the image that is deployed to the cloud. We take an extreme position on specialisation, treating the final VM image as a single-purpose appliance rather than a general-purpose system by stripping away functionality at compile-time</em></blockquote><p>(Source: <a href="http://anil.recoil.org/papers/2013-asplos-mirage.pdf">Unikernels: Library Operating Systems for the Cloud</a> )</p><p>If Tom and Jerry only use the VMs to host their web servers, they probably don’t need a general-purpose system for a desktop computer. With a single-purpose Unikernels system, Tom and Jerry could link only the system functionality required by their web servers.</p><p>The benefits are</p><ul><li>improved security with reduced deployed codes and attack surface</li><li>small footprints and fast boot time</li><li>whole-system optimisation across device drivers and application logic</li></ul><h3>Now, what is a Unikernel</h3><blockquote><em>Unikernels are specialised, single-address-space machine images constructed by using library operating systems.</em></blockquote><p>(Source: <a href="https://en.wikipedia.org/wiki/Unikernel">Unikernel</a>)</p><p>Unikernel is built on library operating system (libOS).</p><h3>Library Operating System</h3><p>In a library operating system,</p><ul><li>there is only a single address space</li><li>device drivers are implemented as a set of libraries</li><li>access control and isolation are enforced as policies in the application layer</li></ul><p>Without the separation of user space and kernel space, there is no need for repeated privilege transitions to move data between them. On the other hand, the single address space makes it difficult to isolate mulitple applications running side by side on a libOS.</p><p>The drawback has been overcame by modern hypervisor which provides virtual machines with CPU time and strongly isolated virtual devices.</p><p>Another drawback is protocol libraries (e.g. device drivers) have to been rewritten to replace that of a tradtional operating system.</p><p>Fortunately, they have been implemented in <a href="https://mirage.io/">MirageOS</a>, the first prototype of a Unkernel.</p><h3>MirageOS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*HsmGUSUkjlY0HUGM.png" /></figure><p>The traditional software stacks are reduced to application code and mirage runtime. OS functionalities are imported as libraries provided by MirageOS. Every application is compiled into its own specialised OS.</p><p>Users could target their codebase towards different platforms, Unix for test and Xen on x86 or Xen on ARM for deployement, which are run on the cloud or embedded systems.</p><p>Besides MirageOS, there is an <a href="http://unikernel.org/projects/">expanding Unikernel ecosystem</a>.</p><blockquote><em>Some systems (like Rumprun) are language-agnostic, and provide a platform for any application code based on the requests it makes of the operating system… Unikernels can run in containers, on hypervisors, and on a wide array of bare-metal hardware.</em></blockquote><h3>Is Unikernel really needed</h3><p>My colleagues are uncovinced. On the internal sharing, they don’t think it’s worthwhile to throw all the existing software stacks in the Linux ecosystem. Meanwhile, the benefits brought by Unikernel are doubtable. For example, tailored Linux systems could be also small. <a href="https://github.com/gliderlabs/docker-alpine">docker-alpine</a> image is only 5 MB.</p><p>At the time of writing, another post popped up from Pocket Recommended, <a href="https://www.joyent.com/blog/unikernels-are-unfit-for-production">Unikernels are unfit for production </a>. The author questions the main reaons (performance, security, footprint) to adopt a Unikernel system. He thinks the most important reason that Unikernel is unfit for production is that <strong>Unikernels are entirely undebuggable</strong> (please read the original article for thorough explanations).</p><h3>Summary</h3><p>In this post, we get to know what Unikernel is and why it exists. It’s unclear whether Unikernel will become a new revolution for cloud computing like Docker (I also list the discussions on Hacker News and Reddit in the reference below).</p><h3>References</h3><ol><li><a href="http://queue.acm.org/detail.cfm?id=2566628">Unikernels: Rise of the Virtual Library Operating System</a> by Anil Madhavapeddy and David J. Scott.</li><li><a href="https://speakerdeck.com/amirmc/unikernels">Unikernels at PolyConf 2015</a> by Amir Chaudhry.</li><li><a href="https://news.ycombinator.com/item?id=10945219">Discussions on Hacker News</a></li><li><a href="https://www.reddit.com/r/programming/comments/4206cv/unikernel_systems_joins_docker/">Discussions on Reddit</a></li></ol><p><em>Originally published at </em><a href="http://manuzhang.github.io/2016/01/28/unikernel.html"><em>manuzhang.github.io</em></a><em> on January 28, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67b1d2aa734c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>