<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alfredo Fomitchenko on Medium]]></title>
        <description><![CDATA[Stories by Alfredo Fomitchenko on Medium]]></description>
        <link>https://medium.com/@alfredo.fomitchenko?source=rss-37332331be9a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tQKhS3gBxomF2B_A1QRx8A.jpeg</url>
            <title>Stories by Alfredo Fomitchenko on Medium</title>
            <link>https://medium.com/@alfredo.fomitchenko?source=rss-37332331be9a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:10:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alfredo.fomitchenko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Meet “spark-sight”: Spark Performance at a Glance]]></title>
            <link>https://medium.com/data-reply-it-datatech/meet-spark-sight-spark-performance-at-a-glance-b100a755538c?source=rss-37332331be9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b100a755538c</guid>
            <dc:creator><![CDATA[Alfredo Fomitchenko]]></dc:creator>
            <pubDate>Mon, 14 Aug 2023 08:01:49 GMT</pubDate>
            <atom:updated>2023-08-14T08:01:49.636Z</atom:updated>
            <content:encoded><![CDATA[<h4>Quick and easy insights on your Spark application performance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AWOVtzDgZmB3YEaI" /><figcaption>Photo by <a href="https://unsplash.com/@jeztimms?utm_source=medium&amp;utm_medium=referral">Jez Timms</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Apache Spark has become the go-to framework for big data processing. Its distributed computing architecture enables lightning-fast processing speed, fault tolerance, and the ability to handle massive amounts of data.</p><p>However, monitoring the performance of a Spark application can be a challenging task, particularly when dealing with complex and intricate workflows.</p><p>I personally love the ecosystem that the open-source community has created around Apache Spark, but I find that the mainstream monitoring tools are quite cumbersome to use. This is especially true for the official Spark UI, which, in my opinion, provides too many fine-grained details instead of big-picture summaries.</p><p>Have some solutions emerged in order to extend and complement the Spark UI experience? As an example, <a href="https://www.datamechanics.co/">Data Mechanics</a> has been developing <a href="https://github.com/datamechanics/delight">Delight</a>, sharing with the world</p><ul><li>an open-source agent collecting information from within the Spark application;</li><li>a <strong>closed-source </strong>dashboard displaying the performance of a Spark application in terms of CPU efficiency and memory usage.</li></ul><p>Even though the service is free of charge,</p><ol><li>you need to integrate into your Spark application their custom listener that collects and sends out data to their servers;</li><li>it may be difficult for the client to approve such practice (privacy concerns), or outright unable to (e.g. your application runs in a Glue job inside a VPC without internet access).</li></ol><p>I was inspired by this effort, and I was willing to gather everything I know about Spark. So I took on the challenge of recreating the same amazing experience of Delight for everybody to enjoy and contribute to.</p><p>This is why I created the Python package <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a>.</p><p>Here’s some highlights of the User Experience:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7VOOalHrmuAaOc5kScxlg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R_bg-n80VyniS7Vg3VQZzA.gif" /></figure><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is a less detailed, more intuitive representation of what is going on inside your Spark application in terms of performance:</p><ul><li>CPU time spent doing the “actual work”;</li><li>CPU time spent doing shuffle reading and writing;</li><li>CPU time spent doing serialization and deserialization;</li><li>Spill intensity per executor</li></ul><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is not meant to replace the Spark UI altogether, rather it provides a bird’s-eye view of the stages. This allows you to gain general information about the performance of your Spark application, and identify at a glance which portions of the execution may need improvement.</p><h3>User interface: the main Plotly figure</h3><p>The Plotly figure is the main component of the UI. It consists of three charts with synced x-axis.</p><p>Let’s dive into each chart.</p><h4>Top chart: efficiency in terms of CPU cores available for tasks</h4><p>The chart at the top is a stacked bar chart that breaks down, for each interval shown, how the Spark application has spent that period of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*GdbtyKadZWcR_erwUpiHLw.gif" /></figure><p>There are three categories shown:</p><ul><li><strong>Actual task work</strong>: percentage of CPU time spent doing the “actual work”, i.e. the work related to the tasks that executors were assigned to do. <br>I choose green for the color of this bar because it is the portion you wish to be as high as possible: the higher this portion, the higher the Spark application performance.</li><li><strong>Shuffle read and write</strong>: percentage of CPU time spent while shuffle reading or shuffle writing. <br>No distinction is made between the two operations, in fact they are ultimately considered to be time unrelated to “actual work”: the higher this portion, the lower the Spark application performance.</li><li><strong>Serialization, deserialization</strong>: this is the percentage of CPU time spent while serializing and deserializing tasks. <br>As in the case above, no distinction is made between the two operations, highlighting a lower Spark performance.</li></ul><h4>Middle chart: spill intensity per executor</h4><p>The chart in the middle is a timeline chart showing, for each executor, when and how much the executor has spilled to disk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*EPBmgb_h23qXYBEPcqQj6g.gif" /></figure><p>The chart can be read as follows:</p><ul><li>A constant value on the y-axis represents one of the executors, as shown by the legend on the left-hand side;</li><li>Time is depicted on the x-axis, and synchronized with all other charts;</li><li>Each portion is color-graded proportionally to the intensity of the spill. In the image above, you will notice that a more intense purple is assigned to higher values of spill.</li></ul><p>Caring about the spill of your Spark application is very important: when one of the partitions is too large to fit in the executor memory, the disk is accessed and performance decreases by orders of magnitude.</p><p>You can read more on spill <a href="https://michaelheil.medium.com/understanding-common-performance-issues-in-apache-spark-deep-dive-data-spill-7cdba81e697e">here</a>.</p><h4>Bottom chart: timeline of the Spark stages</h4><p>The chart at the bottom is a timeline chart showing the boundaries of the Spark stages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*PwR6AP9_DAEDl74PwEHfVA.gif" /></figure><p>This chart is useful for quick debugging of which stages are to be analyzed and improved, logically grouping what is shown in more detail in the other charts.</p><h3>How to install the package</h3><p>To install spark-sight,</p><pre>$ pip install spark-sight</pre><pre>$ spark-sight --help</pre><p>The help shows the parameters that the application receives:</p><ul><li>path : the local path to the Spark event log you have collected from your Spark application;</li><li>cpus : the total number of CPUs the cluster was composed of;</li><li>deploy_mode : how the Spark application was deployed. Can be cluster mode or client mode, where cluster mode will subtract the CPU used by the Spark driver from the CPU count</li></ul><p>Here’s an example of how to run spark-sight on your Spark event log:</p><ul><li>Unix</li></ul><pre>$ spark-sight \<br>    --path &quot;/path/to/spark-application-12345&quot; \<br>    --cpus 32 \<br>    --deploy_mode &quot;cluster_mode&quot;</pre><ul><li>Windows PowerShell</li></ul><pre>$ spark-sight `<br>    --path &quot;C:\path\to\spark-application-12345&quot; `<br>    --cpus 32 `<br>    --deploy_mode &quot;cluster_mode&quot;</pre><p>A new browser tab will be opened, where you will find the Plotly chart described above.</p><p>For more information, head over to the <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight Github repo</strong></a>.</p><h3>How is the chart built?</h3><p>The Spark event log is a simple text file that Spark is natively able to store somewhere for you to open and look through:</p><pre>--conf spark.eventLog.enabled=true<br>--conf spark.eventLog.dir=file:///c:/somewhere<br>--conf spark.history.fs.logDirectory=file:///c:/somewhere</pre><p>As described in the <a href="https://spark.apache.org/docs/latest/monitoring.html#executor-task-metrics">Spark documentation</a>, hidden somewhere in the text file are the performance data:</p><ul><li>SparkListenerTaskEnd events: for how long the task was shuffling, serializing, deserializing, and doing the “actual work” it was supposed to do in the first place</li><li>SparkListenerStageCompleted events: for how long the corresponding stage was in the submitted state</li></ul><p>Given <strong>a stage</strong>, the efficiency of a stage is the ratio between</p><ol><li><strong>Used CPU time</strong>: <br>total CPU time of “actual work”<br>across all the tasks of the stage</li><li><strong>Available CPU time</strong>: <br>total CPU time (idle or busy) <br>across all cluster nodes during the stage submission</li></ol><h3>Well, that was simple, right?</h3><p>No.</p><h3>Parallel stages</h3><p>Onto the cluster, multiple stages can be submitted at the same time. So you can’t really compute the efficiency of a stage by itself, because in the meantime the cluster could be executing other stages.</p><p>We need to change the definition of efficiency.</p><p>Given <strong>a time interval</strong>, the efficiency of the time interval is the ratio between</p><ol><li><strong>Used CPU time</strong>: <br>total CPU time of “actual work”<br>a̶c̶r̶o̶s̶s̶ ̶a̶l̶l̶ ̶t̶h̶e̶ ̶t̶a̶s̶k̶s̶ ̶o̶f̶ ̶t̶h̶e̶ ̶s̶t̶a̶g̶e̶ <br>across all the tasks across all the stages submitted in that time interval</li><li><strong>Available CPU time</strong>: <br>total CPU time (idle or busy) <br> ̶a̶c̶r̶o̶s̶s̶ ̶a̶l̶l̶ ̶c̶l̶u̶s̶t̶e̶r̶ ̶n̶o̶d̶e̶s̶ ̶d̶u̶r̶i̶n̶g̶ ̶t̶h̶e̶ ̶s̶t̶a̶g̶e̶ ̶s̶u̶b̶m̶i̶s̶s̶i̶o̶n̶<br>across all cluster nodes in that time interval</li></ol><h3>Which time interval? Splitting tasks</h3><p>You may think stage boundaries may be enough, but tasks can run across those boundaries. In the following diagram, notice task 1 running across the boundary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*bUmdecCuePEdRQqII-oytQ.png" /></figure><p>How do you split task metrics in two?</p><p>For task 1, compound information regarding CPU usage is reported. As an example, the following situations are reported equivalently as</p><ul><li>the task ran for 10 seconds</li><li>the task used the CPU for 4 seconds</li></ul><pre>{<br>  &quot;Event&quot;:&quot;SparkListenerTaskEnd&quot;,<br>  &quot;Stage ID&quot;:0,<br>  &quot;Task Info&quot;: {<br>    &quot;Task ID&quot;:0,<br>    &quot;Task Metrics&quot;: {<br>      &quot;Executor CPU Time&quot;: 4000000000 (nanoseconds)<br>    }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*PKWJN1i6ZtNYJkbN3ScJgw.png" /></figure><p>The simplest solution for splitting the task is splitting CPU usage and the other metrics proportionally with respect to the resulting duration of the split (technically speaking, we assume a uniform distribution of probability across the interval).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*lJGfHxJk2mZeFsHrrEtv4Q.png" /></figure><p>Notice that this approximation may create artifacts, e.g. going above 100%.</p><h3>The Plotly figure</h3><p>I used the Python library <a href="https://plotly.com/python/">Plotly</a>, so easy to enable you to streamline a simple visualization like this one, providing a lightweight and interactive interface.</p><p>Notice that the visualization has an improvement over the time intervals discussed above. In fact, the top bar chart further splits the time intervals identifying when the first and last task of the stage has actually started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5crWX0US9mFTBA_n-cg1cg.png" /></figure><h3>A real-world scenario: why is the client’s pipeline slow?</h3><p>A client of Data Reply was expressing concerns regarding the performance of a Spark application running on AWS Glue. Glue can provide some metrics on the performance, but we wanted to go a little deeper, identify quickly where the problem was and iterate rapidly.</p><p>With the following parameters, we were able to output the Spark event log to an S3 bucket of our choice.</p><pre>--conf spark.eventLog.enabled=true<br>--conf spark.eventLog.dir=s3://bucket_prod/spark_event_logs/<br>--conf spark.history.fs.logDirectory=s3://bucket_prod/spark_event_logs/</pre><p>Saving the file locally and running spark-sight on it, the following situation was depicted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muiE68NQfkr-D91v9rSR3w.png" /></figure><h4>Top chart: drops in CPU usage due to faulty repartitions</h4><p>We noticed that, in the chart at the top, a frequent drop of performance was happening in some stages. Notice, as an example,</p><ul><li>stage 150 achieves a healthy &gt; 80% of efficiency</li><li>stage 165 drops to 1% of efficiency</li></ul><p>Thanks to further investigation of the Spark code, we found the faulty calls to df.repartition(1) that were causing the application to use a single CPU of the cluster.</p><h4>Middle chart first half: executors are too small</h4><p>We noticed, in the first half of the middle chart, that spilling was happening</p><ul><li>for all executors</li><li>in equal amounts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*ADQX32N4gJgtlphFG3LPFQ.gif" /></figure><p>This is a clear indication that the parameter <strong>spark.executor.memory</strong> is too low: in fact, all executors are equally struggling to keep in memory their partitions. Finetuning the parameter and iterating quickly solved the issue.</p><h4>Middle chart second half: skewed data</h4><p>We noticed, in the second half of the middle chart, that spilling was happening at a higher intensity for some executors. Notice, as an example, that during stage 180, executor 7 has spilled three times more than the other executors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*akQ_h5BcwNP90h_7-VpmBA.gif" /></figure><p>This is a clear indication of skewed data: executors are assigned partitions of different sizes, so some of them will spill more and others less.</p><p>This intuition, built quickly and intuitively by spark-sight, was confirmed upon further investigation inside the Spark UI, where stage 180 shows the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*MWCiTL2ThfBDrarDXD-s3g.png" /></figure><h3>Conclusions</h3><p>In conclusion, we have highlighted how monitoring the performance of a Spark application can be a challenging task, particularly when dealing with complex and intricate workflows.</p><p>However, the Python package <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is a less detailed, more intuitive representation of what is going on inside your Spark application in terms of performance, and it conveys intuitively and effectively information that would take hours to reconstruct from other tools such as the Spark UI.</p><p>The simplicity of <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a><strong> </strong>can enable rapid iteration on bottleneck identification and finetuning, solving real-world scenarios and increasing efficiency with very little effort.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b100a755538c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-reply-it-datatech/meet-spark-sight-spark-performance-at-a-glance-b100a755538c">Meet “spark-sight”: Spark Performance at a Glance</a> was originally published in <a href="https://medium.com/data-reply-it-datatech">Data Reply IT | DataTech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineering Meets Full-Stack Development: How To Be A Full-Stack Engineer]]></title>
            <link>https://medium.com/data-reply-it-datatech/data-engineering-meets-full-stack-development-how-to-be-a-full-stack-engineer-6f3184b4dda9?source=rss-37332331be9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f3184b4dda9</guid>
            <dc:creator><![CDATA[Alfredo Fomitchenko]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 07:54:33 GMT</pubDate>
            <atom:updated>2023-06-13T07:54:33.516Z</atom:updated>
            <content:encoded><![CDATA[<h4>Showcasing full-stack engineering applied to personalized learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z1yTaifZFE_G1Wu3" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Data engineering is revolutionizing the way we approach learning and skill development. Finding relevant and specific information is extremely important when professionals and enthusiasts want to improve their knowledge and skills in various disciplines.</p><p>While being an expert in a specific field is always seen as a valuable asset, being able to integrate a diverse range of technologies is what really opens up new opportunities for growth and innovation.</p><p>In this article, we explore a case study demonstrating the power of a big-picture approach, combining data engineering and full-stack development, aimed at creating a personalized learning experience for the world of chess.</p><h3>A little bit of background</h3><p>Chess may seem like a game of strategy and intuition, but it is also a game of data. Every move can be analyzed and categorized, and studying theory and past games can give players a significant advantage.</p><p>I got into chess in 2020, with the Netflix show “The Queen’s Gambit” and Chess.com “PogChamps 3” tournament. In the beginning, I started playing casually, and I didn’t think much of the game. But after a while, I realized how much fun I was having, and how much I wanted to learn more, preferably in a structured and planned way.</p><p>I started to look for courses on chess openings. A chess opening is the set of moves, the first 10 to 15, that you memorize in order to have an advantage right from the beginning of the game. The chess opening theory is especially important at the beginning of the game, because the pieces always start in the same place, and the first moves have been studied for centuries throughout history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SugQVDozlaEUZVz8" /><figcaption>Photo by <a href="https://unsplash.com/es/@cristina_gottardi?utm_source=medium&amp;utm_medium=referral">Cristina Gottardi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I found the website Chessable.com, where many Grandmasters (the highest title a person can earn in the chess world) publish their own courses, and I bought the course “The Iron English: Botvinnik Variation”, where Grandmaster Simon Williams teaches the English opening 1. c4. The chess theory is so complex that this single course is composed of hundreds of lines, so I quickly became overwhelmed: how could I know where to start studying?</p><p>I noticed that indeed there was a way to know where to start: I could</p><ul><li>download all the past games I played on Chess.com</li><li>find some software that could import the games</li><li>go through all the opening lines my opponents played against me</li><li>go through the opening course and find the opening lines from which to start studying</li></ul><p>It goes without saying that this would have been tedious, time-consuming, and definitely not very fun. Instead, I quickly realized that this was an engineering problem that a web app could solve, and I’m very much used to engineering problems: I’m a Data Engineer, and my passion for programming dates back to when I was about ten years old and I was reading books about HTML and developing some simple websites with that good-old 2000’s fashion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4zJn0u_9t42RhIVO" /><figcaption>Photo by <a href="https://unsplash.com/@kobuagency?utm_source=medium&amp;utm_medium=referral">KOBU Agency</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So this is how I found myself eager to build a web app that would help me study the lines of the course I bought, while also making it fun and smarter than what Chessable.com currently offers, which is presenting the lines to train in random order.</p><h3>Objectives</h3><p>The first objective of the project was to build a web app where</p><ul><li>a chessboard would be displayed in the front-end</li><li>I could drag and drop the pieces</li><li>the move I made on the board would be sent to the back-end</li><li>the back-end would return a move that my opponent would play on the board against me</li></ul><p>The second objective was to instruct the back-end on</p><ul><li>checking whether the move I made on the board was the best move according to the chess opening course I bought</li><li>playing against me the lines of the course sorted by popularity on Chess.com, instead of presenting them in random order</li></ul><h3>Laying out the front-end: HTML and Javascript</h3><p>For the chessboard to display, I integrated the library <a href="https://justinfagnani.github.io/chessboard-element/">chessboard-element.js</a>, a more compact and modern way of showing a chessboard with pure HTML compared to previous hybrid HTML + Javascript approaches such as <a href="https://www.chessboardjs.com/">chessboard.js</a>.</p><p>Thanks to this choice, I had a fully-functioning chessboard in my simple front-end in a couple of days: dragging and dropping pieces followed all chess rules, reversing the board perspective from white to black and vice-versa was as easy as changing an attribute, and I would add some Javascript code that leveraged another library (<a href="https://github.com/jhlywa/chess.js/">chess.js</a>) to track what moves were being made on the chessboard HTML element.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*_YdogQV7azjPvQh87PTO2A.gif" /></figure><h3>Laying out the back-end: Django</h3><p>Since I’m fluent in Python, the obvious choice for the back-end was <a href="https://www.djangoproject.com/">Django, the web framework for perfectionists with deadlines</a>, an incredibly versatile and easy-to-learn tool for building web apps with very little time and effort.</p><p>Setting up a new Django project is as easy as</p><pre>pip install django<br>django-admin startproject mysite<br>python manage.py runserver</pre><p>and the back-end was already up and running.</p><h3>Connecting front-end and back-end: Redis, Channels</h3><p>The basic scaffoldings of front-end and back-end had been set up, but how to connect the two? And more importantly, how to make moves on the chessboard go back and forth between front-end and back-end as fast as possible?</p><p>Inspired by <a href="https://github.com/kevinjoseph61/chess">this project</a>, the simplest way is using <a href="https://redis.io/">Redis</a>, an in-memory key-value store capable of sub-millisecond latencies. On my computer, I spun up a Docker container with Redis via docker-compose:</p><pre>version: &#39;3.1&#39;<br>services:<br>  redis:<br>    image: redis:5<br>    ports:<br>      - &quot;6379:6379&quot;</pre><p>Redis can be integrated with Django with the Python packages channels and channels-redis:</p><pre>pip install channels channels-redis</pre><p>channels provides the Consumer interface, which serves as a basis for the implementation of a web socket for sending and receiving messages between the front-end and the back-end. For the implementation, I extended the JsonWebsocketConsumer:</p><pre>from channels.generic.websocket import JsonWebsocketConsumer</pre><pre>class SingleConsumer(JsonWebsocketConsumer):<br>    def connect(self):<br>        self.accept()<br>        self.send_json(...)</pre><pre>    def receive_json(self, content):<br>        &quot;&quot;&quot;A move has been made on the front-end chessboard&quot;&quot;&quot;<br>        ...<br>        <br>        self.send_json(content)</pre><pre>    def disconnect(self):<br>        ...</pre><p>The Consumer would be started by Django when the app was launched, and would listen for incoming JSON messages on the Redis channel configured in Django’s settings.py:</p><pre>CHANNEL_LAYERS = {<br>    &quot;default&quot;: {<br>        &quot;BACKEND&quot;: &quot;channels_redis.core.RedisChannelLayer&quot;,<br>        &quot;CONFIG&quot;: {<br>            &quot;hosts&quot;: [(&quot;localhost&quot;, 6379)],<br>        },<br>    },<br>}</pre><p>The last piece of the puzzle was obviously going back to the Javascript code in the front-end Django template and adding some more code to receive the JSON message from the back-end and play the move on the chessboard.</p><h3>Full-stack communication is established, what’s next?</h3><p>So far so good: I had a fully-functioning web app with the following architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ivv1hQZoo9VI2Oj2pIRotA.png" /></figure><p>The web-app could</p><ul><li>display a chessboard</li><li>send my move to the back-end as a JSON message</li><li>send back a response with the move to play</li><li>play the move on the chessboard</li></ul><p>The next step was obvious: I could play any move I wanted, and the back-end could respond with as bad a move as it wanted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*lgHUf6k86l89Sux9sLO9bA.gif" /></figure><p>How to make the back-end play the Chessable.com course lines against me?</p><h3>Exporting the Chessable course</h3><p>To export the lines of the opening course, I needed to</p><ol><li>export all the lines of the course to files (called <a href="https://en.wikipedia.org/wiki/Portable_Game_Notation">PGN</a>)</li><li>store the lines in a database</li><li>make the back-end look in the database if the move I played in the front-end was the best one according to the course</li><li>if it was, the back-end would respond with the opponent’s move present in the course</li><li>if it wasn’t, the back-end would send a message to the front-end to rewind my move and let me retry</li></ol><p>Of course, Chessable doesn’t let you easily export all the lines out of fear of piracy. However, I bought and own the content, so I am legally entitled to “private copies for format shifting or backup”, a right protected by the <a href="https://www.europarl.europa.eu/doceo/document/TA-7-2014-0179_EN.html">European Parliament resolution of 27 February 2014 on private copying levies (2013/2114(INI))</a>.</p><p>Leveraging the experience that I gained in one of my past lives as a Robotic Process Automation developer, I wrote a scraper using the Python framework <a href="https://www.selenium.dev/">Selenium</a>, which automates web browser operations. With it, I launched a script and impersonated myself going through each chapter in the interface of Chessable.com, while recording the moves and writing them to files.</p><h3>Lines database: MongoDB</h3><p>What database and formats to choose for the back-end to retrieve the lines? As a Data Engineer at heart, this was the most fun part of the project.</p><p>In the beginning, I used a graph database with <a href="https://neo4j.com/">Neo4j</a>, by modeling chessboard positions as nodes and moves as edges. However, I soon realized how low the performance of writing and retrieving lines as graph trees was.</p><p>Instead, I needed a low-latency online database, capable of millisecond latencies to allow the web app to be as responsive as possible, so I opted for the document database <a href="https://www.mongodb.com/">MongoDB</a>. I added new Docker containers to the docker-compose:</p><ul><li>a MongoDB instance with a volume bind to persist the data locally</li><li>a <a href="https://github.com/mongo-express/mongo-express">Mongo Express</a> instance to visualize and debug the data during the process</li></ul><pre>mongo:<br>    image: mongo:4.4.1<br>    ports:<br>      - &quot;27017:27017&quot;<br>    volumes:<br>      - &quot;C:/...:/data/db&quot;</pre><pre>  mongo-express:<br>    depends_on:<br>      - mongo<br>    image: mongo-express:0.54.0<br>    ports:<br>      - &quot;8081:8081&quot;<br>    volumes:<br>      - &quot;C:/.../wait-for-it.sh:/wait-for-it/wait-for-it.sh&quot;<br>    command: sh -c &#39;/wait-for-it/wait-for-it.sh mongo:27017 -- node app&#39;</pre><p>The format for storing the lines is tightly coupled with the nature of document-based databases: each item must be a self-contained collection of information, without needing to be joined with other items. For this reason, I constructed each item to be a snapshot of the chessboard, extended to include the previous and the following move, with the following fields</p><ul><li>move number</li><li>move (in <a href="https://en.wikipedia.org/wiki/Universal_Chess_Interface">UCI</a> notation)</li><li>the position before the move was made (in <a href="https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation">FEN</a> notation)</li><li>the position after the move was made (in <a href="https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation">FEN</a> notation)</li><li>the comment of the Grandmaster explaining why the move is the best one</li></ul><p>I used the Python package <a href="https://python-chess.readthedocs.io/">python-chess</a> to parse each file and pre-process it before ingesting it into MongoDB with <a href="https://pymongo.readthedocs.io/">pymongo</a>.</p><h3>The web app knows about course lines, what’s next?</h3><p>After adding some more Javascript code in the Django template, the web app was now capable of</p><ul><li>checking whether the move I played was the best one according to the course</li><li>letting me retry if it wasn’t</li><li>sending back to the back-end what the opponent’s best move was according to the course</li></ul><p>At this point, the architecture looked as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNDNuVlfiKub_iszaLdN9w.png" /></figure><p>All in all, what I had implemented was the same experience of using Chessable.com: having a chessboard to interact with the course lines, reading the Grandmaster comments displayed on the side, and training lines presented in random order. The most important part was missing: how to let me train the most popular lines among my opponents on Chess.com? Why spend hours studying all the lines at random, given that I would probably never face a large percentage of them?</p><p>What was missing was the ability for the web app to</p><ol><li>retrieve my games from Chess.com</li><li>create some statistics about the opening moves played by my opponents</li><li>use those statistics to sort the course lines by the popularity among my opponents, and make me study the most popular lines first</li></ol><h3>Retrieving my games from Chess.com</h3><p>Games on Chess.com can be retrieved via the <a href="https://www.chess.com/news/view/published-data-api">REST API</a><br>at the following endpoint:</p><pre><a href="https://api.chess.com/pub/player/%7Buser_name%7D/games/archives/">https://api.chess.com/pub/player/{user_name}/games/archives/</a></pre><p>A simple Python script using <a href="https://requests.readthedocs.io/">requests</a> and extracting the field pgn from the JSON response sufficed for downloading all the games in PGN format.</p><pre>url = f&#39;https://api.chess.com/pub/player/{user_name}/games/archives/&#39;</pre><pre>archives = (<br>    requests.get(url)<br>    .json()<br>    [&quot;archives&quot;]<br>)</pre><pre>for archive in archives:<br>    json_data = get_from_cache_or_download(<br>        archive=archive,<br>    )<br>    <br>    for game in json_data[&#39;games&#39;]:<br>        (<br>            output_path<br>            .open(&quot;w&quot;)<br>            .write(game[&quot;pgn&quot;])<br>        )</pre><h3>From games to statistics about openings</h3><p>Now that I had all my games in files, I needed to extract some statistics regarding the opening moves played by my opponents.</p><p>I was facing another Data Engineering challenge: I was using a document database such as MongoDB, which is not the right tool for performing analysis and aggregations. This kind of task, theoretically possible with MongoDB, is highly discouraged because it is not natively supported by the database, hence it must be implemented separately, increasing the overall complexity of the web app.</p><p>Instead, I opted for a more natural approach: integrating yet another Docker container in the architecture with a more suitable relational database for aggregations. I chose a <a href="https://www.postgresql.org/">PostgreSQL</a> instance.</p><pre>postgres-service:<br>    image: postgres:15.1<br>    ports:<br>      - &quot;5432:5432&quot;<br>    volumes:<br>      - &quot;C:/.../pg_hba.conf:/etc/postgresql/pg_hba.conf&quot;<br>      - &quot;C:/.../initdb-user.sh:/docker-entrypoint-initdb.d/initdb-user.sh:ro&quot;</pre><p>For this new target, I created two ingestion flows with the Python package <a href="https://www.psycopg.org/">psycopg2</a>:</p><ol><li>a flow to ingest into PostgreSQL the downloaded games;</li><li>a flow to ingest into PostgreSQL the course lines already being ingested into MongoDB, by reusing the expensive parsing of the PGN files performed by the Python package python-chess. These course lines would be eventually joined with and sorted by the analysis performed on the games.</li></ol><p>Now that I had games and course lines in PostgreSQL, I needed to come up with an idea for extracting statistics about the popularity of openings among my opponents.</p><p>This was the idea. Let’s think of a course line as a sequence of positions. For each position, I could assign a numerical value based on</p><ol><li>how many games that position occurred in;</li><li>weighing more if a position was deeper in the line, e.g. arising after 15 moves instead of 3. The reasoning is simple: the deeper the position, the more difficult it is to recall, because many pieces are scattered around the chessboard;</li><li>weighing less the games that happened further in the past. This is logical: the less recent a game, the more different my rating on Chess.com, so the more different the opponent’s mindset and preparation.</li></ol><p>Then, the overall popularity of a single course line was the sum of all the values assigned for each position of the line.</p><h3>The final product</h3><p>After this exciting journey full of challenges, here’s the final architecture of the web app:</p><ul><li>Front-end developed with HTML, CSS, Javascript</li><li>Back-end based on the Django framework, communicating with the front-end via a Redis consumer</li><li>Data architecture composed of unstructured data with MongoDB and structured data with Postgres</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2daIJgtsWHdBmDXvbccH7Q.png" /></figure><p>The original objectives have been accomplished:</p><ul><li>a fully-functioning web app with a chessboard where to try opening moves</li><li>a back-end capable of sorting the lines to show based on the popularity in past games</li><li>a data architecture enabling both fast user experience and offline analysis</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*TfjYALlr8IyRe3nDTaea2w.gif" /></figure><h3>Conclusion</h3><p>This case study showcases how powerful it is to have a big-picture approach to problem-solving, combining diverse disciplines such as data engineering and full-stack development. The initial idea of personalized learning was pretty straightforward but creating a comprehensive solution entailed planning long-term and integrating a diverse range of technologies.</p><p>As engineers, we are expected to solve complex real-world problems with sophisticated solutions by connecting a wide range of systems, and personalized learning becomes ever more important in this context and many fields. The success of this project highlights the potential for innovative solutions when multiple areas of expertise collaborate towards a common goal.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f3184b4dda9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-reply-it-datatech/data-engineering-meets-full-stack-development-how-to-be-a-full-stack-engineer-6f3184b4dda9">Data Engineering Meets Full-Stack Development: How To Be A Full-Stack Engineer</a> was originally published in <a href="https://medium.com/data-reply-it-datatech">Data Reply IT | DataTech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn Chess Openings That Matter To YOU]]></title>
            <link>https://medium.com/getting-into-chess/learn-chess-openings-that-matter-to-you-1da94ebf460b?source=rss-37332331be9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1da94ebf460b</guid>
            <category><![CDATA[chess-openings]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[improvement]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[chess]]></category>
            <dc:creator><![CDATA[Alfredo Fomitchenko]]></dc:creator>
            <pubDate>Sat, 21 Jan 2023 08:15:45 GMT</pubDate>
            <atom:updated>2023-01-21T09:43:22.812Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/getting-into-chess/tagged/improvement">Improvement</a> | <a href="https://medium.com/getting-into-chess/tagged/chess-openings">Chess Openings</a> | <a href="https://medium.com/getting-into-chess/tagged/apps">Apps</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z1yTaifZFE_G1Wu3" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Welcome to the world of chess</h3><p>So you bought an opening course on <a href="https://chessable.com/">Chessable.com</a> or <a href="https://chessly.com/">Chessly.com</a>, and you are getting excited about <strong>crashing </strong>all the 1100s who come knocking at your door on <a href="https://chess.com/">Chess.com</a> or <a href="https://lichess.org/">Lichess.org</a>.</p><p>The course has 400 lines, so you simply start from chapter one where the opponent replies with the King’s Indian. You spend hours reading the GM comments, understanding concepts, memorizing positions.</p><blockquote>You are ready now.</blockquote><p>You sit down on Chess.com or Lichess.org, and next thing you know 3 hours have passed, only <strong>2 people out of 65</strong> have played the King’s Indian, and in 63 games you were completely on your own by move 3.</p><p>You have two choices now.</p><h3>First choice: spiritual journey</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hsh6kSrzXmCgK_S5" /><figcaption>Photo by <a href="https://unsplash.com/@zacdurant?utm_source=medium&amp;utm_medium=referral">Zac Durant</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><ol><li>Download all 65 games you played</li><li>Find some software that can import the games and figure out how to use the software</li><li>Start another spiritual journey where you go through all the variations your opponents played, and figure out <strong>what the average 1100 plays against you</strong></li><li>Start another spiritual journey where you go through your opening course to find where the GM was talking about those lines</li></ol><h3>Second choice: I didn’t even like chess in the first place</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oRwZBQeRE1kNMibJ" /><figcaption>Photo by <a href="https://unsplash.com/@notso?utm_source=medium&amp;utm_medium=referral">Christopher Ott</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s face it: you started playing chess just for the rush of it. You don’t want to spend hours finding the <strong>lines you should study</strong>.</p><p>You accept that you financed some GM, then go back to playing bullet and losing 200 ELO in one sitting.</p><h3>There must be a better way</h3><p>What if I told you that I am developing a tool that…</p><ol><li>can import your Chess.com or Lichess.org games to figure out <strong>what the average 1100 plays against you</strong></li><li>can import any opening course you bought and let you train the <strong>lines YOU should study</strong>, leaving behind all the lines you will never face in your 1100 (soon-to-be 1600) career</li></ol><h3>Nah, I don’t believe you</h3><p>Let me show you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*TfjYALlr8IyRe3nDTaea2w.gif" /></figure><p>This web app I’m developing…</p><ul><li>Keeps responding to my English opening with <strong>1… e5</strong> and <strong>2… Nf6</strong>, because the majority of games I played on Chess.com was 1. c4 e5 2. Nc3 Nf6 (<em>1100s don’t know any better</em>)</li><li>Lets me retry the move when I incorrectly play <strong>4. a3</strong> to attack the bishop (<em>I see bishop, I attack bishop</em>)</li><li>In the background, it <strong>keeps track of the time I spend</strong> on each move to analyze the performance</li></ul><h3>Is it working for you?</h3><p>Thanks to this app, I’m putting in a lot of focused training, and I’m not lost at move 3 anymore.</p><blockquote>I have been crashing in Blitz ever since.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*7IjVzkcr8dUGo95O5NGwmA.png" /></figure><h3>Nice, can I have it?</h3><p>The app is under active development, and transforming it into a final product for everyone takes time and effort.</p><blockquote>If enough people like and share this post, I will put time and effort into open-sourcing the app.</blockquote><p>If you want access to the app and crash your opponents, subscribe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*S9TFWkLOd9gzzZ9_-kj7OA.png" /></figure><p>Clap the story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/1*KaEKy95FytsWHsrHsy_LuQ.gif" /></figure><p>And share it with someone who loves chess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*pjJiaQrK5iERDKxfnPUNgA.gif" /></figure><h3>Can you at least tell me how you did that?</h3><p>This gets very technical very quickly, so bear with me.</p><h3>Downloading all your games</h3><p>My Chess.com username is <em>alfredo-f</em>.</p><p>This script downloads all the games I played since 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*unTTVGgmNU39ccboIThgfw.png" /></figure><p>Here are all my games.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*5c6eAIYkIlwtlgOsM0imNg.png" /></figure><h3>Downloading the opening course you bought</h3><p>Last year on Chessable.com I bought the wonderful course <em>The Iron English: Botvinnik Variation</em> by the wonderful GM Simon Williams.</p><p>As you can see, progress has been very low.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gCOdBkFXQTkN_tZpeaPkYw.png" /></figure><h3>A chessboard to rule them all</h3><p>The interface consists of a Javascript chessboard based on the same library used by Chess.com and Lichess.org.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*-3Z3evfGPhZEMV0x5-67gA.gif" /></figure><p>It shows…</p><ul><li>buttons to control the timer tracking my moves. I can add any button, “the 64 squares are the limit”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*yvaMjgxuH7Sy0QImiNW4sQ.gif" /></figure><ul><li>sections for the move comments</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*vNbcBO6ycMKgGL0P3aFa9A.gif" /></figure><h3>Where the magic happens</h3><p>The Python backend is based on Django.</p><p>Games and opening lines are ingested into MongoDB and Postgres databases, and opening lines are assigned an importance score based on the games you played</p><ul><li>how frequently they have appeared</li><li>how deeply they have appeared (e.g. many opponents have prep until move 10)</li></ul><h3>Ok, can I have it?</h3><blockquote>If enough people like and share this post, I will put time and effort into open-sourcing the app.</blockquote><p>If you want to crash opponents, clap the story and share it with someone who loves chess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*S9TFWkLOd9gzzZ9_-kj7OA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/109/1*KaEKy95FytsWHsrHsy_LuQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*pjJiaQrK5iERDKxfnPUNgA.gif" /></figure><h3>Before you leave, can you share more details?</h3><p>If you like software and you want me to go into details about the architecture I developed, let me know in the comments. There’s so much meat to get into:</p><ul><li>Frontend, backend: <strong>Python, HTML, CSS, Javascript</strong></li><li>Frontend-backend communication: <strong>Redis</strong></li><li>Graph modeling: <strong>Neo4j </strong>(ditched for performance issues)</li><li>Persisting data: <strong>MongoDB, Postgres</strong></li><li>Orchestration: <strong>Docker, Docker Compose</strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1da94ebf460b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/getting-into-chess/learn-chess-openings-that-matter-to-you-1da94ebf460b">Learn Chess Openings That Matter To YOU</a> was originally published in <a href="https://medium.com/getting-into-chess">Getting Into Chess</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“spark-sight” Shows Spill: Skewed Data and Executor Memory]]></title>
            <link>https://medium.com/@alfredo.fomitchenko/spark-sight-shows-spill-skewed-data-and-executor-memory-9169b79ca66e?source=rss-37332331be9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9169b79ca66e</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Alfredo Fomitchenko]]></dc:creator>
            <pubDate>Tue, 07 Jun 2022 07:21:55 GMT</pubDate>
            <atom:updated>2022-06-13T08:23:44.165Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5IJLgrmOnmQ4HSDNl87Sdw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@thomholmes">Thom Holmes</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is a less detailed, more intuitive representation of what is going on inside your Spark application in terms of performance.</p><p>This story is part of a series:</p><ol><li>Part 1: <a href="https://betterprogramming.pub/spark-sight-spark-performance-at-a-glance-c2316d2a251b"><strong>Meet “spark-sight”: Spark Performance at a Glance</strong></a></li><li>Part 2: This story</li></ol><p>When you launch your Spark application, do you ever wonder</p><ol><li>Whether it suffers from <strong>skewed data</strong>?</li><li>Whether the value of <strong>spark.executor.memory </strong>is right?</li></ol><p>Wonder no more!</p><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> v0.1.8 adds a new chart for spill information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-Fm_2PMi6eQRBPG7j8OkQ.gif" /></figure><p>The middle chart shows, for each executor, when and how much the executor has spilled to disk in that time interval.</p><p>And maybe, with the help of <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a>, one day you will get to see this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*vbGHnoQyh9PU84HNrbDP7Q.png" /></figure><h3>Ok, I need it NOW</h3><pre>$ pip install spark-sight&gt;=0.1.8</pre><h3>Why care about spill?</h3><p>I’ll let this wonderful story guide you</p><p><a href="https://michaelheil.medium.com/understanding-common-performance-issues-in-apache-spark-deep-dive-data-spill-7cdba81e697e"><strong>Understanding common Performance Issues in Apache Spark — Deep Dive: Data Spill</strong></a></p><p>Simply put, when one of the partitions is too large to fit in the executor memory, the disk is accessed and performance decreases by orders of magnitude.</p><h3>A real-world scenario</h3><p>This chart is so helpful because it gives you a lot of information at a glance.</p><p>To illustrate this, the following is a real-world Spark application I analyzed with <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muiE68NQfkr-D91v9rSR3w.png" /></figure><h4>First half: <strong>spark.executor.memory is too low</strong></h4><p>First half of the spill chart shows that</p><ul><li><strong>all </strong>executors are spilling</li><li>the executors are spilling <strong>equal</strong> amounts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*ADQX32N4gJgtlphFG3LPFQ.gif" /></figure><p>This is a clear indication that the parameter spark.executor.memory is too low: in fact, all executors are equally struggling to keep in memory their partitions.</p><h4>Second half: skewed data</h4><p>Second half of the spill chart shows that</p><ul><li>the executors are spilling <strong>different</strong> amounts</li></ul><p>For example, during stage 180, executor 7 has spilled three times more than the other executors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*akQ_h5BcwNP90h_7-VpmBA.gif" /></figure><p>This is a clear indication of skewed data. In fact, if you open stage 180 in the Spark UI, you will see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*MWCiTL2ThfBDrarDXD-s3g.png" /></figure><h3>What’s next?</h3><p>Let me know in the comments what I should address next:</p><ul><li>Adding a chart for peak memory usage per executor</li><li>Converting the current figure into a full-fledged Dash (Plotly) application to improve the UX</li></ul><h3>What’s next for you?</h3><ul><li>Clap this story</li><li>Read part 1 of this series: <a href="https://betterprogramming.pub/spark-sight-spark-performance-at-a-glance-c2316d2a251b"><strong>Meet “spark-sight”: Spark Performance at a Glance</strong></a></li><li>Follow me here on Medium for future stories</li><li>If you find this project useful, head over to the <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight Github repo</strong></a> and don’t be gentle on the star and watch buttons</li><li>If you encounter any problems, head over to the <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight Github repo</strong></a> and don’t be gentle on the issue button</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9169b79ca66e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet “spark-sight”: Spark Performance at a Glance]]></title>
            <link>https://medium.com/data-reply-it-datatech/spark-sight-spark-performance-at-a-glance-c2316d2a251b?source=rss-37332331be9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2316d2a251b</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Alfredo Fomitchenko]]></dc:creator>
            <pubDate>Sat, 07 May 2022 08:25:40 GMT</pubDate>
            <atom:updated>2022-06-07T07:24:16.330Z</atom:updated>
            <content:encoded><![CDATA[<h4>An open-source project I created</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AWOVtzDgZmB3YEaI" /><figcaption>Photo by <a href="https://unsplash.com/@jeztimms?utm_source=medium&amp;utm_medium=referral">Jez Timms</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is a less detailed, more intuitive representation of what is going on inside your Spark application in terms of performance.</p><p>This story is part of a series:</p><ol><li>Part 1: This story</li><li>Part 2: <a href="https://medium.com/@alfredo.fomitchenko/spark-sight-shows-spill-skewed-data-and-executor-memory-9169b79ca66e"><strong>“spark-sight” Shows Spill: Skewed Data and Executor Memory</strong></a></li></ol><p>I love Spark, but I don’t love the Spark UI.</p><p>This is why, at first, I was excited to find out that <a href="https://www.datamechanics.co/">Data Mechanics</a> was developing <a href="https://github.com/datamechanics/delight">Delight</a>, sharing with the world</p><ul><li>an open-source agent collecting information from within the Spark application;</li><li>a <strong>closed-source </strong>dashboard displaying the performance of a Spark application in terms of CPU efficiency and memory usage.</li></ul><p>Even though the service is free of charge,</p><ol><li>you need to integrate into your Spark application their custom listener that collects and sends out data to their servers;</li><li>it may be difficult for your boss to approve such practice (privacy concerns), or outright unable to (e.g. your application runs in a Glue job inside a VPC without internet access).</li></ol><p>Willing to gather everything I know about Spark, I took on the challenge of recreating the same amazing experience of Delight for everybody to enjoy and contribute to.</p><p>This is why I am sharing with you <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*39I3dsXti75RvO_6Kn6Bcw.gif" /></figure><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is a less detailed, more intuitive representation of what is going on inside your Spark application in terms of performance:</p><ul><li>CPU time spent doing the “actual work”</li><li>CPU time spent doing shuffle reading and writing</li><li>CPU time spent doing serialization and deserialization</li><li>Spill intensity per executor (Out now! Read part 2 of the series: <a href="https://medium.com/@alfredo.fomitchenko/spark-sight-shows-spill-skewed-data-and-executor-memory-9169b79ca66e"><strong>“spark-sight” Shows Spill: Skewed Data and Executor Memory</strong></a>)</li><li>(coming) Memory usage per executor</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3Fempy7aGCVjySrpYh0eQ.gif" /></figure><p><a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight</strong></a> is not meant to replace the Spark UI altogether, rather it provides a bird’s-eye view of the stages allowing you to identify at a glance which portions of the execution may need improvement.</p><h3>Plotly figure</h3><p>The Plotly figure consists of two charts with synced x-axis.</p><h4>Top: efficiency in terms of CPU cores available for tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*GdbtyKadZWcR_erwUpiHLw.gif" /></figure><h4>Bottom: stages timeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*PwR6AP9_DAEDl74PwEHfVA.gif" /></figure><h3>No more talk, let me use it IMMEDIATELY</h3><p>To install it,</p><pre>$ pip install spark-sight</pre><p>To meet it,</p><pre>$ spark-sight --help</pre><p>To launch it,</p><ul><li>Unix</li></ul><pre>$ spark-sight \<br>    --path &quot;/path/to/spark-application-12345&quot; \<br>    --cpus 32 \<br>    --deploy_mode &quot;cluster_mode&quot;</pre><ul><li>Windows PowerShell</li></ul><pre>$ spark-sight `<br>    --path &quot;C:\path\to\spark-application-12345&quot; `<br>    --cpus 32 `<br>    --deploy_mode &quot;cluster_mode&quot;</pre><p>A new browser tab will be opened.</p><p>For more information, head over to the <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight Github repo</strong></a>.</p><h3>Ok, but how did you do this?</h3><p>The Spark event log is a simple text file that Spark is natively able to store somewhere for you to open and look through:</p><pre>--conf spark.eventLog.enabled=true<br>--conf spark.eventLog.dir=file:///c:/somewhere<br>--conf spark.history.fs.logDirectory=file:///c:/somewhere</pre><p>As described in the <a href="https://spark.apache.org/docs/latest/monitoring.html#executor-task-metrics">Spark documentation</a>, hidden somewhere in the text file are the performance data:</p><ul><li>SparkListenerTaskEnd events: for how long the task was shuffling, serializing, deserializing, and doing the “actual work” it was supposed to do in the first place</li><li>SparkListenerStageCompleted events: for how long the corresponding stage was in the submitted state</li></ul><p>Given <strong>a stage</strong>, the efficiency of a stage is the ratio between</p><ol><li><strong>Used CPU time</strong>: <br>total CPU time of “actual work”<br>across all the tasks of the stage</li><li><strong>Available CPU time</strong>: <br>total CPU time (idle or busy) <br>across all cluster nodes during the stage submission</li></ol><h3>Well, that was simple, right?</h3><p>No.</p><h3>Parallel stages</h3><p>Onto the cluster, multiple stages can be submitted at the same time. So you can’t really compute the efficiency of a stage by itself, because in the meantime the cluster could be executing other stages.</p><p>We need to change the definition of efficiency.</p><p>Given <strong>a time interval</strong>, the efficiency of the time interval is the ratio between</p><ol><li><strong>Used CPU time</strong>: <br>total CPU time of “actual work”<br>a̶c̶r̶o̶s̶s̶ ̶a̶l̶l̶ ̶t̶h̶e̶ ̶t̶a̶s̶k̶s̶ ̶o̶f̶ ̶t̶h̶e̶ ̶s̶t̶a̶g̶e̶ <br>across all the tasks across all the stages submitted in that time interval</li><li><strong>Available CPU time</strong>: <br>total CPU time (idle or busy) <br> ̶a̶c̶r̶o̶s̶s̶ ̶a̶l̶l̶ ̶c̶l̶u̶s̶t̶e̶r̶ ̶n̶o̶d̶e̶s̶ ̶d̶u̶r̶i̶n̶g̶ ̶t̶h̶e̶ ̶s̶t̶a̶g̶e̶ ̶s̶u̶b̶m̶i̶s̶s̶i̶o̶n̶<br>across all cluster nodes in that time interval</li></ol><h3>Well, which time interval? Splitting tasks</h3><p>You may think stage boundaries may be enough, but tasks can run across those boundaries. In the following diagram, notice task 1 running across the boundary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*bUmdecCuePEdRQqII-oytQ.png" /></figure><p>How do you split task metrics in two?</p><p>For task 1, compound information regarding CPU usage is reported. As an example, the following situations are reported equivalently as</p><ul><li>the task ran for 10 seconds</li><li>the task used the CPU for 4 seconds</li></ul><pre>{<br>  &quot;Event&quot;:&quot;SparkListenerTaskEnd&quot;,<br>  &quot;Stage ID&quot;:0,<br>  &quot;Task Info&quot;: {<br>    &quot;Task ID&quot;:0,<br>    &quot;Task Metrics&quot;: {<br>      &quot;Executor CPU Time&quot;: <strong>4000000000 (nanoseconds)</strong><br>    }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*PKWJN1i6ZtNYJkbN3ScJgw.png" /></figure><p>The simplest solution for splitting the task is splitting CPU usage and the other metrics proportionally with respect to the resulting duration of the split (technically speaking, we assume a uniform distribution of probability across the interval).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*lJGfHxJk2mZeFsHrrEtv4Q.png" /></figure><p>Notice that this approximation may create artifacts, e.g. going above 100%.</p><h3>The figure</h3><p>I used the Python library <a href="https://plotly.com/python/">Plotly</a>, so easy to enable you to streamline a simple visualization like this one, providing a lightweight and interactive interface.</p><p>Notice that the visualization has an improvement over the time intervals discussed above. In fact, the top bar chart further splits the time intervals identifying when the first and last task of the stage has actually started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5crWX0US9mFTBA_n-cg1cg.png" /></figure><h3>What’s next?</h3><h4>Medium term</h4><p>I plan to add charts for</p><ul><li>Spill intensity per executor (Out now! Read part 2 of the series: <a href="https://medium.com/@alfredo.fomitchenko/spark-sight-shows-spill-skewed-data-and-executor-memory-9169b79ca66e"><strong>“spark-sight” Shows Spill: Skewed Data and Executor Memory</strong></a>)</li><li>Memory usage per executor</li></ul><p>and then convert the simple figure into a full-fledged Dash (Plotly) application to improve the UX.</p><h4>Long term</h4><p>I plan to add</p><ul><li>the ability to read the Spark event log from other data sources (e.g. S3, GCS, Azure Storage, HDFS, …)</li><li>showing multiple Spark applications at the same time so that performance can be compared (e.g. you ran the same application with different spark.sql.shuffle.partitions, spark.executor.memory, …)</li><li>showing the efficiency of multiple Spark applications running on the same cluster at the same time</li><li>taking into account non-static configuration of the cluster (now it is assumed the number of CPUs does not change)</li></ul><h3>What’s next for you?</h3><ul><li>Clap this story</li><li>Read part 2 of this series: <a href="https://medium.com/@alfredo.fomitchenko/spark-sight-shows-spill-skewed-data-and-executor-memory-9169b79ca66e"><strong>“spark-sight” Shows Spill: Skewed Data and Executor Memory</strong></a></li><li>Follow me here on Medium for future stories</li><li>If you find this project useful, head over to the <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight Github repo</strong></a> and don’t be gentle on the star and watch buttons</li><li>If you encounter any problems, head over to the <a href="https://github.com/alfredo-f/spark-sight"><strong>spark-sight Github repo</strong></a> and don’t be gentle on the issue button</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*dDNRqmHtVO239VJDOuC1MA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2316d2a251b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-reply-it-datatech/spark-sight-spark-performance-at-a-glance-c2316d2a251b">Meet “spark-sight”: Spark Performance at a Glance</a> was originally published in <a href="https://medium.com/data-reply-it-datatech">Data Reply IT | DataTech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>