<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sami Konyarov on Medium]]></title>
        <description><![CDATA[Stories by Sami Konyarov on Medium]]></description>
        <link>https://medium.com/@samuilkonyarov?source=rss-c67fce608cd3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8KmYRGSyaOZj1-J0-TtlxA.jpeg</url>
            <title>Stories by Sami Konyarov on Medium</title>
            <link>https://medium.com/@samuilkonyarov?source=rss-c67fce608cd3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:03:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samuilkonyarov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Everyday I am Shufflin’ Shufflin’]]></title>
            <link>https://medium.com/@samuilkonyarov/everyday-i-am-shufflin-shufflin-cc4f3edf654d?source=rss-c67fce608cd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc4f3edf654d</guid>
            <dc:creator><![CDATA[Sami Konyarov]]></dc:creator>
            <pubDate>Sun, 15 Dec 2024 22:02:46 GMT</pubDate>
            <atom:updated>2024-12-15T22:52:12.583Z</atom:updated>
            <content:encoded><![CDATA[<p>Data shuffling is often regarded as the arch enemy of data engineers. But shuffle is actually great — and I’m not being sarcastic. For the most part there is nothing bad about it and you should shuffle, it is there for a reason, and it makes our lives much much easier. On the other hand, shuffle is also the least scalable part of Spark and as the scale grows, the usage of shuffle becomes more painful. Once you get into the tens of TBs per day, you will not be able to get away with shuffling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*j2VhsI8lJSyWSjday8P0aQ.png" /></figure><h4>Quick run through of the article:</h4><ol><li>An overview of shuffling in Spark, its role in distributed computing, and why it is both essential and challenging.</li><li>An exploration of the Spark operations — like aggregations, joins, and sorts — that cause shuffling, and their impact on performance.</li><li>A detailed explanation of how Spark batches data by key and routes it to executors to enable efficient aggregation.</li><li>Methods for minimizing shuffle, including:Using fact arrays for compact data representation, leveraging sorted merge bucketing (SMBs), choosing specific types of joins, such as broadcast and bucket joins.</li><li>How Spark buckets tables?</li></ol><p><strong>1.What Is Shuffle?<br></strong>Shuffle is the bottleneck for parallelism. In a pipeline that does not need/use shuffling, tasks can be pretty much as parallel as you want it to be because no specific chunk of data needs to reside on a particular executor. Parallelism and efficiency go hand in hand, the more parallel the pipelines are, the more effective they are too, and they can crunch more data.</p><p><strong>2.From Fast to Not-So-Fast Operations<br></strong>Shuffling is called upon differently depending on the operations being performed.</p><p><strong>2.1.Select/Filter<br></strong>Selecting columns from a dataset and/or filtering records are nearly infinitely scalable operations. Each executor works independently, without needing to know about the data that is processed on the other executors. These types of jobs do not require a shuffle, making them instant, cost-effective, and highly performant.</p><p><strong>2.2.Aggregations<br></strong>When we get into the realm of aggregations we begin to notice a decline in performance. Operations like GROUP BY, JOINs, and aggregate functions fall under this category. These operations start to form bottlenecks, because all the data for a specific key must be moved to a single executor to produce the correct result. In a hypothetical scenario, where a user has 100 records, and each record is on a different executor. To perform the aggregation, all of the user’s data needs to be moved to the same executor. This process of redistributing specific data to specific machines is what shuffle is.</p><p><strong>2.3.Sort/Order By<br></strong>Sorting data is one of the least parallelisable and worst performant operations in distributed systems. To sort data spread across multiple executors, all data must pass through a single machine. This makes sorting inherently costly and should be used sparingly and at the right time. For example, sorting on already aggregated dataset that has significantly less rows would perform much better than sorting on tens or hundreds of millions of rows, in the latter case the job is going to have a bad time.</p><p>It is important to highlight that sorting as part of a window function does not mean you will be faced with the same performance challenges. I mean, you will if you do not partition or if a ranking function is computed over the entire dataset. Otherwise, sorting within a window function is applied to a much smaller subset of data than the global sort and more importantly all the required data is already on the same executor.</p><p><strong>3.How Does Shuffle Batch Data for Aggregation?<br></strong>The bread and butter of shuffling is batching the data by key and routing it to the appropriate executor. For example, when grouping by device_id, the allocation to a specific partition is determined by dividing the device_id value by the total number of partitions. The record is then routed to the partition corresponding to the remainder. For instance, you there are 5 partitions and device_id of the first record is divided by 5, a remainder of 0 routes the record to partition 1, a remainder of 1 routes it to partition 2, and so on.</p><p>Things get even trickier for a join operation. In such cases, Spark will have to shuffle all files in both tables involved in the join. This effectively doubles the workload, as Spark must align all the keys on the left table with their corresponding keys on the right table. This highlights why shuffle can significantly increase the complexity and cost of distributed processing.</p><p><strong>4.How to reduce shuffle?<br>4.1. Data modelling<br></strong>One of the most effective ways to reduce shuffle is through thoughtful data modeling, particularly by reducing the size of your fact data. This approach isn’t just for big tech companies, even smaller organisations working with large datasets can benefit from reduced fact tables in terms of both cost and performance.</p><p>To implement this, it’s essential to have a deep understanding of the domain, the ways data is used, and the core metrics. This would be a harder challenge for organisations that are still maturing their data products, because reduced fact data is much harder to maintain.Fact tables are very flexible and excel at answering specific, short-term questions, but they struggle when dealing with queries over longer time horizons. These tables store each event as a separate record, resulting in extremely high data volumes.</p><p>A practical next step is to aggregate fact data into a more manageable form. For example, daily aggregation drastically reduces the volume and depending on the use case it could result in shrinking the data by tens, or even hundreds of times. However, when it comes to yearly analysis even daily aggregation can be problematic due to the sheer volume of data involved.</p><p><strong><em>4.1.1 Long-Array Fact Data: The Belle of the Ball in Reduced Facts</em></strong><em><br></em>Long-array fact data is a powerful solution for reducing fact data while retaining essential context. Let me illustrate this with an example. Imagine fact data generated from IoT devices publishing events minutely. The data is aggregated hourly, which is a significant improvement. However, analysing this data on a weekly or monthly basis might be a very expensive and slow task especially with a large fleet of IoT devices.</p><p>The fact data can be reduced further without losing context by implementing array facts. The data is aggregated weekly, resulting in one row per device per week but here comes the catch! The week is modelled as an array with 168 elements, where each element represents one hour of the week. The part that fascinates me the most is that approach allows analysts to retrieve the measurement for any specific hour by using the array index — eliminating the need to sift through thousands of rows.</p><p>To illustrate the scale of the reduction: in its raw format, one device would generate 10,080 rows per week. With array fact data, this shrinks to a single row. The aggregated data is much easier to work with in terms of speed of query and performance. However, it comes with the trade-off of reduced flexibility. Since the data is pre-aggregated, you lose the ability to adjust or reinterpret it on the fly. Array facts are fantastic, very performant and you do not lose the granularity, but they also highlight a major challenge in data modelling: backfilling. Additionally, adjusting aggregation logic invalidates previously calculated data, making revisions or tweaks to such models complex and resource-intensive.</p><p><strong>4.2.Sorted Merge Buckets (SMBs)<br></strong>Another effective way to reduce the shuffling is to use Sorted Merge Buckets (SMBs). SMBs pre-buckets your data based on some high-cardinality field as device_id, and it does the shuffling just once. For all subsequent operations when that aggregate data on that field, Spark skips the shuffling and leverage the buckets. This approach is conceptually similar to bucket joins, which is described in the section below. By reducing the need for repeated shuffling, SMBs can significantly improve performance for high-volume datasets.</p><p><strong>4.3.Consider your JOINs<br></strong>There are three types of JOINs in Spark — shuffle sort-merge join, broadcast hash join, bucket join.</p><ul><li><strong>Shuffle Sort-Merge Join<br></strong>Shuffle sort-merge join is the least performant, but at the same time is also the most versatile as it works in almost all cases. However, its reliance on shuffling makes it slower and more resource-intensive.</li><li><strong>Broadcast Hash Join<br></strong>Broadcast joins are highly performant but work best under specific conditions. Broadcast join works best when one side of the join is small, instead of shuffling data, Spark broadcasts the smaller dataset to all executors. The caveat is that the small side of the join must be truly small. The default value is 10 MB, the executor can tolerate without additional fine tuning up to 1GB, in the range of 1GB to 10GB a broadcast join would still work but it is expected to encounter memory issues.</li><li><strong>Bucket Join<br></strong>Bucket joins are ideal for large datasets where you want to avoid repeated shuffling.While it’s not entirely shuffle-free, with bucket join the dataset is shuffled once during the bucketing. After is placed in the respective bucket, Spark can perform subsequent joins without additional shuffling, making bucket joins particularly effective for large datasets that are used in multiple joins. If you’re only performing a single join or aggregation, bucket joins may not be worth the effort. Their biggest value is when working with big datasets involved in multiple joins, as the upfront cost of bucketing is offset by the repeated performance gains in subsequent operations.</li></ul><p><strong>5. How is Bucketing Done?<br></strong>How to pick number of buckets is probably the big question, unsurprisingly, the answer depends on the volume of your data. The goal is to avoid having too many buckets with very little data, as this leads to inefficiency. Additionally, empty buckets can create further problems. A good rule of thumb is to aim for approximately 10 GB of data per bucket.</p><p>Even if the tables involved in a bucket join don’t have the same number of buckets, the join can still work as long as the bucket counts are multiples of each other. For this reason, it’s best to choose a bucket count that is a power of 2.</p><p>If the data volume is 1TB and you aim for 10GB per bucket, you will need 100 buckets. However, to allow sources of different bucket sizes to still be joinable, you must set the bucket count to a power of 2, which means adjusting it to either: 128 buckets (~8 GB per bucket), or 64 buckets (~15 GB per bucket). This ensures efficient bucket utilization while maintaining flexibility for future joins with datasets that may have different bucket counts.</p><p><strong>Final Words<br></strong>This article took over 10 hours to write, and it still barely scratches the surface of shuffling in Spark. Shuffling is such a rabbit hole, initially I was planning on exploring how it works behind the scenes and then I found myself reading about data modelling and reduced facts. Without a doubt, shuffling is a cornerstone of distributed computing, it is by far one of the most resource-intensive operations and when properly understood and managed correctly, it can unlock significant improvements in cost and performance.</p><p>Please don’t hesitate to leave your feedback, especially if it is constructive I would greatly appreciate it! Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc4f3edf654d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Green Data Engineer — The Evangelist for Sustainability in the data world]]></title>
            <link>https://medium.com/@samuilkonyarov/green-data-engineer-the-evangelist-for-sustainability-in-the-data-world-47abb73560f8?source=rss-c67fce608cd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/47abb73560f8</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[sustainability]]></category>
            <dc:creator><![CDATA[Sami Konyarov]]></dc:creator>
            <pubDate>Sun, 01 Dec 2024 21:02:39 GMT</pubDate>
            <atom:updated>2024-12-01T21:02:39.263Z</atom:updated>
            <content:encoded><![CDATA[<h3>Green Data Engineer — The Evangelist for Sustainability in the Data World</h3><p>I know AI has truly become a part of society when my grandma starts talking to me about it. One major topic of discussion is the extent to which AI will disrupt the IT industry. While you won’t find the answer to that question here, I believe that the increased efficiency brought by AI, will lead to the emergence of a new, widespread role in the data world. I call it the Green Data Engineer, but names like Sustainability Data Engineer, Carbon Data Officer — all of these captures the same essence.</p><p>AI is making us more efficient at writing code, and the expectation is that engineers’ efficiency will grow even further in the future. I believe this shift will allow professionals to focus more on optimising architecture, applying best practices and most importantly, reducing the carbon footprint of their code and queries. At first glance this might seem unnecessary, but the fact that data centres account for more carbon emissions than the aviation industry and consume 2% of the world’s total electricity, this calls for action.</p><figure><img alt="Google Cloud Data Centre" src="https://cdn-images-1.medium.com/max/590/1*GyqXZxNSBcCsKsg_7GIwEg.jpeg" /><figcaption>Google cloud data centre</figcaption></figure><p>The shift in thinking about sustainability has already begun at its source. Cloud vendors are optimising hardware and software to reduce resource consumption in data centres. For example, AWS have released several iterations of their Graviton processor that is known for its lower energy use. Additionally, all major cloud vendors now offer carbon monitoring tools, giving clients insights into the greenhouse gas emissions produced by their processes. Data suggests that AWS infrastructure can be up to four times more efficient than on-premises systems. When workloads are optimised on AWS, the associated carbon footprint can be reduced by as much as 99%. Despite these impressive reductions, brute-force software solutions still contribute significantly to CO2 emissions.</p><blockquote>“The cloud vendor is responsible for the sustainability of the cloud; the engineer is responsible for sustainability in the cloud.”</blockquote><p>AWS’s sustainability mantra highlights the foundation of the Green Data Engineer role: “The cloud vendor is responsible for the sustainability of the cloud; the engineer is responsible for sustainability in the cloud.” One major challenge in addressing an organisation’s carbon impact is the lack of transparency regarding resource usage across instances and tools. In modern data centres, the largest power consumers are memory and CPU, excluding GPUs. For example:</p><p>· An EC2 instance consumes approximately 240 watts per hour, depending on the number of cores, threads, and memory.</p><p>· Memory consumption is roughly 0.4 watts per I/O GB of information and, in some configurations, can exceed CPU power consumption.</p><p>· Storage consumption varies, with SSDs using about 1.2 watts per terabyte-hour and HDDs about 0.65 watts per terabyte-hour.</p><p>These figures emphasize the need for more efficient software and hardware designs, as well as a greater focus on optimising workloads to minimise environmental impact.</p><p>The responsibility of data professionals is twofold: ensuring that their pipelines produce accurate, high-quality data and that they run efficiently. While researching for this article, I stumbled upon the interesting fact that Airbnb processes all transactions since 2016 every time they run their unit economics pipeline — at least once daily. Because the data is accurate and the process does not block downstream workflows, there is no perceived need or motivation to change it. While Airbnb is often criticised due to its stature in the industry, the reality is that almost every data professional has, at some point, implemented a solution that worked, which was not done in the most efficient manner and was never revisited for improvement.</p><p>I hope that both small and large organisations begin to value the refactoring of such processes more. Optimising these pipelines for efficiency might result in cost savings that seem negligible for a high-revenue company, but this type of “recycling” is, in my view, a crucial first step in contributing to sustainability. Another important aspect of this conversation is that reducing the carbon footprint also lowers AWS bills. However, this does not always translate to reduced costs. Refactoring and improving numerous processes — many of which started as quick-and-dirty solutions and were never revisited — requires additional overhead for human work hours with the assistance of AI to improve the code.</p><p>I don’t believe that anyone can predict with accuracy how the tech industry will shape up in the years to come. However, the one thing which is clear is that data professionals must recognise their responsibility of delivering accurate, scalable, high-quality processes that are built with sustainability in mind. While refactoring and improving existing processes may require significant time and resources, this effort should not be motivated just by cost savings but as a critical step toward reducing our collective carbon footprint. By prioritising these changes, we can make a meaningful impact on both the environment and the industry as a whole.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47abb73560f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IT Certification — Yay or Nay?]]></title>
            <link>https://medium.com/@samuilkonyarov/it-certification-yay-or-nay-52f4c7d5b825?source=rss-c67fce608cd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/52f4c7d5b825</guid>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[growth]]></category>
            <dc:creator><![CDATA[Sami Konyarov]]></dc:creator>
            <pubDate>Mon, 11 Nov 2024 00:20:39 GMT</pubDate>
            <atom:updated>2024-11-11T00:31:30.830Z</atom:updated>
            <content:encoded><![CDATA[<h3>IT Certifications — Yay or Nay?</h3><p>When it comes to IT certifications, it’s always more nuanced than just a simple “yay or nay”, more often the answer is that“it depends.” I have a soft spot for certifications, and my inspiration for writing on this topic came from passing the Databricks Data Engineer Professional exam a little over a week ago. The learning process is the main motivation to spend the time to study, but personally I also enjoy the thrill of sitting a proctored exam. I love the pressure of potential failure looming with each question and the high of hitting the “submit” button and seeing the word “Passed.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*3iQkUEz6gZA3RGqY_3hVAw.png" /></figure><p>And yes, certain doors will open easier with certifications than without. However, none of this guarantees success, one could have dozens of certifications and do them no good. There are several specific use cases related to this topic that I find interesting, and I would love for people to join in the comments to share their thoughts.</p><p><strong>Breaking into the industry / career advancement<br><em> TL;DR</em></strong><em> Certifications can boost your chances of landing an interview, especially for junior roles as they demonstrate initiative and basic skills, but their impact lessens at senior levels.</em></p><p>One of the main benefits of getting a certification is that it can significantly help when looking for a job. I am not a recruiter, nor do I hire people in my role, so take everything with a pinch of salt. In my opinion, certifications can help you with landing an interview, but no one will give you the job because of them, and there’s an important caveat. Certifications are useful in job hunting up to a certain level of seniority. I believe the highest level where they hold real value is senior; beyond that, I doubt an employer would offer an interview solely because someone has, say, 15 AWS certifications.</p><p>So if you’re just starting out in the industry and if you can afford it, taking an exam for the technology you’ll be working with can be extremely beneficial. I will be the first to raise my hand and openly say that is certainly one of the main reasons I managed to get into the industry. When an employer sees a “junior” with a certification, it tells them two things right away: “they’re proactive” and “I won’t have to teach them the abc.” Again, a certification won’t guarantee you a job, but it will get you one step closer. It is important to highlight, you do not need a certification to get your foot in the door, you can have a portfolio/GitHub repo that outshines any certificate out there and that will hold even more value.</p><p><strong>Learning new technology / continued learning<br>TL;DR</strong><em> Studying for a certification strengthens your knowledge of a technology and enhances your job skills, especially when you’re less experienced, though the benefit diminishes as your familiarity with the tool grows.</em></p><p>Studying for an exam has the valuable by-product of learning new information. In most cases, you will become better at using a technology after taking an exam on it. For example, Databricks certifications are carefully designed to cover a wide range of its tools, from SQL queries to ETL pipelines, Delta Live Tables, monitoring, and more. Preparing for the exam requires learning about all of these areas. You will not become an expert in either of them, but this knowledge will undoubtedly improve your overall skills on the job.</p><p>Just as certifications are particularly useful for junior professionals, they become less impactful as someone gains more experience. For instance, when I took the Databricks Associate exam for data engineers, I had been using Databricks for less than six months. Studying for it was very hard because it involved learning a lot of net new concepts across a wide range of topics. Fast forward 18 months to when I began preparing for the Professional certification, I found it much easier compared to the Associate level. While there was still a lot of material to go over, I could connect the dots quicker and I believe that was down to having more exposure to the technology.</p><p>In short, if you’re just starting to use a new tool, studying for a certification will offer more benefits than if you’ve been using that technology for a long time. While you’ll still learn new things regardless of your experience level, the learning curve won’t be as steep.</p><p><strong>Progress without certifications<br>TL;DR </strong><em>Certifications can boost confidence and are useful for career advancement, but they require a significant time investment so it is essential to weigh their benefits against other priorities.</em></p><p>Whether you’re an expert or not, you will need to spend time studying for a certification, often after working hours. This is the main drawback of getting certified — it is time-consuming, and the big question is whether that time could be better spent elsewhere. Certifications naturally provide a confidence boost due to the learning involved and are great for ticking a box, but they don’t provide much guidance on where you want to go in your career or how to get there — areas that, in my opinion, are often overlooked. Some may argue that you can pursue both certifications and self-reflection, but the reality is that time is a scarce resource, and I often find myself without enough to do everything I would like. As I mentioned initially, I think certifications are great, I have been lucky to earn a few and I am sure I will try to get several more in the future, but I’m also aware that they’re only as valuable as you make them, and they require a significant time investment.</p><p>To wrap up, I love certifications and believe they play a strong role in the IT industry. While they’re not a magic solution, they can be highly valuable in certain scenarios.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52f4c7d5b825" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Glorious Spark UI Simulator: Turning Theory into Practice for Debugging Spark Jobs]]></title>
            <link>https://medium.com/@samuilkonyarov/the-glorious-spark-ui-simulator-turning-theory-into-practice-for-debugging-spark-jobs-b56b345716c5?source=rss-c67fce608cd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b56b345716c5</guid>
            <category><![CDATA[spark-ui]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Sami Konyarov]]></dc:creator>
            <pubDate>Sun, 03 Nov 2024 11:16:49 GMT</pubDate>
            <atom:updated>2024-11-03T11:53:21.136Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwOgWU0mnpKSPmTkIgGFVw.jpeg" /></figure><p>Even if you are living a happy life at Databricks land, one day you’ll find yourself needing to visit the Spark UI, and then everything starts to feel like hell. If you’re like me and learn best by doing, you’ll likely find this write-up helpful. I don’t plan to write another article about Spark transformations vs actions in job queries, or topics like executors, stages, tasks, etc, there is loads of it out there and I will end up repeating someone else’s words. However, I was surprised to find that no one seems to be talking about the Spark UI Simulator.</p><p>In the beginning, I alluded how difficult it can be to decipher the output of the Spark UI, a challenge made even harder by the fact that you can’t run real-life production jobs on your local machine to observe the Spark UI under heavy load. As a result, most of what you learn about the Spark UI comes from articles and theory — until, one day, you get paged at work because everything’s on fire. That’s when you get your first real taste of debugging and issue detection, which is far from ideal preparation for a crisis.</p><p>So, to the rescue comes the <a href="https://www.databricks.training/spark-ui-simulator/index.html">Spark UI Simulator</a>. This is absolutely phenomenal work, and I am so excited to write about it. The Databricks team have put together several use cases demonstrating key Apache Spark concepts. Each job is run on extremely large datasets, allowing you to access detailed metrics related to the executions, cluster configurations, and the source code for each simulation.</p><p>So may I present to you the Spark UI Simulator! You are welcomed by rather basic looking index page which lists all experimentations that have been performed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGZy_EZUoZVDcGymiqMtjg.png" /><figcaption>Spark UI home page</figcaption></figure><p>Choose an experiment that catches your attention, and the source code used for the run will load on your screen, with the cluster configuration displayed at the top of the notebook. The one from screenshot below compares filtering a dataset with and without z-order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mn8Wthjw7nBtJJh3cYH-hw.png" /><figcaption>Filtering 1TB dataset w and w/out z-order</figcaption></figure><p>It’s worth noting something you may have already observed: while this is an interactive application, most of the information is presented as screenshots, so you won’t be able to select and copy commands or use every visible hyperlink. Rest assured that the Databricks team have added buttons where necessary, ensuring you will still have access to everything you need. The source code can be downloaded via the “Source Notebook” button in the top right corner.</p><p>Let’s get to the fun part, click on the top left button “Spark UI”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1r56lX4sI0rB9j5WEoEUw.png" /><figcaption>Spark UI “homepage”</figcaption></figure><p>This view is about 99% what you will see in real life for your job runs. At the top, circled in red, you can toggle through the screens for the stages of the chosen job, environment variables, the executors metrics of the run and others. The hyperlink circled in green, takes you to the stage’s specific screen which will display its DAG and the I/O stats (see screenshot below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-LYrC3add5bTkN2diSU6A.png" /><figcaption>A single stage’s screen</figcaption></figure><p>Clicking the hyperlink in the stage description (circled in red) will take you to the tasks for that stage, displaying all associated metrics for each task. And I quickly want to interject to provide my unsolicited advice: when explore the simulator, try not even reading through the names of the experiments and the code. Instead, go straight to the Spark UI and observe the metrics. Look for anything unusual, and only then review the source code. I believe this approach is more effective because reading the title and scanning the code first can lead you to look for specific outcomes, even with a basic understanding of Spark. The screenshot below shows the tasks page for the stage we accessed in the previous screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FCBtHkabzKNZ05kN2IkCpw.png" /><figcaption>Tasks associated with a stage screen</figcaption></figure><p>Try to interpret the metrics without relying on any hints from the experiment. In the example shown, the stage ran for almost 5 hours, with each executor reading 60GB of data and handling 500 tasks. That sounds like a lot, it is also a reminder why we use the simulator, and we do not run it on our machines, but the more important question is “was that a healthy run?”. A long duration doesn’t necessarily indicate inefficiency and clearly was churning through a lot of data. If we look at the summary metrics, we see that the longest task took 25% longer than the 75th percentile, which suggests a healthy stage (if the maximum duration was over 50% longer than the 75th percentile, that would indicate further investigation).</p><p>I could go on and on about the Spark UI simulator as I really do believe it is amazing. However, I’ll keep this brief and wrap up here. My main takeaways:</p><p>1. The simulator won’t teach me how to optimise performance, I will still need to learn how to read DAGs, analyse performance plans and understand healthy metric ratios.</p><p>2. The simulator can be incredibly useful when preparing for a Proof of Concept.</p><p>3. Using the Spark session method “setJobDescription()” is a game changer for navigating the UI effectively.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b56b345716c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recap of the Data Science Festivsal in London]]></title>
            <link>https://medium.com/@samuilkonyarov/recap-of-the-data-science-festivsal-in-london-acc9bb62f428?source=rss-c67fce608cd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/acc9bb62f428</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sami Konyarov]]></dc:creator>
            <pubDate>Sun, 27 Oct 2024 22:27:37 GMT</pubDate>
            <atom:updated>2024-10-27T22:27:37.699Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week I attended the Data Science Festival in London. This was my first data conference I ever went to, and I absolutely loved it. The best part was meeting people from different backgrounds, with various interests and passions. The second-best thing was the sessions that were prepared by the speakers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D2Z7w9pjKTuxkeXrVoDzbQ@2x.jpeg" /></figure><p>Choosing which sessions to attend must have been the hardest part of the day. My passion lies in the engineering part of the data science and it is what I find most interesting, at the same time I am very conscious of the need to constantly broaden my horizons and learn new things. Fortunately, the organiser got everyone covered and one can watch the <a href="https://datasciencefestival.com/event/oktoberfest-2024/">recordings</a> of all sessions later.</p><p>The talk of the town for the whole conference was LLMs and I am sure this does not come as a surprise to anyone, but more interesting I was curious how relevant will be the content of the sessions considering the rapid rate of change in the AI field and the constant release of new models and tools. With all that said, I must admit that the speakers have done a tremendous job, and I am sure a lot of last-minute refactoring to provide us with the latest information on subjects covered.</p><p>Going to this conference I was aware that a lot of what the speakers talk about will go over my head as it will be data science centric, however the main motivation to attend was to understand what challenges they face in their world and how a data engineer might have helped. I was surprised by how often I heard people say they wished they had better data. The term “better data” came in different shapes – some said it was stale by the time it made it to the ML model and the predictions were irrelevant, others said that their data had such bad quality issues that they could not use it at all for inference. The same sentiment was echoed by the speakers when talking about the roadblocks they faced, if it was not about the quality of data, it was the regulatory restrictions that prevented them from using the full capabilities of what they have created. It was compelling to see that the very issues faced by statistical researchers, BI dashboards, and ML models are now affecting the state-of-the-art technology like LLMs as well.</p><p>In the end I would like to encourage you to have a look at the recordings of the conference, there is some incredible sessions that should be seen. My personal favourite is how Toyota Europe solved for sending immediate notifications to their users when a car failure was detected.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acc9bb62f428" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>