<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zhidong Ke on Medium]]></title>
        <description><![CDATA[Stories by Zhidong Ke on Medium]]></description>
        <link>https://medium.com/@zhidongke?source=rss-cd18b8daacf4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ewALemeX8Ihup1KJ3_Nq1Q.jpeg</url>
            <title>Stories by Zhidong Ke on Medium</title>
            <link>https://medium.com/@zhidongke?source=rss-cd18b8daacf4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:47:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zhidongke/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Engagement Activity Delta Lake]]></title>
            <link>https://medium.com/salesforce-engineering/engagement-activity-delta-lake-2e9b074a94af?source=rss-cd18b8daacf4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e9b074a94af</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[scale]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Zhidong Ke]]></dc:creator>
            <pubDate>Wed, 23 Sep 2020 15:21:45 GMT</pubDate>
            <atom:updated>2020-10-21T14:46:16.290Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Authors: Zhidong Ke, Utsavi Benani, Heng Zhang, Kevin Terusaki, Yifeng Liu, Percy.Mehta, Priyadarshini Mitra, Jayanth Parayil Kumarji</em></p><h3>Introduction</h3><p>To support our customers using <a href="https://www.salesforce.com/products/sales-cloud/tools/high-velocity-sales/">High Velocity Sales</a> to intelligently convert leads and create new opportunities, we built the engagement activity platform to automatically capture and store user engagement activities. The engagement delta lake is one of the key components supporting <a href="https://www.salesforce.com/products/einstein-analytics/features/">Einstein Analytics</a> for creating powerful reports and dashboards and <a href="https://www.salesforce.com/products/sales-cloud/features/sales-cloud-einstein/">Sales Cloud Einstein</a> for training machine learning models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XeIiPVtDhxBGVTpRQodeDw.png" /></figure><p>Unlike data in our other data lakes, engagement activity is mutable and the mutation ratio is high, which creates a huge challenge for us. We explored a couple of solutions and ultimately decided that the open source tool <a href="https://delta.io/">Delta Lake</a> met all of our requirements. In this blog, we will walk through our journey through the following sections:</p><ol><li>Engagement activity ingestion</li><li>Incremental Read</li><li>Support exact once write across tables</li><li>Handle mutation with cascading changes</li><li>Normalize tables in data lake</li><li>Performance tuning</li></ol><h3>Engagement Activity Ingestion</h3><h4>High Level Architecture</h4><p>Our engagement data lake is a distributed key-value object store which is similar to our existing data lake known as <a href="https://engineering.salesforce.com/our-journey-to-optimal-job-sizes-for-apache-spark-e787501e63ef">Shared Activity Store</a>. To support the basic requirement from Delta Lake, the ingestion pipeline starts when data is pushed to our internal Kafka queue. Then we use Spark Streaming Framework to consume engagement in micro-batches and write the data to Delta Lake table(s),as Figure 1 shows below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2vjOphSTD3-UcHRnbVGVNg.png" /></figure><p>We start with a single table called Data Table, partitioned by organization ID and engagement activity timestamp to support basic queries, such as a time range query for our downstream consumers. However, our customers requested incremental read support. Delta Lake’s native API requires tables to be append-only, but ours is mutable, so we had to implement incremental read support on our own.</p><h4>Incremental Read</h4><p>We created a separate table called Notification Table that was partitioned by organization ID and ingestion timestamp. When new engagement is inserted into Data Table or when existing engagement is updated or deleted, we notify downstream consumers of those changes by inserting a new entry with the engagement metadata into the notification table, as Figure 2 shows below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIE-f3S9GmVFV5XJEFuDAQ.png" /></figure><p>The downstream consumers can either use streaming mode to listen to Notification Table, or they can periodically pull new notifications from the Notification Table. Based on the value of the data, e.g. organization ID or engagement date, they are able to fetch the actual engagement data from our Data Table.</p><h3>Exact Once Write Across Tables</h3><p>While introducing the Notification Table solves our incremental read problem, it creates another challenge for us. We found that when our job tries to process and write data to two tables, Delta API only supports ACID transactions within each table; there is no batch transaction support across tables. For example, if the job writes to Data Table successfully but fails to write to Notification Table, the downstream completely misses this batch since it doesn’t get a notification. This becomes a common problem across our project when a job needs to write to multiple tables in a batch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0f00kHjHy_hpTFG27-AjeQ.png" /></figure><p>So, we created a Checkpoint Store that stores the start, end offset, Kafka metadata, and last job state for a given checkpoint. The first task of the Spark job, then, is to read this store to get the last checkpoint metadata and the last job state.<br> <br>Since our job executes in a micro-batch mode, and each batch includes multiple processes (process or write data to Table 1, Table 2, Table 3, Table 4 and so on), we also need to store the batch state. We created a Batch Metadata Store to store job name, batch ID (the last succeed batch ID provided by the Spark foreachBatch API), process name, and timestamp of last modified time. After reading from the Checkpoint Store, the Spark job will get the last job state. If the last batch completed successfully, the Spark job will increase the last batch ID; otherwise, it will keep the last batch ID. Then the Spark job fetches the Batch metadata based on the job name.<br> <br>By comparing the batch ID provided from the Spark job to the one from our Batch Metadata Store, we will know which process succeeded or failed. (If batch ID matches, it indicates success; otherwise, it indicates a failure.) If all processes succeed, we will read the data from Kafka based on the start offset and fetch new data. Otherwise, we will read a certain amount of data based on the start and end offset from Kafka, and resume from the last failed process by skipping the ones with matched batch ID.<br> <br>Our job will execute the processes in sequence across tables: process/write data to Table 1, Table 2, Table 3, Table 4 and so on. After each process completes, it will update the Batch Metadata Store with the current batch ID and timestamp. After the completion of all processes, the job will update the Checkpoint Store with the offset and the job state including the batch ID.<br> <br>The following Figure 4 shows the complete workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yc-FuIBYZG-siLtl3dB2Og.png" /></figure><h3>Handling Mutation In Data Lake</h3><p>Our engagement data is mutable, which means we not only need to support insertion but also need to support update and deletion in data lake. As we all know, mutation operation is a much more expensive operation than insertion in batch jobs. Furthermore, due to the nature of the engagement data, the mutation request could be cascaded, and our mutation volume is almost 20% of insertion volume. So the goal for us is to reduce the mutation operation as much as possible in each batch.</p><h4>Use Graph To Detect Cascading Mutation</h4><p>One simple example for cascading mutation is id1→id2, id2→id3, id3→id4. If we receive such a request in one batch and just execute the update query, the output will be incorrect, e.g. id1 will be id2, but we expect it to be id4. In order to execute them in a batch against our data lake, we will need to pre-process mutation requests. As Figure 5 shows, we break it down into several steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L6_SygpVk673WESGBt_UFg.png" /></figure><ul><li>We read the mutation request from the mutation table in a batch and group them by organization and sorted by mutation execution time.</li><li>For each group, we use Spark Graph API to build direct graph(s) where the node is the IDs and the edge is the request, and we find all connected components.</li><li>We find the final state of each node; the process goes through each node in each sub-graph, and, based on the execution time, finds its corresponding final node.</li><li>Finally, we convert each connected component into a list request for each organization, then apply the components in a batch.</li></ul><h4>Normalize Table In Data Lake</h4><p>As we mentioned, the mutation request volume is high, which impacts our performance a lot. We took a closer look into our data shape and found that a lot of records need to be mutated for any one request. For example, a request to change t_name_1 to t_name_2 can touch thousands of records because we store the t_name for each record (thanks to our “flatten data” scheme design).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jd0gIbGxNmdMzJ1n5U40LQ.png" /></figure><p>As the Figure 6 shows, we extract out the “NAME” fields from our data table into a separate table called mapping table, and, in this table, we store the mapping from ID to name. By changing our original design, we also need to add an extra write in the ingestion job and an extra update in the mutation job for both data table and mapping table. But we do see a big performance jump since we significantly reduce the number of records/files updates in each mutation batch.</p><h3>Performance Tuning</h3><p>Finally, we’ve solved most of our design challenges and have reached the performance tuning stage. Based on the performance reports for our ingestion and mutation jobs, we achieved product SLA on the ingestion part. But addressing the performance issues in the mutation job took much longer than we expected. So we looked into the DeltaLog and ran the analysis against our tables and found the following issues:</p><ol><li>Due to the partition scheme (organization_id/engagement_date) and traffic across many thousands of organizations, there are many small files (in MBs or KBs) in certain partitions, which caused the slow mutation operation.</li><li>Our mutation job doesn’t repartition by the partition key before it writes, so many tasks are writing to the same partition and slowing down the write process.</li><li>Spark set the default spark.default.parallelism to 200, which limited our join and read parallelism.</li></ol><p>To address those issue, we applied some fixes:</p><ol><li>We compacted those small files by running the “OPTIMIZE” table command provided by Delta Lake.</li><li>We added one more stage at the end of our mutation job to partition by organization_id and engagement_date to make sure only one task is writing to a partition.</li><li>We increased the spark.default.parallelism to 400.</li></ol><p>After those changes, we saw a performance gain while running the performance test, but it still didn’t meet our goal, so further optimization is required. We started experimenting with the partition scheme to only partition by organization_id, since this is the only way we can improve the file distribution, but we also needed to pay attention to downstream consumer read performance (most queries are based on organization_id and date). We ran an AB test against two partition schemes:</p><ol><li>We created a new table with organization_id as the partition key and loaded the same amount of data as the original table.</li><li>We ran the OPTIMIZE command with z-ordering by engagement_date to improve read performance.</li><li>We analyzed the file size per organization and compared the read/update/insert operation time.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*StGQ1DCjtplpv15eqvLHSw.png" /></figure><p>To our surprise, we saw a more than 50% read performance gain, more than 300% update performance improved, and 10% more time for insert (since we added an extra job for OPTIMIZE). This is a huge performance improvement for our data lake, and we summarized the reasons as follows:</p><ul><li>The mutation requests in each batch are going to touch a big time window, which means they will need to touch a large amount of partitions if we have date as sub-partition.</li><li>By changing the partition scheme to organization_id, it highly reduces the I/O in both read/write/update.</li><li>The Delta Lake is using the Data Skipping and Z-Ordering algorithm for I/O pruning where it supports our read query pattern.</li></ul><h3>Conclusion</h3><p>Building our Engagement Activity Delta Lake was a fun process. We hope our journey may help those who are designing a data lake that supports batch/incremental read, want to support mutation of the data in data lake, want to scale up data lake with performance tuning, or want to support exact once write across tables with Delta Lake.</p><p><em>Read more about our Delta Lake implementation:</em></p><ul><li><a href="https://engineering.salesforce.com/global-synchronousness-and-ordering-in-delta-lake-9b912d980ebf">Global Synchronousness and Ordering in Delta Lake</a></li><li><a href="https://engineering.salesforce.com/boost-delta-lake-performance-with-data-skipping-and-z-order-75c7e6c59133">Boost Delta Lake Performance with Data Skipping and Z-Order</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e9b074a94af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/salesforce-engineering/engagement-activity-delta-lake-2e9b074a94af">Engagement Activity Delta Lake</a> was originally published in <a href="https://medium.com/salesforce-engineering">Salesforce Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guaranteed Data Delivery System for Distributed Services]]></title>
            <link>https://medium.com/salesforce-engineering/guaranteed-data-delivery-system-for-distributed-services-3b07fd24775d?source=rss-cd18b8daacf4------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b07fd24775d</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Zhidong Ke]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 15:46:01 GMT</pubDate>
            <atom:updated>2020-06-30T15:46:01.041Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Authors: Zhidong Ke, Utsavi Benani, Kevin Terusaki</em></p><p>With Sales Cloud Einstein, relevant activities like emails and meetings from your Inbox are automatically crawled and added to your organization timeline. While designing the data pipeline using distributed system, we keep our customers’ trust as our top priority, which means we need to make our system reliable, with as high a fault tolerance as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zdCMDGAHkKibS4a95q4L_w.png" /></figure><p>No matter how long the pipeline is, you never want to see data dropping off along the way. And yet that’s exactly what we were finding: emails and meeting notices were sometimes disappearing, and we didn’t have a way to trace each message. Debugging was extremely difficult, in part due to untraced log. So we set out to build an End-to-End Verification System to trace every message to its downstream consumer and to automatically retry if a message failed to deliver rather than manually re-crawling.</p><p>Here’s how we summed up the requirements our system would need to meet:</p><ol><li>Able to trace every email and meeting notice at scale</li><li>Able to support creating a new entry when a new notice is created</li><li>Offer support for acknowledgment by each downstream consumer</li><li>Able to support a query that returns all messages that failed to delivery</li><li>Kicks-off automatic retries when a message fails to deliver</li><li>Can track how many messages failed initially, passed after system implementation, and failed in spite of new system implementation.</li></ol><p>Before starting to build our own tool, we evaluated the open source options Jaegar and Datadog.</p><p>Jaegar was powerful enough to support the distributed tracing we needed and offer latency metrics, but it would have required us to install the agent on every machine it needed to run on. And, it’s not designed for an end-to-end verification system like we needed, since it doesn’t support querying for all failed messages.</p><p>Datadog was also powerful enough to support our needs, and we already have it installed on our machines. But it had the same lack of an end-to-end verification system and is also relatively new, so we were wary of going all in on it at this time.</p><p>Ultimately neither of these solutions would have worked for us out-of-the box, so we settled on building our own.</p><h3>High-level Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULnzcGirUM4qialA7Sr2RQ.png" /></figure><ol><li>Create an end-to-end verification module and implement insert/update/delete/query operation with Cassandra as data store.</li><li>Integrate the module with our realtime crawler to insert a message for each notice.</li><li>Support downstream consumers to acknowledge back.</li><li>Create a verification job to query the system every 10 minutes to fetch the messages that were not delivered by at least one consumer, then kick off the retry for them.</li><li>Implement a checkpoint to verify every completed bucket.</li></ol><h3>Component Level Details</h3><h3>Cassandra as storage</h3><p>The schema for our Cassandra datastore would look like this:</p><p>CREATE TABLE verification (<br>bucket timestamp,<br>datasource text,<br>id text,<br>crawlTime timestamp,<br>priority int,<br>service_1 boolean,<br>service_2 boolean,<br>retry int,<br>PRIMARY KEY ((bucket), datasource, crawlTime, externalId))<br>WITH CLUSTERING ORDER BY (datasource DESC, crawlTime DESC, externalId DESC)</p><p>Based on the number of events occurring per hour during peak times, so we used 10 minutes as a bucket, with a Time to Live (TTL) of seven days for the collected data.</p><h3>Connectors</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vM_RalxfGW5CluKVX10VGQ.png" /></figure><p>When the collector (Exchange/Google) collects a notice/datasourceId, it inserts a message into Cassandra through the Module based on the schema above. The Module will figure out the bucket based on the current timestamp and will keep every consumer check box as empty, waiting for them to acknowledge. This acknowledgement will happen at each downstream consumer after they successfully process the data. We also built an API as a proxy into our core infrastructure, so, for every message received by consumer in core infrastructure, we call the API to acknowledge it.</p><h3>Verification/Retry job</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3bap07xVdyvep5Zdu6Z4mw.png" /></figure><p>For checkpointing, we’re using Zookeeper as our metadata store. In our implementation of the re-try job, Spark reads the last checkpoint from it. Then, the verification job queries Cassandra for each bucket and gets all of the messages that need to be retried. The verification job passes the retry request to the Retry Job, grouped by the id. The Connector Re-processor makes a batch call to the external datasource and emits the missing activity to the downstream consumers accordingly, and creates a new entry with the current bucket in Cassandra and increases the retry counter. Once all messages are completed in the retry, the driver gets notified and creates a checkpoint for this bucket.</p><p>With this design in place, our system is becoming more robust and is able to heal automatically after any errors happen. Of course, there is always some room to improve, so we are currently working on trying to extend it to support the span or branch and retry failures from any point of the data flow. But as it stands, we are able to trace and view problematic components in real time. The advantage of this design is that it is simple to implement and easy to embed into any kind of application by installing the module. It’s worked so well for us that we are in the process of open sourcing it, so stay tuned for a link to a GitHub repo in the future!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b07fd24775d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/salesforce-engineering/guaranteed-data-delivery-system-for-distributed-services-3b07fd24775d">Guaranteed Data Delivery System for Distributed Services</a> was originally published in <a href="https://medium.com/salesforce-engineering">Salesforce Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Journey to Optimal Job Sizes for Apache Spark]]></title>
            <link>https://medium.com/salesforce-engineering/our-journey-to-optimal-job-sizes-for-apache-spark-e787501e63ef?source=rss-cd18b8daacf4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e787501e63ef</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[scale]]></category>
            <dc:creator><![CDATA[Zhidong Ke]]></dc:creator>
            <pubDate>Thu, 01 Aug 2019 16:31:03 GMT</pubDate>
            <atom:updated>2019-08-01T23:57:20.339Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Zhidong Ke, Kevin Terusaki, Franklin Ye, Narek Asadorian, and Praveen Innamuri all contributed work to the project and this blog post.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ybqKg8m6DVEs0d-KuRNnQ.jpeg" /></figure><p>With Sales Cloud Einstein, we built relationship intelligence and productivity tools to help sales reps save time by automatically capturing their email and calendar events in Salesforce. We store and manage tons of data to power these features and to train the machine learning models for Einstein.</p><p>However, the way we store this data is inefficient for our Einstein teams because they must read in large amounts of data that’s distributed across hundreds of files. So we compact the activity data into fewer, bigger files. We initially assigned a fixed computation resource for each organization. However, each organization has different data volumes and we can’t efficiently utilize resources in a fixed manner with such an unbalanced data distribution set. Separately, there are EU General Data Protection Regulation (GDPR) laws that legally require us to remove certain activity records on-demand. Both of these issues pose unique challenges around managing the storage of our customers’ activities. <br> <br>With all that in mind, we’ll follow our journey toward a data compaction framework through the following waypoints:</p><ol><li>Activity Data Lake</li><li>Challenges</li><li>Requirements</li><li>The first version of compaction job</li><li>Metadata Store</li><li>Dynamic job scheduler</li><li>Failure recover system</li><li>Use Case and Performance Impacts</li><li>Future work</li></ol><h3>Activity Data Lake</h3><p>Our Activity Data Lake, also known as the Shared Activity Store (SAS), is a distributed key-value object store. The ingestion of data takes place in several real time micro-batch jobs depending on the type of data. Specifically, we are using the Apache Spark Streaming framework to implement micro-batch processing of activity data. As Figure 1 illustrates, the Spark Streaming job listens to the Apache Kafka queue and processes activity data by batching activities per organization. Last but not least, the job dumps this activity file to SAS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ln_MH3Q0Z1jTyYej_pBEg.png" /></figure><h3>Challenge: The Small Files Problem</h3><p>Our micro-batch streaming job works perfectly for the primary purpose of consistently storing data in real time. However, the micro-batched activity files have caused some side effects downstream. Downstream consumers have seen a decrease in performance due to the large number of files per organization, even though the size of each file itself is miniscule. This is a typical side effect of micro-batch processes and is commonly known as the <em>small files problem</em>. If the ratio of the number of files compared to the average file size is skewed, then you’ll see degradation in performance due to more I/O per input size when processing the data.</p><p>One might think that, if we have too many small files, then we should increase the micro-batch window size so that each batch will contain more data. Unfortunately, the trade off here would put more stress on the memory utilization of our Spark Streaming job which could potentially cause data loss or corruption. Keep in mind, we have one Spark Streaming job that aggregates the data of all of our customers by organization. Therefore, organizations that produce a large amount of activity data could cause problems for writing data for other organizations.</p><p>Another idea we explored is creating more Spark Streaming jobs that micro-batch data in the same way but read from different Kafka topics. Organizations with high activity volumes would come from a designated Kafka topic and the Spark Streaming Job would have a smaller window size, while organizations with low activity volumes would come from a different Kafka topic, and the Spark Streaming Job would have a larger window size. However, we decided not to go with this implementation for the following reasons:</p><ul><li>Bucketing organizations by data volumes (e.g. high volume, medium volume, low volume) is difficult and could change drastically depending on unforeseen factors.</li><li>Managing multiple Spark jobs and Kafka topics could become cumbersome and was not scalable if we wanted to create more buckets of organizations.</li></ul><h3>Requirements: General Data Privacy Regulation (GDPR)</h3><p>In order to be compliant with GDPR, we were required us to support on-demand requests to delete data for the Shared Activity Store. Delete requests could vary between an org wide delete, a user delete, or a deletion of a specific record solely based on an email address. The latter proved to be the most resource-intensive due to the structure of our activity data. In order to complete this request we have to scan all of an organization’s activity batches to a delete specific record. For this specific use case, we again encountered the small files problem for organizations with low data volume.</p><h3>The first version of compaction job</h3><p>Our initial design (Figure 2) was a Spark job that read activity data for all organizations. It partitioned the data by organization, and then again into single-day activity batch files. We started the Spark job with a large amount of memory, but the job died after 15 hours because of out of memory issues. We tried increasing the memory as well as the number of executors, but the job still died. And each time the job died, we had to clean up the dirty files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ddoCVhCmr1Es0un6HASjw.png" /></figure><p>After a few executions, we found that it’s not a good idea to run Spark jobs for an extended period of time because the cost of failure is high. Instead, we broke down a single compaction job into smaller sub-jobs where each sub-job spanned a defined number of days. For example, if we wanted to compact the files for a given organization for the entire year, we created twelve sub-jobs where each sub-job is responsible for compacting the data for a different month of that year.<br> <br> In most cases, this adjustment worked well. However, for some sub-jobs, the input was too big (around 5T!) and those took more than 10 hours to complete and exhibited high failure rates. Again, we changed our approach. We decided to chunk the input data differently depending on the total input size instead of the fixed time range. We tried to limit the input size to less than 100G per Spark job.<br> <br> This first version design worked pretty well, but it required too much manual work up front. Too often, we found ourselves calculating our input size for a compaction job to ensure we would not run into memory issues. This process was so slow and painful that we ultimately decided to automate. Much of the manual work to calculate the input size came from just gathering metadata:</p><ol><li>Gathering the total input size (memory of all activity batch files) for an organization</li><li>Given a date range and the total input size, calculating the required number of sub-jobs.</li><li>Monitoring the health of the compaction job</li></ol><p>Instead of running the above steps manually, we decided to design a better compaction scheduler that could intelligently partition the sub jobs for a given compaction request based on data lake metadata that we automatically gather. Additionally, we wanted to distribute the sub jobs across multiple Spark jobs to speed up the execution time. We’ll dive into the metadata store, dynamic scheduler, and distributed compaction in the following sections.</p><h3>Metadata Store</h3><p>We designed a backfill job to collect the metadata from our activity data lake for the existing data, and we scheduled the near-realtime job to consistently update the metadata storage (Figure 3).</p><p><strong>What metadata we store:</strong></p><ul><li>Number of Activity Batch Files</li><li>Average Files Size</li><li>Number of Users</li></ul><p><strong>How we collect Metadata:</strong></p><ul><li>Query the activity data lake for file names and sizes</li><li>Perform metadata calculations</li><li>Write those metadata to storage</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Avh2ylFTNTadqD8UKkTdLg.png" /></figure><h3>Dynamic Job Scheduler</h3><p>Initially, our scheduler had a single worker responsible for the entire date range of crawled files for all organizations. We found that we could improve the resiliency by breaking up each request into smaller sub-jobs, but found it tedious to do this manually. In addition, we wanted the ability to allocate more resources for bigger requests, which resulted in switching to a distributed job flow. For example, an organization with more than 30,000 users requires more than 1 TB of memory, which is why we need a dynamic job scheduler and distributed workers based on the input size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JhbzjH6NErWLZKtOAFLxSg.png" /></figure><p><strong>Scheduling for Distributed Flow</strong></p><ul><li>Each organization is split between workers in a distributed flow based on current resource allocation.</li></ul><p><strong>Resource Scheduler</strong></p><ul><li>Now that we’ve gathered metadata, we can intelligently schedule jobs based on the proposed input size. The Resource Scheduler reads average file size, number of files, and total days stored from the metadata store. This metadata helps estimate the size of creating a compacted file for one day, and gives each sub-job a number of days such that the size is not too big (bigger than the memory of the executor).</li><li>The resource scheduler initializes the job state in the database for the compaction job tracker.</li></ul><p><strong>Compaction Job Tracker</strong></p><ul><li>The overall request keeps track of all the sub-jobs and their states. Once a sub job compacts the relevant dates, it moves to a completed state and doesn’t need to be processed again.</li></ul><p><strong>The failure recovery system</strong></p><ul><li>By breaking a compaction job into multiple sub-jobs, we create checkpoints in addition to memory management so we can recover from unexpected failure. If a sub-job fails in the processing state, it is picked up and completed on the next run.</li></ul><p><strong>Distributed Compaction</strong></p><ul><li>After breaking up a compaction request into discrete sub-jobs, we can increase parallelism by running multiple compaction jobs. The Dynamic Scheduler not only creates the sub-jobs, but assigns the sub-job.</li></ul><h3>Use Case and Performance Impacts</h3><p>With this new design, we’ve created an intelligent and fault-tolerant compaction framework that can handle large input sizes using a divide and conquer approach. In production, downstream consumers saw a 10x read performance gain. Additionally, this framework allows us to be GDPR compliant with its ability to not only compact files, but to remove activities (e.g. all activities associated with a given email address) related to a GDPR request.</p><h3>Future work</h3><p>Although we are quite satisfied with this Compaction Framework, there’s room for improvement.</p><ul><li>We want to build an auto compaction scheduler so compactions job aren’t just scheduled to run, but are triggered by reaching the size threshold of each organization.</li><li>Dynamically increase the number of worker nodes depending on the number of sub jobs scheduled, allowing us to efficiently schedule compaction jobs based on the current demand.</li><li>Sub Job compaction performance tuning to decrease the runtime of the compaction job process.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e787501e63ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/salesforce-engineering/our-journey-to-optimal-job-sizes-for-apache-spark-e787501e63ef">Our Journey to Optimal Job Sizes for Apache Spark</a> was originally published in <a href="https://medium.com/salesforce-engineering">Salesforce Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>