<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Ding on Medium]]></title>
        <description><![CDATA[Stories by Jeff Ding on Medium]]></description>
        <link>https://medium.com/@tracymacding?source=rss-cc243c061c14------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OZurMwECSqPjJKpdcDj2qA.jpeg</url>
            <title>Stories by Jeff Ding on Medium</title>
            <link>https://medium.com/@tracymacding?source=rss-cc243c061c14------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:44:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tracymacding/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep Dive To StarRocks IO Model , Part 1 - Concepts]]></title>
            <link>https://medium.com/starrocks-engineering/deep-dive-to-starrocks-io-model-part-1-concepts-ec67044881b2?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec67044881b2</guid>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Mon, 03 Nov 2025 02:48:03 GMT</pubDate>
            <atom:updated>2025-12-12T01:40:55.636Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Dive To <strong>StarRocks I/O Model , Part 1 - Concepts</strong></h3><h3><strong>Background</strong></h3><p>As a high-performance MPP database designed for high concurrency and low-latency analytical workloads, StarRocks’ data layout and underlying IO flow have always been somewhat mysterious. Especially after introducing the Shared-Data architecture, increased IO latency can lead to fluctuations in query performance. Compared with local disk access, the characteristics of object storage (high bandwidth, high latency, and weaker consistency) introduce a completely new set of challenges. How to maintain StarRocks’ consistent sub-second performance under such conditions is one of the most critical problems in both system architecture and IO layer design.</p><p>This series of articles aims to uncover the internal design of the StarRocks IO layer.</p><p>In this first article, we will systematically introduce the core concepts and mechanisms involved in executing queries on Shared-Data tables within StarRocks. By understanding the core elements — such as Tablet, Fragment, Pipeline, Cache, ScanOperator, Morsel, and others — we’ll gain a clear picture of how data flows from the underlying storage system into the execution engine, and how the system approaches latency hiding, concurrency scheduling, and hot/cold data management.</p><p>By the end of this article, you’ll have a bottom-up understanding of the StarRocks IO model — laying the foundation for deeper exploration of StarRocks performance optimization principles in the upcoming parts of this series.</p><h3><strong>Basic Concepts</strong></h3><h4>Tablet</h4><p>A <em>Tablet</em> is the fundamental physical unit of data storage in StarRocks. You can think of it as a horizontal shard of a table’s data. As an MPP database, when a table is created, StarRocks automatically splits the table’s data into multiple Tablets based on predefined rules, and each Tablet stores a portion of the table’s data. These Tablets are then evenly distributed across different CN nodes as much as possible, enabling distributed data storage and management.</p><p>In StarRocks, the number of Tablets is determined by the number of Buckets specified when the table is created — the number of buckets is equal to the number of Tablets. When creating a table, the bucket strategy and bucket count are specified using the DISTRIBUTED BY clause. For example, when creating an orders table, distributing by hashing on user_id with 10 buckets:</p><pre>CREATE TABLE orders (<br>    order_id BIGINT,<br>    user_id BIGINT,<br>    amount DECIMAL(10,2)<br>) DISTRIBUTED BY HASH(user_id) BUCKETS 10;</pre><p>This CREATE TABLE statement will create 10 Tablets, and the data will be evenly distributed across these 10 Tablets based on the hash value of user_id.</p><p>As the basic computation unit in StarRocks, Tablets play several important roles in the system:</p><ol><li><strong>Enable parallel computation and improve query performance.</strong></li><li>When executing queries, StarRocks can scan multiple Tablets concurrently, achieving parallel processing. For example, if a table has 10 Tablets, the query engine can launch 10 parallel scan-and-compute pipelines, each processing data from one Tablet independently, and then aggregate the results. This parallelism allows StarRocks to fully utilize multi-core CPU and multi-node compute power, significantly improving performance.</li><li><strong>Serve as the fundamental unit for data management.</strong>Many StarRocks data management operations are performed at the Tablet level. For example, data ingestion writes data on a per-Tablet basis; compaction operations also merge data files within a single Tablet; and when load balancing is required, Tablets are migrated between nodes. This granularity makes management flexible and efficient.</li><li><strong>Support horizontal scalability. </strong>When a new CN node is added to a cluster, StarRocks automatically migrates some Tablets onto the new compute node to rebalance the load. Combined with Shared-Data architecture, this enables seamless and fast horizontal scaling.</li></ol><p>As a core concept, Tablet bridges data storage, query execution, and system elasticity. Choosing an appropriate bucket count is critical to achieving optimal StarRocks performance.</p><h4>Fragment</h4><p>In StarRocks, the Frontend (FE) is responsible for generating the physical execution plan, which is a tree composed of physical operators. After the user’s SQL goes through parse, analyze, rewrite, and optimize phases, a physical execution plan is finally constructed.</p><p>A <strong>Plan Fragment</strong> is a component of the physical execution plan. The FE splits the execution plan into multiple Plan Fragments so that the plan can be executed in parallel across multiple nodes. Each Plan Fragment also consists of physical operators. In addition, a Plan Fragment contains a <strong>DataSink</strong>, which allows an upstream fragment to send data to the downstream fragment’s <strong>Exchange</strong> operator.</p><p>In the Pipeline execution engine, a Plan Fragment will be further split into multiple Pipelines by the CN.</p><h4>Pipeline</h4><p>A <strong>Pipeline</strong> is an execution chain composed of a series of operators. Data flows in from a <strong>Source Operator</strong>, is processed by various intermediate Operators, and then passed to a <strong>Sink Operator</strong>, which forwards the processed data to the next Pipeline. This process repeats iteratively.</p><p>The basic flow structure is illustrated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MEJFftW7OEJNAdayQ-ZhjQ.png" /></figure><p>The <strong>SourceOperator</strong> serves as the starting operator of a Pipeline and provides input data for the Pipeline. A SourceOperator can obtain data in the following ways:</p><ol><li>Reading from local files or external data sources, such as the <strong>ScanOperator</strong></li><li>Receiving output data from the upstream Fragment Instance, such as the <strong>ExchangeSourceOperator</strong></li><li>Receiving output data from the upstream Pipeline’s SinkOperator, such as the <strong>LocalExchangeSourceOperator</strong></li></ol><p>The <strong>SinkOperator</strong>, as the final operator in a Pipeline, is responsible for outputting the Pipeline’s computed results to downstream consumers.</p><p>The intermediate operators in a Pipeline both consume input chunks from upstream operators and produce output chunks for downstream operators.</p><p>The basic unit of data exchange between operators in a Pipeline is a <strong>Chunk</strong>, which generally represents a fixed batch of rows. During Pipeline execution, the Pipeline first obtains the input Chunk from the SourceOperator, then hands it off to downstream operators. After processing, the output is also packaged as a Chunk, which is then passed to the next operator. Through this continuous streaming-style processing, the final result eventually reaches the SinkOperator.</p><p>For each pair of adjacent operators, the execution thread invokes the pull_chunk function on the upstream operator to fetch a Chunk and then calls the downstream operator’s push_chunk method to push that Chunk forward.</p><p>For details on how a user query is decomposed into Fragments and Pipelines, please refer to this article: <a href="https://medium.com/starrocks-engineering/how-ssb-query-run-in-starrocks-0011e7e7bf6e">https://medium.com/starrocks-engineering/how-ssb-query-run-in-starrocks-0011e7e7bf6e</a></p><h4>ScanOperator</h4><p>The <strong>ScanOperator</strong> is the operator responsible for data scanning in StarRocks’ Pipeline execution engine. During query execution in StarRocks, the ScanOperator acts as the starting point of the data flow. Its role is to read data from the storage layer and pass the data it reads downstream in the form of Chunks (data blocks) for further processing.</p><p>You can think of the ScanOperator as the <strong>“data source”</strong> of the execution pipeline. Just like the first station on a production line provides the raw materials, the ScanOperator supplies the raw data on which all subsequent operations — filtering, aggregation, sorting, etc. — depend.</p><p>In StarRocks’ Pipeline execution model, a query is decomposed into multiple parallel execution streams (Pipelines). Each Pipeline contains one ScanOperator instance, and these ScanOperator instances work in parallel to complete the full table scan collectively.</p><p>For example, the OLAP_SCAN node shown at the upstream end in the following typical Pipeline diagram represents the ScanOperator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*FWhYG1h8p7yyOSIRXxnoVA.png" /></figure><h4>Morsel</h4><p>A <strong>Morsel</strong> is the basic unit of data scheduling in the StarRocks Pipeline execution engine. You can think of it as a “work package” or a “data shard.” During query execution, the data within a Tablet is divided into multiple Morsels, with each Morsel representing an independent scan task. The ScanOperator continuously fetches Morsels from the Morsel queue, and then creates a ChunkSource to perform the actual data scanning work.</p><p>If we compare data scanning to moving cargo, then the Tablet is like a large warehouse, while Morsels are the individual cargo boxes within that warehouse. The worker (ScanOperator) doesn’t need to move the entire warehouse at once — instead, it takes one box (Morsel) at a time from the pending queue, processes it, and then fetches the next one. This design allows multiple workers to collaborate efficiently: whoever becomes idle simply picks up the next available box, enabling dynamic load balancing.</p><p>In the simplest case, one Tablet corresponds to one Morsel. But when Tablet-level parallel scanning is enabled, a Tablet can be split into multiple smaller Morsels, providing more fine-grained parallelism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C-_FbGGjCFnYg3tzkBiobg.png" /></figure><p>Introducing Morsels in StarRocks enables two primary goals:</p><ol><li>Achieve intra-Tablet parallelism to improve scan efficiency</li><li>Better handle data skew</li></ol><p>To better understand the rationale behind this, let’s walk through a complete example. Suppose we have a user behavior log table that records user access events:</p><pre>CREATE TABLE user_logs (<br>      log_id BIGINT,<br>      user_id BIGINT,<br>      action VARCHAR(50),<br>      log_time DATETIME<br>)DISTRIBUTED BY HASH(user_id) BUCKETS 4;</pre><p>This table has 4 Tablets, each storing approximately 5 million rows, for a total of 20 million rows. Now we want to execute the following query:</p><pre>SET pipeline_dop = 8;<br>SELECT COUNT(*) FROM user_logs WHERE action = &#39;click&#39;;</pre><p><strong>Advantage 1: Achieving intra-Tablet parallelism to improve IO efficiency</strong></p><p>When intra-Tablet parallelism is disabled, each Tablet corresponds to a single Morsel, resulting in only 4 Morsels. Even if the system creates 8 ScanOperator instances, because there are only 4 Morsels available, only 4 ScanOperators can actually work concurrently. The execution process looks like this:</p><blockquote>ScanOperator 0 obtains Morsel 0 (Tablet 0) and begins scanning 5 million rows</blockquote><blockquote>ScanOperator 1 obtains Morsel 1 (Tablet 1) and begins scanning 5 million rows</blockquote><blockquote>ScanOperator 2 obtains Morsel 2 (Tablet 2) and begins scanning 5 million rows</blockquote><blockquote>ScanOperator 3 obtains Morsel 3 (Tablet 3) and begins scanning 5 million rows</blockquote><blockquote>ScanOperator 4–7 remain idle with no Morsel to process</blockquote><p>In this example, the actual parallelism is only 4, leaving 4 ScanOperators idle and CPU resources underutilized.</p><p>If intra-Tablet parallel scan is enabled, and assuming the system determines that each Morsel should contain roughly 256K rows, then each Tablet (5 million rows) will be split into ~20 Morsels (5,000,000 / 256,000 ≈ 20). With 4 Tablets, a total of roughly 80 Morsels will be generated. The execution process then becomes:</p><blockquote>Initially, all 8 ScanOperators concurrently pull Morsels from the shared Morsel queue:</blockquote><blockquote>ScanOperator 0 obtains Morsel 0 (the 1st shard of Tablet 0, rows 0–256K)</blockquote><blockquote>ScanOperator 1 obtains Morsel 1 (the 2nd shard of Tablet 0, rows 256K–512K)</blockquote><blockquote>ScanOperator 2 obtains Morsel 2 (the 3rd shard of Tablet 0, rows 512K–768K)</blockquote><blockquote>…</blockquote><blockquote>ScanOperator 7 obtains Morsel 7 (a shard from Tablet 1)</blockquote><blockquote>Whenever a ScanOperator finishes its current Morsel, it immediately pulls the next one from the queue, continuing this until all 80 Morsels are processed.</blockquote><p>Assuming each Morsel takes 2 seconds to scan:</p><ul><li><strong>Without intra-Tablet parallelism</strong>: 4-way parallel, each processing a single large Morsel → ~40 seconds (5M rows ≈ 40s)</li><li><strong>With intra-Tablet parallelism</strong>: 8-way parallel, 80 small Morsels → total time ≈ 80 × 2 / 8 = <strong>20 seconds</strong></li></ul><p>By enabling intra-Tablet parallelism, we increase effective parallelism from 4 to 8, fully utilizing all ScanOperators — cutting query time in half.</p><p><strong>Advantage 2: Better handling of data skew</strong></p><p>Assume the 4 Tablets have uneven row distribution:</p><blockquote>Tablet 0: 1 million rows</blockquote><blockquote>Tablet 1: 3 million rows</blockquote><blockquote>Tablet 2: 5 million rows</blockquote><blockquote>Tablet 3: 11 million rows</blockquote><p>If intra-Tablet parallelism is disabled, each Tablet maps to one Morsel:</p><blockquote>ScanOperator 0 handles Tablet 0 → 8 sec</blockquote><blockquote>ScanOperator 1 handles Tablet 1 → 24 sec</blockquote><blockquote>ScanOperator 2 handles Tablet 2 → 40 sec</blockquote><blockquote>ScanOperator 3 handles Tablet 3 → 88 sec</blockquote><p>Total time is determined by the slowest ScanOperator — <strong>88 seconds</strong> — while others finish early and go idle, wasting resources.</p><p>If intra-Tablet parallelism is enabled, each Tablet is split into multiple Morsels:</p><blockquote>Tablet 0 → 4 Morsels</blockquote><blockquote>Tablet 1 → 12 Morsels</blockquote><blockquote>Tablet 2 → 20 Morsels</blockquote><blockquote>Tablet 3 → 44 Morsels</blockquote><blockquote>Total: 80 Morsels</blockquote><p>All 8 ScanOperators pull Morsels dynamically from the shared queue. Even though Tablet 3 contains the largest amount of data, its Morsels will be processed collectively by all 8 ScanOperators. ScanOperators that finish small Tablets will automatically help process Morsels from the large Tablet, achieving automatic load balancing.</p><p>Total execution time: approximately <strong>80 × 2 / 8 = 20 seconds</strong>, which is more than <strong>4× faster</strong> than the 88 seconds required without splitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_lxCueCkIhcz2I_oXKOHQ.png" /></figure><h4>ChunkSource</h4><p>A <strong>ChunkSource</strong> is the component in the StarRocks Pipeline execution engine that actually performs data scanning. You can think of it as a “data reader” or a “scan execution unit.” Each ChunkSource holds a Morsel (a data shard) and is responsible for reading the data corresponding to that Morsel from the storage layer, caching the scanned data in the form of Chunks, and supplying it to the ScanOperator for consumption.</p><p>If we compare data scanning to ordering food at a restaurant, then a Morsel is like a dish on the menu, a ChunkSource is the chef who actually cooks that dish in the kitchen, and the ScanOperator is the head chef coordinating all the chefs. The ScanOperator may create multiple ChunkSources as needed; each ChunkSource receives a Morsel, reads the corresponding data from the storage layer, returns the scanned data to the ScanOperator, and then can be reassigned to handle the next Morsel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C-_FbGGjCFnYg3tzkBiobg.png" /></figure><p>ChunkSource works asynchronously. When a ChunkSource begins reading data, it wraps the read operation into a ScanTask and submits it to the global IO thread pool, instead of blocking the execution of the ScanOperator. This mechanism allows multiple ChunkSources to perform IO concurrently, fully leveraging system-level concurrency.</p><p>Each ScanOperator maintains a fixed-size array of ChunkSources internally. The size of this array determines how many ChunkSources the ScanOperator can run at the same time — which also determines how many Morsels this ScanOperator can process concurrently.</p><p>ChunkSource plays a critical bridging role in the scanning pipeline — linking the scheduling layer (Morsel queue / ScanOperator) with the execution layer (IO thread pool / storage system).</p><ol><li><strong>Performing actual data reads. </strong>ChunkSource is the component that directly interacts with the storage layer. Based on information contained in the Morsel (e.g., Tablet ID, row ranges, file paths, etc.), ChunkSource invokes the low-level storage interfaces to read data.</li><li><strong>Implementing asynchronous IO + batch processing. </strong>ChunkSource submits IO tasks asynchronously to the global IO thread pool, preventing Pipeline execution threads from blocking. Each task reads multiple Chunks (each Chunk contains many rows) rather than reading row by row. This batching amortizes scheduling overhead and improves IO efficiency. The scanned Chunks are placed into ChunkSource’s internal buffer, allowing the ScanOperator to fetch data quickly without waiting for IO completion.</li><li><strong>Supporting fine-grained concurrency control. </strong>A ScanOperator can manage multiple ChunkSources concurrently, and each ChunkSource can scan a different Morsel in parallel. For example, if a ScanOperator is configured with 16 ChunkSource slots, it can launch 16 data read tasks simultaneously — all running concurrently in the IO thread pool — significantly improving IO throughput. Meanwhile, ScanOperator dynamically adjusts the number of active ChunkSources based on memory usage to avoid exceeding memory limits.</li></ol><h4>ScanTask</h4><p>A <strong>ScanTask</strong> is the actual task unit submitted to the IO thread pool for execution in StarRocks. You can think of it as a “data read job.” When a ChunkSource needs to read data from the storage layer, it does not perform the read synchronously. Instead, it encapsulates the read operation into a ScanTask, and submits this task to the global IO thread pool for asynchronous execution. The worker threads in the IO thread pool then fetch ScanTasks from the queue and perform the actual data read.</p><p>If we compare data scanning to running a restaurant: ChunkSource is the chef, and a ScanTask is the work order handed out by the chef. The IO thread pool is the back-kitchen working area. The chef does not personally go to the warehouse to carry raw ingredients (IO). Instead, he creates a work order (ScanTask) and hands it to the back-kitchen workers (IO threads). Once the ingredients are fetched, the chef continues cooking. This division of labor ensures that front-end service (Pipeline execution) is not blocked by back-office work (IO operations).</p><p>A ScanTask contains an executable function object (typically a lambda expression) which encapsulates the actual data read logic. Each ScanTask also carries metadata such as task priority, WorkGroup ownership, and performance counters — which are used for task scheduling and performance monitoring.</p><p>ScanTask plays several critical roles in StarRocks query execution and is one of the key mechanisms enabling high-concurrency, high-throughput data scanning.</p><ul><li><strong>First</strong>, it enables asynchronous IO and non-blocking execution. The primary benefit of ScanTask is that it offloads IO operations from Pipeline execution threads. In StarRocks’ pipeline model, each Pipeline thread is responsible for driving the entire operator chain. If IO were executed directly inside this thread, it would block for long periods waiting for IO completion and would be unable to process other work. With ScanTask, Pipeline threads only submit tasks and consume results, while actual IO is handled in a separate IO thread pool. This decoupling allows Pipeline threads to continue processing other Pipelines, greatly improving concurrency.</li><li><strong>Second</strong>, it enables massive IO parallelism. The global IO thread pool typically contains many worker threads. These threads can execute hundreds of ScanTasks concurrently, achieving hundreds of parallel IO operations. This is particularly important with remote storage systems (e.g. S3, HDFS), because the latency of a single request can be very high (tens to hundreds of milliseconds). High throughput can only be achieved through large-scale concurrency. For example, if each request takes 100ms, throughput is only 10 ops/sec; with 100 concurrent requests, throughput becomes 1000 ops/sec.</li></ul><h4>ChunkBuffer</h4><p>A <strong>ChunkBuffer</strong> is the buffer used in the StarRocks Pipeline execution engine to store scanned data temporarily — you can think of it as a “data transit station” or “temporary staging area.” After a ChunkSource reads data from the storage layer via ScanTask, the resulting Chunks are not immediately delivered to the ScanOperator. Instead, they are cached in the ChunkBuffer. The ScanOperator then fetches Chunks from the ChunkBuffer when needed and passes them downstream.</p><p>If we continue with the logistics analogy: the storage layer is the warehouse, the ChunkSource is the delivery vehicle, the ChunkBuffer is the staging area at the distribution station, and the ScanOperator is the worker responsible for sorting and dispatching. Delivery trucks do not directly hand merchandise to customers — they unload it into the staging area first; then operators pick up parcels, inspect, sort, and deliver them onward. The presence of this staging area allows the delivery and dispatch phases to operate asynchronously without blocking each other.</p><p>The core purpose of the ChunkBuffer is to decouple data production (IO reads) from data consumption (ScanOperator processing). IO threads quickly read data and fill the ChunkBuffer; Pipeline threads fetch data from the ChunkBuffer for processing. The two collaborate through a producer-consumer model, improving overall execution efficiency.</p><p>For example, in the diagram below, all Chunks read by ScanOperators are first placed into the ChunkBuffer, and then further processed by downstream ACC Operators.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUo-j--Fz1dLL5TsPhYvEw.png" /></figure><h3>Summary</h3><p>The diagram below illustrates a global overview of all the concepts discussed above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebyKp87yGoq4HbjbzHHysA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec67044881b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starrocks-engineering/deep-dive-to-starrocks-io-model-part-1-concepts-ec67044881b2">Deep Dive To StarRocks IO Model , Part 1 - Concepts</a> was originally published in <a href="https://medium.com/starrocks-engineering">StarRocks Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster and Cheaper: Real-time Analytics with StarRocks Slashes Costs]]></title>
            <link>https://medium.com/@tracymacding/faster-and-cheaper-real-time-analytics-with-starrocks-slashes-costs-5ceddaf5ca17?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ceddaf5ca17</guid>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[starrocks]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Tue, 23 Sep 2025 01:25:52 GMT</pubDate>
            <atom:updated>2025-09-23T01:25:52.215Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preface</h3><p>Since version 3.0, StarRocks has introduced a new architecture that separates storage and compute, enabling the persistence of compute node states to object storage. Since its release, hundreds of users have achieved significant cost reductions and efficiency gains.</p><p>To match the real-time performance of tightly integrated systems, we’ve made numerous optimizations in this architecture — including mechanisms like batch publish and the cloud-native primary key index — which help ensure 15 to 30 seconds of data freshness.</p><p>However, due to the billing model of mainstream object storage services (e.g., AWS S3 charges not only for storage capacity but also for each API call), you might encounter a hidden cost risk when using StarRocks in real-time scenarios. In some cases, you may even find that the cost of S3 API calls exceeds that of storage itself, which could be a barrier to unifying all your business workloads under the StarRocks shared data architecture.</p><p>But now, there’s no need to worry. In the upcoming version 4.0, StarRocks introduces targeted optimizations to dramatically reduce API call costs in real-time scenarios — effectively eliminating this concern and enabling fearless adoption for real-time use cases.</p><h3>Background</h3><p>Before introducing the optimizations, let’s explain why a large number of S3 API calls might occur in real-time scenarios, which ties directly to the StarRocks storage-compute separation data model.</p><p>In StarRocks, table data can be hierarchically partitioned by partition and bucket. Typically, we recommend partitioning by a time field — this allows the optimizer to prune partitions during queries, significantly reducing scanned data and improving performance.</p><p>On top of partitioning, bucketing (i.e., splitting partition data into multiple Tablets via hash) allows for a second level of pruning — narrowing queries down to just a few Tablets — thus further improving performance.</p><p>This structure is backed by a multi-version data model. Here’s a simplified example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oPcoXDyadSMdUOQJ" /></figure><p>This structure is backed by a multi-version data model. Here’s a simplified example:</p><ul><li><strong>Load Txn 1:</strong> Writes new data files (file-1 &amp; file-2) and generates Tablet Meta V1 referencing these files.</li><li><strong>Load Txn 2:</strong> Writes file-3 &amp; file-4, creates Tablet Meta V2 including all prior and new files.</li><li><strong>Load Txn 3:</strong> Adds file-5, creates Tablet Meta V3 referencing all five files.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JYt3VFm0oTX2YwF8" /></figure><p>Each import generates multiple actions per Tablet:</p><ol><li>One S3 PUT for each data file (typically one per import in real-time scenarios)</li><li>One S3 PUT for the transaction log</li><li>One S3 PUT for the new tablet meta file</li><li>One S3 DELETE for the transaction log after commit</li></ol><p>In real-time scenarios, each update results in a significant number of S3 API calls. If the table wasn’t created with targeted optimizations (e.g., it’s a non-partitioned table), and the data ingestion frequency and concurrency are high, you’ll observe a large number of S3 write API calls. This can even trigger throttling on the object storage side. Moreover, for backend storage systems like HDFS, a large volume of small files can severely degrade overall system performance.</p><p>To address this issue, we’ve implemented several optimizations, including but not limited to:</p><ol><li><strong>Batch Publish</strong>: Merges multiple transactions into a single commit, significantly reducing the number of Tablet Meta files.</li><li><strong>Merge Commit</strong>: Combines multiple data ingestion transactions into one, greatly reducing the number of file writes in high-concurrency and high-frequency ingestion scenarios.</li></ol><p>These optimizations already help the majority of users mitigate API cost issues in real-time scenarios. However, we didn’t stop there. While the above improvements focus on the control layer, in the new version we’ve implemented deep optimizations at the data I/O layer — achieving a dramatic reduction in write API calls. Now, there’s no need to worry about API costs or small file issues in real-time use cases.</p><p>Next, I’ll conduct targeted testing of the new file bundling feature introduced in version 4.0 to evaluate its effectiveness in optimizing S3 costs under high-frequency real-time scenarios.</p><h3>Effectiveness Test</h3><h3>Testing Method</h3><p>This test uses a configuration with one FE and one CN. We simulated a real-time import scenario where 200 concurrent clients used Stream Load to import data into the same table. The original data was in CSV format, with the entire dataset split into files of approximately 1MB each. There was a 30-second interval between each batch.</p><p>We created two identical tables: one with the file aggregation feature disabled and the other with it enabled. Using the system’s built-in S3 call count monitoring, we observed the number of API write calls to the object storage before and after the optimization.</p><p>The following CREATE TABLE statements were used to create the tables for the test, one with the optimization disabled and one with it enabled:</p><pre>-- Disable File Bundling<br>CREATE TABLE `lineorder` (<br>  `lo_orderkey` bigint(20) NOT NULL COMMENT &quot;&quot;,<br>  `lo_linenumber` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_custkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_partkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_suppkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_orderdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_orderpriority` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>  `lo_shippriority` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_quantity` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_extendedprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_ordtotalprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_discount` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_revenue` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_supplycost` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_tax` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_commitdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_shipmode` varchar(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP <br>DUPLICATE KEY(`lo_orderkey`)<br>COMMENT &quot;OLAP&quot;<br>DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 128 <br>PROPERTIES (<br>&quot;datacache.enable&quot; = &quot;true&quot;,<br>&quot;file_bundling&quot; = &quot;false&quot;,<br>&quot;storage_volume&quot; = &quot;builtin_storage_volume&quot;<br>);<br><br>-- Enable File Bundling<br>CREATE TABLE `lineorder` (<br>  `lo_orderkey` bigint(20) NOT NULL COMMENT &quot;&quot;,<br>  `lo_linenumber` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_custkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_partkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_suppkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_orderdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_orderpriority` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>  `lo_shippriority` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_quantity` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_extendedprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_ordtotalprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_discount` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_revenue` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_supplycost` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_tax` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_commitdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_shipmode` varchar(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP <br>DUPLICATE KEY(`lo_orderkey`)<br>COMMENT &quot;OLAP&quot;<br>DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 128 <br>PROPERTIES (<br>&quot;datacache.enable&quot; = &quot;true&quot;,<br>&#39;file_bundling&#39;=&#39;true&#39;,<br>&quot;storage_volume&quot; = &quot;builtin_storage_volume&quot;<br>);</pre><p>Adjustment made, increased compact_threads on CN to 128 for faster background merges. All other system parameters were default (out-of-the-box configuration).</p><pre>update information_schema.be_configs set value = 128 where name = &quot;compact_threads&quot;;</pre><h3>Test Results</h3><p>The chart below shows metrics such as the total number of S3 write API calls and IOPS before and after the optimization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eh_XV8cZkJiq_mp-" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VtP28x3gJUe_QswL" /></figure><p><em>Disable File Bundling</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rwye8Vx6pYpKbucO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WTqa8H308CzKVT58" /></figure><p><em>Enable File Bundling</em></p><p>S3 API call counts and IOPS metrics show that enabling file bundling reduced write API calls by ~99%, eliminating performance degradation and cost spikes from high-frequency real-time ingestion.</p><p>(Note: actual impact may vary depending on cluster size, partition/bucket count, etc. Please test in your own environment.)</p><h3>How to Use</h3><p>Starting in 4.0, file bundling is enabled by default for new tables. If you want to explicitly enable it:</p><pre>CREATE TABLE `t` (<br>  `id` bigint(20) NOT NULL COMMENT &quot;&quot;,<br>  `value` int(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP <br>DUPLICATE KEY(`id`)<br>DISTRIBUTED BY HASH(`id`) BUCKETS 1 <br>PROPERTIES (<br>&#39;file_bundling&#39;=&#39;true&#39;<br>);</pre><p>For existing tables:</p><pre>alter table test.t set (&#39;file_bundling&#39;=&#39;true&#39;);</pre><p>This change is instant and does not require rewriting data.</p><h3>Conclusion</h3><p>With version 4.0, StarRocks has addressed the final bottleneck in the storage-compute separation architecture — API costs in real-time scenarios. From now on, users can confidently adopt this architecture across all workloads, enjoying both cost-efficiency and real-time performance without compromise. Try it out and experience the difference.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ceddaf5ca17" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10× latency reduction, cold queries no longer a concern]]></title>
            <link>https://medium.com/@tracymacding/10-latency-reduction-cold-queries-no-longer-a-concern-2d56c2f9962f?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d56c2f9962f</guid>
            <category><![CDATA[bigdata-analytics]]></category>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[shared-data]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Wed, 27 Aug 2025 07:37:02 GMT</pubDate>
            <atom:updated>2025-08-27T07:38:50.747Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preface</h3><p>In today’s era of explosive data growth, since the launch of AWS S3 in 2009, object storage has gradually become the de facto standard for massive data storage. With advantages such as simple APIs, low cost, high reliability and availability, and virtually unlimited scalability, it has been widely adopted by major cloud providers and private storage vendors alike.</p><p>Now, 15 years after the debut of AWS S3, more and more services are being built on top of object storage. Because these services are inherently based on a standalone storage layer, they naturally follow a storage-compute separation architecture, which not only reduces costs but also brings extreme elasticity. Cluster scalability is no longer constrained by data capacity: with all data stored in S3, compute resources can be spun up or released at any time without the risk of data loss.</p><p>Object storage is a treasure trove, but it is not a silver bullet. In particular, I/O latency remains a challenge — typical P99 latency for a 1MB read can reach several hundred milliseconds. StarRocks’ storage-compute separation is built on top of object storage, so whenever a cache miss occurs, it must access the underlying object storage directly. In scenarios with heavy data scans, this can result in increased latency and degrade business performance.</p><p>To address these challenges, StarRocks has continued to optimize its storage-compute separation. In version 3.3.0, we introduced a number of improvements, most notably the intra-Tablet parallel scan capability. This feature breaks down large, sequential I/O into multiple smaller I/Os, which are then processed in parallel using a thread pool — significantly improving I/O efficiency. This article demonstrates the query performance improvements enabled by this new mechanism.</p><h3>Performance Comparison</h3><h3>Test Scenarios</h3><p>We know what everyone cares about most: performance before and after cold query optimization. To “show the horse in action,” we designed three test scenarios based on the SSB 1TB dataset (see Appendix for table schema). All tests were run on the latest version of StarRocks storage-compute separation.</p><ol><li><strong>With Data Cache disabled</strong>: comparing query performance before and after enabling intra-Tablet parallel scan on the SSB 1TB dataset.</li><li><strong>With Data Cache enabled</strong>: comparing query performance before and after enabling intra-Tablet parallel scan on the SSB 1TB dataset.</li><li><strong>In storage-compute integrated mode</strong>: comparing query performance before and after enabling intra-Tablet parallel scan on the SSB 1TB dataset.</li></ol><p>In the tests, each Tablet was controlled to be around 4–5GB in size.</p><h3>Test Resources</h3><p>The test environment used AWS EC2 with 1 FE + 5 CN nodes. The compute node specifications are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*gPRCfp0fqcdLv949AaZJOQ.png" /></figure><h3>Test Result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrRnGkrWLf7zPj_fq62gcQ.png" /></figure><p><strong>Notes:</strong></p><ol><li><strong>Cache Disabled — Before Optimization</strong>: Data Cache disabled in StarRocks storage-compute separation, and Tablet parallel scan disabled.</li><li><strong>Cache Disabled — After Optimization</strong>: Data Cache disabled in StarRocks storage-compute separation, and Tablet parallel scan enabled.</li><li><strong>Cache Enabled — Before Optimization</strong>: Data Cache enabled in StarRocks storage-compute separation, and Tablet parallel scan disabled.</li><li><strong>Cache Enabled — After Optimization</strong>: Data Cache enabled in StarRocks storage-compute separation, and Tablet parallel scan enabled.</li><li><strong>Storage-Compute Integrated</strong>: Query performance in integrated storage-compute mode with parallel scan optimization disabled.</li><li><strong>Storage-Compute Integrated — With Parallel Scan Optimization</strong>: Query performance in integrated storage-compute mode with parallel scan optimization enabled.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/0*zVPrrJIqhfTWJEjq" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tVr9FXrInzois6U8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-3vsVRRI0mlYsM21" /></figure><h3>Test Conclusion</h3><ol><li>For the Cache Disabled scenario, after enabling parallel scan and related optimizations, query latency dropped significantly — by 5 to 20×.</li><li>With parallel scan and other optimizations enabled, the performance gap between Cache Disabled and the storage-compute integrated mode has narrowed considerably (overall difference is only about 2.x×).</li><li>For the Cache Enabled scenario, performance is almost identical to that of the storage-compute integrated mode when both have parallel scan and related optimizations enabled.</li></ol><h3>Conclusion</h3><p>Although the above benchmark tests on a standard dataset are not fully comprehensive, they do validate the feasibility of mitigating the side effects of I/O latency when using object storage as the storage medium through technical optimizations. Moreover, one major advantage brought by this parallel optimization is that it allows each Tablet to manage larger amounts of data.</p><p>Previously, to improve parallelism, we typically recommended a Tablet size of about 1GB. However, with the intra-Tablet parallel scan technology, users are no longer bound by this limitation. (For example, in the tests presented in this article, each Tablet managed around 5GB of data.) This enables increasing Tablet size while reducing the number of Tablets.</p><p>Fewer Tablets not only reduce memory consumption on the FE, but also significantly lower the number of I/O operations required to access object storage during data writes — resulting in both performance improvements and cost reductions, achieving multiple benefits at once.</p><p>Finally, if you are interested in this feature, you can download the 3.3.0 Release version and enable it with the following two configuration commands to experience a truly <em>“flying”</em> performance boost:</p><pre>set enable_lake_tablet_internal_parallel = true;<br>set tablet_internal_parallel_mode = &quot;force_split&quot;;</pre><h3>Appendix</h3><p>The table creation statement used in the tests is as follows:</p><pre>CREATE TABLE IF NOT EXISTS `lineorder` (<br>    `lo_orderkey` bigint(20) NOT NULL COMMENT &quot;&quot;,<br>    `lo_linenumber` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_custkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_partkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_suppkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_orderdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_orderpriority` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>    `lo_shippriority` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_quantity` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_extendedprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_ordtotalprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_discount` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_revenue` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_supplycost` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_tax` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_commitdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `lo_shipmode` varchar(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP<br>DUPLICATE KEY(`lo_orderkey`)<br>COMMENT &quot;OLAP&quot;<br>PARTITION BY RANGE(`lo_orderdate`)<br>(<br>    PARTITION p1 VALUES [(&quot;-2147483648&quot;), (&quot;19930101&quot;)),<br>    PARTITION p2 VALUES [(&quot;19930101&quot;), (&quot;19940101&quot;)),<br>    PARTITION p3 VALUES [(&quot;19940101&quot;), (&quot;19950101&quot;)),<br>    PARTITION p4 VALUES [(&quot;19950101&quot;), (&quot;19960101&quot;)),<br>    PARTITION p5 VALUES [(&quot;19960101&quot;), (&quot;19970101&quot;)),<br>    PARTITION p6 VALUES [(&quot;19970101&quot;), (&quot;19980101&quot;)),<br>    PARTITION p7 VALUES [(&quot;19980101&quot;), (&quot;19990101&quot;)))<br>DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 5<br>PROPERTIES (<br>    &quot;datacache.enable&quot; = &quot;false&quot;<br>);<br><br>CREATE TABLE IF NOT EXISTS `customer` (<br>    `c_custkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `c_name` varchar(26) NOT NULL COMMENT &quot;&quot;,<br>    `c_address` varchar(41) NOT NULL COMMENT &quot;&quot;,<br>    `c_city` varchar(11) NOT NULL COMMENT &quot;&quot;,<br>    `c_nation` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>    `c_region` varchar(13) NOT NULL COMMENT &quot;&quot;,<br>    `c_phone` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>    `c_mktsegment` varchar(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP<br>DUPLICATE KEY(`c_custkey`)<br>COMMENT &quot;OLAP&quot;<br>DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 5<br>PROPERTIES (<br>    &quot;datacache.enable&quot; = &quot;false&quot;<br>);<br>CREATE TABLE IF NOT EXISTS `dates` (<br>    `d_datekey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_date` varchar(20) NOT NULL COMMENT &quot;&quot;,<br>    `d_dayofweek` varchar(10) NOT NULL COMMENT &quot;&quot;,<br>    `d_month` varchar(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_year` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_yearmonthnum` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_yearmonth` varchar(9) NOT NULL COMMENT &quot;&quot;,<br>    `d_daynuminweek` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_daynuminmonth` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_daynuminyear` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_monthnuminyear` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_weeknuminyear` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_sellingseason` varchar(14) NOT NULL COMMENT &quot;&quot;,<br>    `d_lastdayinweekfl` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_lastdayinmonthfl` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_holidayfl` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `d_weekdayfl` int(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP<br>DUPLICATE KEY(`d_datekey`)<br>COMMENT &quot;OLAP&quot;<br>DISTRIBUTED BY HASH(`d_datekey`) BUCKETS 1<br>PROPERTIES (<br>    &quot;datacache.enable&quot; = &quot;false&quot;<br>);<br>CREATE TABLE IF NOT EXISTS `part` (<br>    `p_partkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `p_name` varchar(23) NOT NULL COMMENT &quot;&quot;,<br>    `p_mfgr` varchar(7) NOT NULL COMMENT &quot;&quot;,<br>    `p_category` varchar(8) NOT NULL COMMENT &quot;&quot;,<br>    `p_brand` varchar(10) NOT NULL COMMENT &quot;&quot;,<br>    `p_color` varchar(12) NOT NULL COMMENT &quot;&quot;,<br>    `p_type` varchar(26) NOT NULL COMMENT &quot;&quot;,<br>    `p_size` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `p_container` varchar(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP<br>DUPLICATE KEY(`p_partkey`)<br>COMMENT &quot;OLAP&quot;<br>DISTRIBUTED BY HASH(`p_partkey`) BUCKETS 5<br>PROPERTIES (<br>    &quot;datacache.enable&quot; = &quot;false&quot;<br>);<br>CREATE TABLE IF NOT EXISTS `supplier` (<br>    `s_suppkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>    `s_name` varchar(26) NOT NULL COMMENT &quot;&quot;,<br>    `s_address` varchar(26) NOT NULL COMMENT &quot;&quot;,<br>    `s_city` varchar(11) NOT NULL COMMENT &quot;&quot;,<br>    `s_nation` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>    `s_region` varchar(13) NOT NULL COMMENT &quot;&quot;,<br>    `s_phone` varchar(16) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP<br>DUPLICATE KEY(`s_suppkey`)<br>COMMENT &quot;OLAP&quot;<br>DISTRIBUTED BY HASH(`s_suppkey`) BUCKETS 5<br>PROPERTIES (<br>    &quot;datacache.enable&quot; = &quot;false&quot;<br>);</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d56c2f9962f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[StarRocks Compaction Principle on Shared-Data Architecture]]></title>
            <link>https://medium.com/starrocks-engineering/starrocks-compaction-principle-on-shared-data-93f673ab0c3e?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/93f673ab0c3e</guid>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Tue, 05 Aug 2025 16:10:14 GMT</pubDate>
            <atom:updated>2025-12-12T01:48:46.400Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ugubt488pwvcweYa" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Explanation</h3><p>In StarRocks, each data ingestion generates a new data version, and querying requires merging all versioned data to obtain accurate results. If there are too many historical data versions, the number of files to be read during queries will also be high, leading to decreased query efficiency. Therefore, StarRocks includes internal tasks to periodically consolidate historical data versions, eliminating duplicate data records, a process referred to as Compaction.</p><p>Compaction aims to consolidate different versions of data files into larger files, reducing the number of small files in the system and thereby improving query efficiency. Compared to traditional combined storage and computing tables, StarRocks’ storage and computing separation implements a new Compaction scheduling mechanism, as follows:</p><ol><li>Compaction scheduling is initiated by the Frontend (FE) and executed by the Backend (BE). The FE initiates Compaction tasks based on Partitions.</li><li>Compaction generates a new version and follows the complete process of data import, data writing, commit, and version publishing.</li></ol><p>This document aims to describe the basic implementation principles of Compaction in StarRocks’ storage and computing separation tables, helping developers and operations personnel better understand and adjust Compaction-related configurations according to practical needs to achieve better results.</p><h3>Background</h3><p>As mentioned earlier, each import operation generates a new version within the FE, which is marked on the Partition. Once an import transaction is successfully committed, the visible data version number of the Partition is updated, and the data version number of the Partition increases monotonically.</p><p>It is important to note that a Partition may contain multiple Tablets, all sharing the same data version number. Even if an import operation may involve only a subset of these Tablets, once the import transaction is successfully committed, the versions of all Tablets under the Partition are correspondingly incremented.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3rMG7L8hIuyqNdr3" /></figure><p>For example, in the above diagram, Partition X contains Tablet 1 to N, and the current visible version is 12. Once a new import transaction, “New Load Txn,” is successfully committed, the visible version of Partition X becomes 13.</p><h3>Basic Framework</h3><p>The Compaction of StarRocks’ storage and computing separation tables consists of two roles: the Scheduler (Compaction Scheduler) and the Executor (Compaction Executor). The Scheduler initiates Compaction tasks via RPC, while the Executor is responsible for executing these tasks.</p><p>In StarRocks’ storage and computing separation, the FE acts as the Compaction Scheduler, while the BE or CN serves as the Compaction Executor. Each Compaction Executor contains a dedicated thread pool for executing Compaction tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*haQlfjslwurPyJuM" /></figure><h3>Compaction Scheduler Scheduling</h3><p>The FE has a periodically running thread called the Compaction Scheduler, responsible for scheduling all Compaction tasks. The FE uses Partition as the basic unit for scheduling.</p><p>The FE holds information about the Compaction Score of each Partition, which represents the priority of Compaction for all Tablets within the Partition. The higher the Compaction Score, the more urgent the need for consolidation of the Partition.</p><p>Each time the Compaction Scheduler thread runs, it selects the Partition with the highest Compaction Score and constructs Compaction tasks for these Partitions. Of course, the Compaction Scheduler also controls the maximum number of Compaction tasks initiated each time.</p><p>The logic for constructing Compaction tasks is relatively straightforward. For each Partition, the Scheduler obtains all its Tablets, then constructs a Compaction task for each CN. The task contains the list of Tablets that need to execute the Compaction task on that CN and is then sent to the CN node.</p><p>The entire process is illustrated in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VvuUEM9P2HaFsYHF" /></figure><p>In the above diagram, there are two Partitions on the FE that need to execute Compaction, namely Partition X and Partition Y. Partition X contains 4 Tablets (1 to 4), while Partition Y contains 3 Tablets (5 to 7).</p><p>The Scheduler determines the following:</p><ul><li>For Partition X, Tablet-2 and Tablet-4 are located in the same CN-1, while Tablet-1 and Tablet-3 are located in the same CN-2. Therefore, two Compaction tasks (Task-1 and Task-2) are constructed for Partition X. Task-1 contains Tablet-2 and Tablet-4, while Task-2 contains Tablet-1 and Tablet-3.</li><li>For Partition Y, Tablet-5 and Tablet-7 are located in the same CN-1, while Tablet-6 is located in another CN-2. Therefore, two Compaction tasks (Task-3 and Task-4) are also constructed for Partition Y. Task-3 contains Tablet-5 and Tablet-7, while Task-4 contains Tablet-6.</li></ul><p>Finally, each task is sent to its respective CN.</p><h3>Compaction Executor Task Execution</h3><p>As mentioned earlier, the Compaction Scheduler in the FE generates Compaction tasks and sends them to CN/BE nodes. CN/BE nodes have dedicated thread pools to handle these tasks, and the number of thread pools can be configured (dynamic configuration support is upcoming).</p><p>Each thread retrieves the next task to be executed from the Compaction Task task queue.</p><h3>Compaction Task Execution Principle</h3><p>Compaction essentially involves organizing and merging multiple data files, deleting duplicate records, and creating a larger data file, as shown in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qq8V5MwMaMXsbeBo" /></figure><p>For example, in the above image, after Compaction of version 1 and version 2 data files, the outdated data from version 1 (id = 2, value = 11, id = 5, value = 30) is eliminated, resulting in the creation of a new data version file, version 3.</p><h3>Multi Version File Compaction</h3><p>Currently, StarRocks’ storage and computing separation tables utilize multi-version data technology, and the overall storage structure is illustrated in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HB40lSew54ZOgDrl" /></figure><p>In the above diagram, three data import transactions have been generated:</p><ul><li>Load Txn 1: During the data writing phase of the transaction, new data files file 1 &amp; file 2 are created. After the transaction is committed, Tablet Meta V1 is generated, recording the visible file list for this version as {file-1, file-2}.</li><li>Load Txn 2: During the data writing phase of the transaction, new data files file 3 &amp; file 4 are created. Upon submission, based on the previous version (i.e., Tablet Meta V1), and adding the new data files generated in this import transaction (file-3 &amp; file-4), a new Tablet Meta V2 is created. Therefore, the visible file list for this version is {file-1, file-2, file-3, file-4}.</li><li>Load Txn 3: During the data writing phase of the transaction, a new data file file 5 is generated. Upon submission, based on the previous version (i.e., Tablet Meta V2), and adding the new data file generated in this import transaction (file-5), a new Tablet Meta V3 is created. Therefore, the visible file list for this version is {file-1, file-2, file-3, file-4, file-5}.</li></ul><p>In addition to user import transactions generating new data versions, system background Compaction tasks in the storage and computing separation tables also generate new data versions. The purpose of Compaction is twofold: 1) to merge multiple versions of small files into larger files, reducing the number of random IO operations during queries, and 2) to eliminate duplicate data records, reducing the overall data volume.</p><p>In the storage and computing separation tables, each Compaction also generates an entirely new version. Using the example above, if after Txn 3, a new transaction Txn 4 is a Compaction task and merges the 4 files file1 ~ file4 into file-6, then upon submission, the new version Tablet Meta V4 records the file list as {file-5, file-6}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XIoWFavCGitIAXvz" /></figure><p>Upon observation and consideration of the example, it becomes apparent that if the system never performs Compaction during its operation, the data files in the system can never be deleted. In the example above, Tablet Meta V1 and Tablet Meta V2 files could be deleted, but we cannot delete file-1, file-2, file-3, and file-4 because these files are still referenced by Tablet Meta V3.</p><p>However, with data merging (Compaction), the situation changes. In the example above, due to a single Compaction (Compact Txn 4 in the diagram), file-1, file-2, file-3, and file-4 are merged to create the new file file-6 and generate the new Tablet Meta V4. As the content of file-1 to file-4 already exists in file-6, once versions V1, V2, V3 are no longer accessed, file-1 to file-4 can be safely deleted. The data version situation at this point is as shown in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l6I1yRoAwV5zhLyt" /></figure><p>Therefore, from the above discussion, it is evident that only after Compaction is completed can the original data files be deleted (although the cleaning of Tablet Meta files depends on other rules). Therefore, the most straightforward rule for determining whether a data file can be safely deleted is that the data file is no longer referenced by any Tablet Meta.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93f673ab0c3e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starrocks-engineering/starrocks-compaction-principle-on-shared-data-93f673ab0c3e">StarRocks Compaction Principle on Shared-Data Architecture</a> was originally published in <a href="https://medium.com/starrocks-engineering">StarRocks Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make ingestion smooth, giving you peace]]></title>
            <link>https://medium.com/@tracymacding/make-ingestion-smooth-giving-you-peace-6b3bbf5c52b6?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b3bbf5c52b6</guid>
            <category><![CDATA[lake-house]]></category>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[customer-facing-analytics]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Thu, 03 Apr 2025 11:35:56 GMT</pubDate>
            <atom:updated>2025-04-03T11:35:56.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>Preface</h3><p>With the advancement of technology, we have observed that more and more users are beginning to experience real-time data analytics. StarRocks could be the best choice for achieving real-time data analytics. Even in StarRocks’ storage-compute separation architecture, users can still experience data freshness within 15–30 seconds while supporting high-concurrency data ingestion. We have done significant work to help users achieve this capability.</p><p>However, for newcomers to StarRocks, achieving such high-concurrency and real-time analytics inevitably requires some manual system tuning, with Compaction being one of the most critical aspects.</p><p>In StarRocks’ storage-compute separation mode, each data ingestion generates a new data version. To obtain the correct results during queries, all historical versions data must be merged. If too many historical versions accumulate, the system will need to read an excessive number of files during queries, impacting query performance. To address this, StarRocks has an internal Compaction task that periodically consolidates historical versions to eliminate redundant data records which we call it compaction.</p><p>Therefore, in real-time analytics scenarios, it is essential to closely monitor the Compaction status in the system (measured by Compaction Score). If the Compaction Score of a partition exceeds a specific threshold (e.g., 100 or more), it requires attention. This indicates that the ingestion frequency is too high, surpassing the system’s background compaction capability. At this point, there are two possible actions:</p><ol><li>Reduce the data ingestion frequency (e.g., increase batch size).</li><li>Enhance the system’s background compaction capability.</li></ol><p>This poses a significant challenge for system operations, as administrators may even need to wake up at night to handle these issues.</p><h3>Automatic Ingestion Rate Limiting</h3><p>To prevent system instability caused by high-frequency ingestion, StarRocks introduces automatic ingestion rate limiting (AIRL) in latest version. When the system detects that the ingestion frequency exceeds the compaction capacity, it will automatically control the ingestion speed to maintain balance. Once the system’s compaction rate increases, the ingestion speed will automatically recover.</p><p>This mechanism protects query efficiency while also reducing manual intervention through automation, significantly easing operational burdens and enhancing user experience. It is important to note that this feature currently only applies to StarRocks’ storage-compute separation mode and is enabled by default in version 3.3.</p><h3>Some Parameters</h3><p>The following parameters control the ingestion rate limiting capability.</p><p><strong>lake_enable_ingest_slowdown</strong></p><p>Controls whether to enable the ingestion rate limiting feature. It is an FE configuration and must be enabled for other related parameters to take effect.</p><p>Command to dynamically enable/disable it on the Leader FE:</p><pre>-- Enable ingestion rate limiting<br>admin set frontend config(&quot;lake_enable_ingest_slowdown&quot;=&quot;true&quot;);<br><br>-- Disable ingestion rate limiting<br>admin set frontend config(&quot;lake_enable_ingest_slowdown&quot;=&quot;false&quot;)</pre><p><strong>lake_ingest_slowdown_threshold</strong></p><p>Defines the Compaction Score threshold for triggering ingestion rate limiting. If the Compaction Score of any partition exceeds this threshold, the system will slow down ingestion. Default value is 100.</p><p>You can check the Compaction Score of a specific partition by querying the partitions_meta table in information_schema. The MaxCS column represents the partition’s Compaction Score.</p><p>If an ingestion task involves multiple partitions, the system uses the maximum Compaction Score among them and compares it to lake_ingest_slowdown_threshold.</p><p>For example, if an ingestion task affects two partitions with Compaction Scores of 50 and 200, since 200 &gt; <em>lake_ingest_slowdown_threshold</em>, ingestion rate limiting will be triggered.</p><p>Command to dynamically configure it on the Leader FE:</p><pre>admin set frontend config(&quot;lake_ingest_slowdown_threshold&quot;=&quot;100&quot;);</pre><p><strong>lake_ingest_slowdown_ratio</strong></p><p>Defines the delay ratio applied to ingestion transactions when the Compaction Score exceeds the lake_ingest_slowdown_threshold. Default value: 0.1. This parameter controls how much time is added to the commit delay for an ingestion transaction.</p><p>For example, If an ingestion task takes 10 minutes to complete the write phase, and the current Compaction Score is 105, with <em>lake_ingest_slowdown_threshold</em> set to 100 and <em>lake_ingest_slowdown_ratio</em> set to 0.1:</p><pre>Delay Time = (105 - 100) * 0.1 * 10min = 0.5 * 10min = 5min</pre><p>This means the commit will be delayed by 5 minutes.</p><p>If the delayed commit time exceeds the transaction timeout, the transaction will abort instead of waiting indefinitely.</p><p>Command to dynamically configure it on the Leader FE:</p><pre>admin set frontend config(&quot;lake_ingest_slowdown_ratio&quot;=&quot;0.05&quot;);</pre><p><strong>lake_compaction_score_upper_bound</strong></p><p>Defines the upper limit of the Compaction Score. If any partition’s Compaction Score exceeds this value, ingestion transactions will be blocked from committing until the Compaction Score drops below this threshold. Default value is 2000</p><p>Command to dynamically configure it on the Leader FE:</p><pre>admin set frontend config(&quot;lake_compaction_score_upper_bound&quot;=&quot;1000&quot;);</pre><h3>Examples</h3><p><strong>Example 1: Basic Ingestion Rate Limiting</strong></p><p>Assume ingestion rate limiting is enabled with default parameter values.An ingestion task starts at 12:00:00 and finishes writing at 12:05:00 (write duration is 5 minutes). At 12:05:00, the Compaction Score is 110. So we can caculate the Commit Delay Calculation:</p><pre>Delay Time = 5min * (110 - 100) * 0.1 = 5min</pre><p>This means this ingestion task will be committed at 12:10:00.</p><p>At 12:10:00, the task commits successfully, regardless of the current Compaction Score.</p><p><strong>Example 2: Ingestion Blocking Due to High Compaction Score</strong></p><p>Assume ingestion rate limiting is enabled, with <em>lake_compaction_score_upper_bound</em> set to 150. An ingestion task starts at 12:00:00, finishes writing at 12:05:00 (write duration is 5 minutes). At 12:05:00, the Compaction Score is 110, so the commit is delayed by 5 minutes. At 12:10:00, the Compaction Score has increased to 160, exceeding <em>lake_compaction_score_upper_bound </em>value 150.</p><p>So this task is blocked in commit phase and retries every second until the Compaction Score drops below 150. If the transaction times out, the ingestion fails and aborts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b3bbf5c52b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[S3 Express One: One small step for S3, One giant leap for StarRocks]]></title>
            <link>https://medium.com/@tracymacding/s3-express-one-one-small-step-for-s3-one-giant-leap-for-starrocks-ace46ea21ec9?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/ace46ea21ec9</guid>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Sun, 08 Sep 2024 02:19:16 GMT</pubDate>
            <atom:updated>2024-10-30T02:16:31.007Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>As we all know, in big data era, since the launch of AWS S3 in 2009, object storage has gradually become the de facto standard for large amounts of data storage, thanks to its simple API, low cost, high reliability and availability, and massive scalability. Major cloud providers and private storage service suppliers have also introduced similar services.</p><p>With AWS S3 now in its 15th year, more and more services are being built on object storage. Since these services are built on independent storage, they naturally follow a storage-compute separation architecture, not only saves costs but also brings extreme elasticity. The scale of a user’s cluster is no longer constrained by data capacity, as all data is stored on S3, and compute resources can be scaled up or down at any time without the risk of data loss.</p><p>Object storage has many advantages, but it is not silver bullet. Early object storage systems were mainly designed for storing large amounts of data, such as images and videos, with a focus on low cost, high throughput, and high reliability. A typical scenario involves enterprises storing images and videos on these systems, but with relatively low access QPS and low sensitivity to latency. To handle high concurrent traffic and provide lower latency for accessing hot images and videos, enterprises typically build a CDN in front of the object storage.</p><p>However, with the rise of new data processing paradigms in recent years, such as lakehouse analytics and unified stream-batch processing, we have also discovered certain limitations of object storage that restrict its capabilities in these areas. These limitations are reflected in the following aspects:</p><p>1. <strong>High Latency:</strong> Read and write latencies of tens or even hundreds of milliseconds make object storage increasingly unacceptable for latency-sensitive applications.</p><p>2. <strong>QPS Limits:</strong> While object storage prioritizes high throughput, there are clear limits on QPS. For example, AWS S3 has a maximum write QPS of approximately 3,500 for a single bucket and a maximum read QPS of only 5,500. Furthermore, these limits require careful prefix design to ensure that reads and writes don’t occur in the same prefix.</p><p>3. <strong>Pricing Pitfalls:</strong> When we say object storage is cheap, we usually mean storage costs are low. However, the pricing model for object storage also includes charges for API call frequencies, which many people may overlook and encounter issues. In many real-time applications, the cost of API calls can exceed storage costs, forcing many applications built on AWS S3 to optimize their design to better control costs.</p><p>4. <strong>Poor</strong> <strong>List Performance:</strong> Applications developed using traditional POSIX interfaces often face poor list performance when migrating to object storage, requiring developers to redesign their applications, which incurs significant costs.</p><p>Fortunately, AWS has been working to improve these issues. For example, at the 2023 AWS Re:Invent conference, it introduced <strong>AWS Express One Storage</strong>, specifically designed to address the typical issues of S3 mentioned above, offering a better user experience for certain applications.</p><p>For a detailed introduction to AWS S3 Express One Storage, you can refer to the relevant documentation. StarRocks is also continuously tracking cutting-edge technology and has supported AWS S3 Express One Storage as a backend storage. Today, we will provide a detailed introduction on this.</p><h3><strong>Usage</strong></h3><p>Experiencing S3 Express One Storage in StarRocks is very simple. First, prepare a StarRocks shared data cluster, and then you need to enable the AWS S3 service (currently, only AWS S3 supports this capability).</p><p>Once all the prerequisites are ready, you only need to execute a command in StarRocks to create a Storage Volume based on Express One Zone as the backend. The command is as follows:</p><pre>CREATE STORAGE VOLUME s3_express_one_vol<br>        TYPE = S3<br>        LOCATIONS = (&quot;s3://dingkai-test--usw2-az1--x-s3/&quot;)<br>        PROPERTIES (&quot;aws.s3.region&quot; = &quot;us-west-2&quot;, &quot;aws.s3.access_key&quot;=&quot;xxx&quot;, &quot;aws.s3.secret_key&quot;=&quot;yyy&quot;, &quot;aws.s3.endpoint&quot;=&quot;https://s3express-usw2-az1.us-west-2.amazonaws.com&quot;, &quot;enabled&quot;=&quot;true&quot;);</pre><p>It is important to note that the endpoint, region, and other details for Express One Storage have some unique specifications. You can refer to this link to learn more:</p><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-Regions-and-Zones.html"><em>https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-Regions-and-Zones.html</em></a></p><p>Other fields are identical to a standard S3 Bucket.</p><p>For more details on StarRocks’ Storage Volume capabilities and usage, refer to the documentation.</p><h3><strong>Performance</strong></h3><p>We know that what everyone cares most about is the performance of S3 Express One Zone Storage. To truly evaluate it, we designed the following three test scenarios. All tests were conducted using the latest version of StarRocks with a storage-compute separation architecture:</p><p>1. High-frequency real-time ingestion</p><p>2. Offline data ingestion</p><p>3. Cold data queries</p><p>The testing method is straightforward. We created two independent Storage Volumes: one using a regular S3 bucket, and the other based on an S3 Express One Zone Storage bucket. Then, we created different tables based on these two Storage Volumes to test their performance (thanks to Storage Volumes, we have the flexibility to experiment freely).</p><pre>mysql&gt; show storage volumes;<br>+------------------------+<br>| Storage Volume         |<br>+------------------------+<br>| builtin_storage_volume |<br>| jeff_s3_express_bucket |<br>+------------------------+<br><br>mysql&gt; desc storage volume builtin_storage_volume\G<br>*************************** 1. row ***************************<br>     Name: builtin_storage_volume<br>     Type: S3<br>IsDefault: true<br> Location: s3://starrocks-common/skr2veiad-jeff_load_test-1699877456738<br>   Params: {&quot;aws.s3.region&quot;:&quot;us-west-2&quot;,&quot;aws.s3.use_instance_profile&quot;:&quot;true&quot;,&quot;aws.s3.use_aws_sdk_default_behavior&quot;:&quot;false&quot;,&quot;aws.s3.endpoint&quot;:&quot;https://s3.us-west-2.amazonaws.com&quot;}<br>  Enabled: true<br>  Comment:<br>1 row in set (0.19 sec)<br><br>mysql&gt; desc storage volume jeff_s3_express_bucket\G<br>*************************** 1. row ***************************<br>     Name: jeff_s3_express_bucket<br>     Type: S3<br>     IsDefault: false<br>     Location: s3://dingkai-test--usw2-az1--x-s3/<br>     Params: {&quot;aws.s3.access_key&quot;:&quot;******&quot;,&quot;aws.s3.secret_key&quot;:&quot;******&quot;,&quot;aws.s3.endpoint&quot;:&quot;https://s3express-usw2-az1.us-west-2.amazonaws.com&quot;,&quot;aws.s3.region&quot;:&quot;us-west-2&quot;,&quot;aws.s3.use_instance_profile&quot;:&quot;false&quot;,&quot;aws.s3.use_aws_sdk_default_behavior&quot;:&quot;false&quot;}<br>     Enabled: true<br>     Comment: s3 express one zone type bucket test by jeff<br><br>1 row in set (0.20 sec)</pre><h4><strong>High-Frequency Real-Time Ingestion</strong></h4><p>In the test, we set different client concurrency levels and continuously imported data into StarRocks via the stream load method, with each batch being 1MB in size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNv7_pWGiBXFLpJ6iDTlWw.png" /></figure><p>During the test, we used built-in monitoring system to observe the write I/O latency for the different storage bucket types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YtXqrxOLMYXF4bnSfI26Rg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3q1KjRUkeoeNvEq6IIInyw.png" /></figure><p>Let’s keep you in suspense for a moment — can you guess which one is the standard object storage and which one is Express One?</p><h3><strong>Batch Data Ingestion</strong></h3><p>We used Broker Load to import the largest table, store_sales, from the TPC-DS 1TB dataset.</p><pre>mysql&gt; show load where Label = &quot;store_sales_1_2&quot;\G<br>*************************** 1. row ***************************<br>         JobId: 19169<br>         Label: store_sales_1_2<br>         State: FINISHED<br>      Progress: ETL:100%; LOAD:100%<br>          Type: BROKER<br>      Priority: NORMAL<br>      ScanRows: 2879987999<br>  FilteredRows: 0<br>UnselectedRows: 0<br>      SinkRows: 2879987999<br>       EtlInfo: NULL<br>      TaskInfo: resource:N/A; timeout(s):144000; max_filter_ratio:0.0<br>      ErrorMsg: NULL<br>    CreateTime: 2024-04-17 18:59:55<br>  EtlStartTime: 2024-04-17 18:59:59<br> EtlFinishTime: 2024-04-17 18:59:59<br> LoadStartTime: 2024-04-17 18:59:59<br>LoadFinishTime: 2024-04-17 19:01:59<br>   TrackingSQL: <br>    JobDetails: {&quot;All backends&quot;:{&quot;8a4ed954-78bc-4930-bc74-c9d64d9e5cdf&quot;:[14752,14753,14754,14755,13283,14756,13284,14757,13285,13286,13287,10004,10005,10006,14750,14751]},&quot;FileNumber&quot;:10,&quot;FileSize&quot;:421287950369,&quot;InternalTableLoadBytes&quot;:481093008697,&quot;InternalTableLoadRows&quot;:2879987999,&quot;ScanBytes&quot;:421287950369,&quot;ScanRows&quot;:2879987999,&quot;TaskNumber&quot;:1,&quot;Unfinished backends&quot;:{&quot;8a4ed954-78bc-4930-bc74-c9d64d9e5cdf&quot;:[]}}<br>     Warehouse: default_warehouse<br>1 row in set (0.20 sec)<br><br><br>mysql&gt; show load where Label = &quot;store_sales_2_3&quot;\G<br>*************************** 1. row ***************************<br>         JobId: 19180<br>         Label: store_sales_2_3<br>         State: FINISHED<br>      Progress: ETL:100%; LOAD:100%<br>          Type: BROKER<br>      Priority: NORMAL<br>      ScanRows: 2879987999<br>  FilteredRows: 0<br>UnselectedRows: 0<br>      SinkRows: 2879987999<br>       EtlInfo: NULL<br>      TaskInfo: resource:N/A; timeout(s):144000; max_filter_ratio:0.0<br>      ErrorMsg: NULL<br>    CreateTime: 2024-04-17 19:04:45<br>  EtlStartTime: 2024-04-17 19:04:49<br> EtlFinishTime: 2024-04-17 19:04:49<br> LoadStartTime: 2024-04-17 19:04:49<br>LoadFinishTime: 2024-04-17 19:07:00<br>   TrackingSQL: <br>    JobDetails: {&quot;All backends&quot;:{&quot;4c534297-b69a-42b0-9a41-120f0ad1b0ca&quot;:[14752,14753,14754,14755,13283,13284,14756,13285,14757,13286,13287,10004,10005,10006,14750,14751]},&quot;FileNumber&quot;:10,&quot;FileSize&quot;:421287950369,&quot;InternalTableLoadBytes&quot;:481093008697,&quot;InternalTableLoadRows&quot;:2879987999,&quot;ScanBytes&quot;:421287950369,&quot;ScanRows&quot;:2879987999,&quot;TaskNumber&quot;:1,&quot;Unfinished backends&quot;:{&quot;4c534297-b69a-42b0-9a41-120f0ad1b0ca&quot;:[]}}<br>     Warehouse: default_warehouse<br>1 row in set (0.19 sec)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INmU9I1Bxs4Scc5GxLdmdQ.png" /><figcaption><strong>S3 Express One Zone Storage Read IO</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6IhDXJQVV5XIZ6TtPQh2Q.png" /><figcaption><strong>S3 Standard Storage Read IO</strong></figcaption></figure><p>During the test, it was observed that the CPU on the CN nodes was nearly fully utilized.</p><h3><strong>Query</strong></h3><p>For the test, we selected the TPC-DS 1TB dataset and disabled all caches during the process (both memory-based page cache and local disk-based data cache). We compared the query performance across three scenarios: using a standard S3 bucket, an S3 Express One Zone Storage bucket, and local cache hit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WIMmU5TaFlZ5kj61QlXmyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*90ktIMdtaa8oJzSSjgCjTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ScVkCF2EWL3I__grvqL73w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DZRG83g97CFm3KmGuVexLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5G0aSrWck-XBz2poXfOY1Q.png" /></figure><p>As usual, we also carefully recorded the I/O read latency for both types of storage</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LcKcJ_aWi6ytMcrCPgn0AA.png" /><figcaption><strong>S3 Express One Zone Storage Read IO</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-y2hS1eL1tJgHCprnLLHg.png" /><figcaption><strong>S3 Standard Storage Read IO</strong></figcaption></figure><h4><strong>Some Conclusions</strong></h4><p>1. AWS S3 Express One offers lower I/O latency compared to standard buckets, with particularly stable P99 latency control (a big win!).</p><p>2. AWS S3 Express One Storage has a higher IOPS limit than standard buckets, effectively alleviating the S3 I/O throttling issues commonly seen in real-time applications (in the real-time ingestion scenario, many requests on standard S3 buckets experienced IOPS throttling).</p><p>3. For batch data processing scenarios, standard S3 buckets already provide sufficient bandwidth, making them adequate for such use cases.</p><p>4. Overall, AWS S3 Express One’s low-latency and high-concurrency characteristics make it well-suited for real-time and cache-heavy scenarios, providing a better user experience.</p><h3>At Last</h3><p><strong>Of course, cloud providers don’t offer free lunches.</strong> Compared to standard S3 storage, the new S3 storage type has some differences in terms of pricing, even though the primary cost structure remains based on storage capacity and API call fees:</p><p>1. <strong>Significantly higher storage costs</strong>: Compared to standard S3 buckets, storage fees are much higher. For standard S3 buckets, the cost for the first 50TB of storage is just $0.023/GB per month. In contrast, S3 Express One is $0.16/GB per month, which is seven times more expensive. By comparison, AWS EBS GP2 is priced at just $0.1/GB per month.</p><p>2. <strong>Reduced API call fees</strong>: API call fees are halved compared to standard S3 buckets. For standard S3, update requests (e.g., PUT, DELETE) cost $0.005 per 1,000 requests, and read requests (e.g., GET, HEAD) cost $0.0004 per 10,000 requests. With S3 Express One Storage, update requests cost $0.0025 per 1,000, and read requests are $0.0002 per 10,000.</p><p>I believe that after seeing this pricing, clever readers will have already figured out how to maximize the value of the new storage type.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ace46ea21ec9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How SSB Queries run in StarRocks]]></title>
            <link>https://medium.com/starrocks-engineering/how-ssb-query-run-in-starrocks-0011e7e7bf6e?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/0011e7e7bf6e</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Sat, 07 Sep 2024 23:01:05 GMT</pubDate>
            <atom:updated>2025-11-21T02:15:10.430Z</atom:updated>
            <content:encoded><![CDATA[<p>This article uses a typical SSB query as an example to explain the execution process of a query (Query 1) within StarRocks. The goal is to help readers better understand how queries are executed, so that they can subsequently analyze more complex queries independently and solve everyday issues.</p><p>To simplify the analysis, the test environment consists of two CN nodes, focusing on the execution process in a shared data cluster. However, this is the same for the share nothing mode.</p><h3>Query</h3><pre>select sum(lo_extendedprice*lo_discount) as revenue from lineorder, dates where lo_orderdate = d_datekey and d_year = 1993 and lo_discount between 1 and 3 and lo_quantity &lt; 25;</pre><h3>Query Plan</h3><pre>mysql&gt; explain select sum(lo_extendedprice*lo_discount) as revenue from lineorder, dates where lo_orderdate = d_datekey and d_year = 1993 and lo_discount between 1 and 3 and lo_quantity &lt; 25;<br>+------------------------------------------------------------------------------------------+<br>| Explain String                                                                           |<br>+------------------------------------------------------------------------------------------+<br>| PLAN FRAGMENT 0                                                                          |<br>|  OUTPUT EXPRS:36: sum                                                                    |<br>|   PARTITION: UNPARTITIONED                                                               |<br>|                                                                                          |<br>|   RESULT SINK                                                                            |<br>|                                                                                          |<br>|   9:AGGREGATE (merge finalize)                                                           |<br>|   |  output: sum(36: sum)                                                                |<br>|   |  group by:                                                                           |<br>|   |                                                                                      |<br>|   8:EXCHANGE                                                                             |<br>|                                                                                          |<br>| PLAN FRAGMENT 1                                                                          |<br>|  OUTPUT EXPRS:                                                                           |<br>|   PARTITION: RANDOM                                                                      |<br>|                                                                                          |<br>|   STREAM DATA SINK                                                                       |<br>|     EXCHANGE ID: 08                                                                      |<br>|     UNPARTITIONED                                                                        |<br>|                                                                                          |<br>|   7:AGGREGATE (update serialize)                                                         |<br>|   |  output: sum(CAST(10: lo_extendedprice AS BIGINT) * CAST(12: lo_discount AS BIGINT)) |<br>|   |  group by:                                                                           |<br>|   |                                                                                      |<br>|   6:Project                                                                              |<br>|   |  &lt;slot 10&gt; : 10: lo_extendedprice                                                    |<br>|   |  &lt;slot 12&gt; : 12: lo_discount                                                         |<br>|   |                                                                                      |<br>|   5:HASH JOIN                                                                            |<br>|   |  join op: INNER JOIN (BROADCAST)                                                     |<br>|   |  colocate: false, reason:                                                            |<br>|   |  equal join conjunct: 6: lo_orderdate = 18: d_datekey                                |<br>|   |                                                                                      |<br>|   |----4:EXCHANGE                                                                        |<br>|   |                                                                                      |<br>|   1:Project                                                                              |<br>|   |  &lt;slot 6&gt; : 6: lo_orderdate                                                          |<br>|   |  &lt;slot 10&gt; : 10: lo_extendedprice                                                    |<br>|   |  &lt;slot 12&gt; : 12: lo_discount                                                         |<br>|   |                                                                                      |<br>|   0:OlapScanNode                                                                         |<br>|      TABLE: lineorder                                                                    |<br>|      PREAGGREGATION: ON                                                                  |<br>|      PREDICATES: 12: lo_discount &gt;= 1, 12: lo_discount &lt;= 3, 9: lo_quantity &lt; 25         |<br>|      partitions=7/7                                                                      |<br>|      rollup: lineorder                                                                   |<br>|      tabletRatio=28/28                                                                   |<br>|      tabletList=98793,98794,98795,98796,98797,98798,98799,98800,98801,98802 ...          |<br>|      cardinality=58779223                                                                |<br>|      avgRowSize=16.0                                                                     |<br>|                                                                                          |<br>| PLAN FRAGMENT 2                                                                          |<br>|  OUTPUT EXPRS:                                                                           |<br>|   PARTITION: RANDOM                                                                      |<br>|                                                                                          |<br>|   STREAM DATA SINK                                                                       |<br>|     EXCHANGE ID: 04                                                                      |<br>|     UNPARTITIONED                                                                        |<br>|                                                                                          |<br>|   3:Project                                                                              |<br>|   |  &lt;slot 18&gt; : 18: d_datekey                                                           |<br>|   |                                                                                      |<br>|   2:OlapScanNode                                                                         |<br>|      TABLE: dates                                                                        |<br>|      PREAGGREGATION: ON                                                                  |<br>|      PREDICATES: 22: d_year = 1993                                                       |<br>|      partitions=1/1                                                                      |<br>|      rollup: dates                                                                       |<br>|      tabletRatio=1/1                                                                     |<br>|      tabletList=98839                                                                    |<br>|      cardinality=365                                                                     |<br>|      avgRowSize=8.0                                                                      |<br>+------------------------------------------------------------------------------------------+<br>72 rows in set (0.01 sec)</pre><p>Analyzing the Query Plan we can see that three Fragments are generated:</p><p><strong>Fragment 2:</strong> Scans the dates table, retrieves the specific column <em>d_datekey</em> (Project), and then exchanges data with Fragment 1 via the EXCHANGE operator. In the visualized Profile, it looks like as this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*zb7iag_KD6sVPuV4N4plWQ.png" /></figure><p><strong>Fragment 1:</strong> Scans the lineorder table, retrieves specific columns <em>lo_orderdate</em>, <em>lo_extendedprice</em>, and <em>lo_discount</em> via the PROJECT operator, then exchanges data from Fragment 2 to perform a hash join operation. Afterward, it uses the Project operator to obtain the contents of the two columns <em>lo_extendedprice</em> and <em>lo_discount</em>. Next, the AGG operator will compute the sum (sum(CAST(10: lo_extendedprice AS BIGINT) * CAST(12: lo_discount AS BIGINT))). In the visualized Profile, it looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*6LD2_6WMHtBQU7nGb2q0aQ.png" /></figure><p><strong>Fragment 0:</strong> Not very clear.</p><p>It can be seen that Fragments are divided by the EXCHANGE operator. Once an EXCHANGE is needed, a new Fragment is planned, and different Fragments pass data through the EXCHANGE operator.</p><h3><strong>Fragment Execution</strong></h3><p>We’ll take Fragment 1 (which is more complex) as an example. From the Profile, the key execution process within the Fragment can be observed.</p><p>Since the test cluster has two CN nodes and each partition has 4 tablets, it can be observed that Fragment 1 generated two fragment instances, which were sent to two CN nodes. The logs show that each CN node generated one instance, as follows:</p><pre>Fragment 1:<br>    BackendAddresses: 172.26.92.212:9060, 172.26.92.212:9062<br>    InstanceIds: 601a635c-6ce4–11ef-8e27–2af996751df8, 601a635c-6ce4–11ef-8e27–2af996751df9<br>    BackendNum: 2</pre><p>Each Fragment instance contains half of the tablets from all partitions.</p><pre># The first CN<br>I20240907 14:42:39.257463 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98784, tablet: 98794<br>I20240907 14:42:39.257468 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98784, tablet: 98796<br>I20240907 14:42:39.257471 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98785, tablet: 98798<br>I20240907 14:42:39.257473 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98785, tablet: 98800<br>I20240907 14:42:39.257476 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98786, tablet: 98802<br>I20240907 14:42:39.257478 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98786, tablet: 98804<br>I20240907 14:42:39.257480 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98787, tablet: 98806<br>I20240907 14:42:39.257483 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98787, tablet: 98808<br>I20240907 14:42:39.257485 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98788, tablet: 98810<br>I20240907 14:42:39.257487 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98788, tablet: 98812<br>I20240907 14:42:39.257490 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98789, tablet: 98814<br>I20240907 14:42:39.257493 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98789, tablet: 98816<br>I20240907 14:42:39.257495 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98790, tablet: 98818<br>I20240907 14:42:39.257497 139780171531840 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df8, scan ranges: db: , table: , partition: 98790, tablet: 98820<br><br># The second CN<br>I20240907 14:42:39.273023 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98784, tablet: 98793<br>I20240907 14:42:39.273028 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98784, tablet: 98795<br>I20240907 14:42:39.273031 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98785, tablet: 98797<br>I20240907 14:42:39.273033 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98785, tablet: 98799<br>I20240907 14:42:39.273035 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98786, tablet: 98801<br>I20240907 14:42:39.273038 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98786, tablet: 98803<br>I20240907 14:42:39.273040 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98787, tablet: 98805<br>I20240907 14:42:39.273043 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98787, tablet: 98807<br>I20240907 14:42:39.273045 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98788, tablet: 98809<br>I20240907 14:42:39.273047 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98788, tablet: 98811<br>I20240907 14:42:39.273050 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98789, tablet: 98813<br>I20240907 14:42:39.273052 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98789, tablet: 98815<br>I20240907 14:42:39.273054 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98790, tablet: 98817<br>I20240907 14:42:39.273057 140253722486336 fragment_executor.cpp:452] fragment instance: 601a635c-6ce4-11ef-8e27-2af996751df9, scan ranges: db: , table: , partition: 98790, tablet: 98819</pre><p>Additionally, Fragment 1 built 5 Pipelines, which can be seen in detail from the profile:</p><pre>    Pipeline (id=4):   <br>        EXCHANGE_SINK (plan_node_id=8):<br>          ...<br>        AGGREGATE_BLOCKING_SOURCE (plan_node_id=7):<br>          ...<br>    Pipeline (id=3):<br>        AGGREGATE_BLOCKING_SINK (plan_node_id=7):<br>          ...<br>        PROJECT (plan_node_id=6):<br>          ...<br>        CHUNK_ACCUMULATE (plan_node_id=5):<br>          ...<br>        HASH_JOIN_PROBE (plan_node_id=5):<br>          ...<br>        LOCAL_EXCHANGE_SOURCE (plan_node_id=5):<br>          ...<br>    Pipeline (id=2):<br>        LOCAL_EXCHANGE_SINK (plan_node_id=5):<br>          ...<br>        PROJECT (plan_node_id=1):<br>          ...<br>        CHUNK_ACCUMULATE (plan_node_id=0):<br>          ...  <br>        CONNECTOR_SCAN (plan_node_id=0):<br>          ...<br>    Pipeline (id=1):<br>        HASH_JOIN_BUILD (plan_node_id=5):<br>          ...<br>        LOCAL_EXCHANGE_SOURCE (plan_node_id=5):<br>          ...<br>    Pipeline (id=0):<br>        LOCAL_EXCHANGE_SINK (plan_node_id=5):<br>          ...<br>        EXCHANGE_SOURCE (plan_node_id=4):<br>          ...</pre><p>Based on the flow of the data, the Pipeline diagram can be drawn (numbers represent Plan Node ID):</p><p>Pipeline 1 to 5 looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ie_rXEN3wm3jZFAFnFAjGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LJEd5dRn8LnBN-H_hda5pg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a_hX3wB3SAoo58nG.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5sUJGT_9cszqSL-m.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lm7CdHWHvNy3cmGBI1nKg.png" /></figure><p>As we can see, Fragment Instance 1 is executed around the Hash Join (Plan node 5) operator. Pipelines 2 and 3 are responsible for reading the tablet data from the current node and passing it to the Hash Join probe, while Pipelines 0 and 1 read data from Fragment Instance 2 (located on other CN) through the Exchange operator and build the Hash Join. Pipeline 3 passes the result of the Hash Join to the Project and Agg operators, and the final result is passed by Pipeline 4 through the Exchange Sink operator to Fragment 0.</p><p>I hope you found this SSB query tutorial useful, and if you did, join me on Slack [Slack URL = <a href="https://www.youtube.com/redirect?event=comments&amp;redir_token=QUFFLUhqbmN1QmluR3htTExiUXMyTkZQZ3Y0bmZ0T3gtUXxBQ3Jtc0ttSkktSUM1b3ZiT0VfWGZHUERXNWg0eFhicjgxbC1SNXpsWVctOFNMcy1BZG5HYjhwcXhncW1Sa1QtdFpLR0NPY29ueXAtS0laekkxQ0hUSTJPYy0xcHl4NFZvRHJGdmNSUU5NMFZOZko2UTRUZW9FQQ&amp;q=https%3A%2F%2Ftry.starrocks.com%2Fjoin-starrocks-on-slack">https://try.starrocks.com/join-starrocks-on-slack</a>] and let me know what you think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0011e7e7bf6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starrocks-engineering/how-ssb-query-run-in-starrocks-0011e7e7bf6e">How SSB Queries run in StarRocks</a> was originally published in <a href="https://medium.com/starrocks-engineering">StarRocks Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using IAM roles to allow the StarRocks Cloud Native CN Pods in AWS EKS to access the S3 bucket]]></title>
            <link>https://medium.com/@tracymacding/using-iam-roles-to-allow-the-starrocks-cloud-native-cn-pods-in-aws-eks-to-access-the-s3-bucket-70056a52789b?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/70056a52789b</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[iam-roles]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[starrocks]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Sat, 31 Aug 2024 09:40:44 GMT</pubDate>
            <atom:updated>2024-08-31T09:40:44.091Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This document mainly introduces how to configure access to S3 Bucket objects in EKS. It is assumed that the S3 Bucket and EKS cluster resources mentioned in this document have been created in advance.</p><h3>Obtain OIDC information for EKS cluster</h3><p>Log in to the AWS console, find the EKS cluster you want to set up, and obtain its OIDC information, such as the OpenID Connect provider URL in the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LZOE3tWCLvxt0Cvg" /></figure><p>In this example, the OpenID Connect provider URL is <a href="https://oidc.eks.us-west-2.amazonaws.com/id/E5F3EE6ADFB9324EC4206983AD642FA2">https://oidc.eks.us-west-2.amazonaws.com/id/E5F3EE6ADFB9324EC4206983AD642FA2</a>.</p><h3>Create IAM Identity Provider</h3><p>Enter the AWS console and create an Identity Provider in the IAM service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3HK6oBEPR9dv3h18" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_f2ITkY7Cl6_MmbX" /></figure><p>In the Provider type above, select OpenID Connect, and in the Provider URL, enter the OCID of the EKS cluster obtained in the previous step. In the Audience, enter the above content (remember not to write it randomly).</p><h3>Create IAM Policy</h3><p>Because access to S3 is required, first you need to create an IAM Policy. For simplicity, the policy created here allows owners access to all S3 resources:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CuJyK1Hxit9i1NXO" /></figure><h3>Create IAM Role</h3><p>Next, you need to create an IAM Role and authorize the Web Identity Provider to assume it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xESFNxWhA2vpn0i3" /></figure><p>Next, set the details for the Role, including:</p><ul><li><em>Trusted entity type</em>：Choose Web identity</li><li><em>Identity provider</em>：Choose the Provider we created</li><li><em>Audience</em>：Choose the Audience we created</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xWUJgdpxaE5yQCpc" /></figure><p>Next, select permissions for the Role, selecting the Policy created in the previous step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jY_qEjxV6vTYlgyj" /></figure><p>Finally, enter the Role name and other information to complete the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fX2K3zUg9hgQunwr" /></figure><h3>Create EKS ServiceAccount</h3><p>First, write a yaml file to create an account (assuming the name is jeff-acc.yaml),</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br> name: jeff-acc<br> namespace: default<br> annotations:<br>   eks.amazonaws.com/role-arn:  arn:aws:iam::081976408565:role/jeff-test-role</pre><p>Where the annotations need to be filled in the arn of the <em>jeff-test-role</em> created in the previous step.</p><p>Then execute the following command:</p><pre>kubectl create -f jeff-acc.yaml</pre><p>This command creates a service account named <em>jeff-acc</em> in the <em>default</em> namespace of the EKS cluster. After successful creation, you can use the following command to view the details of the service account:</p><pre>[sr@ip-172-31-29-22 sr_eks]$ kubectl get serviceaccounts/jeff-acc -o yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  annotations:<br>    eks.amazonaws.com/role-arn: arn:aws:iam::081976408565:role/jeff-test-role<br>    kubectl.kubernetes.io/last-applied-configuration: |<br>      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;kind&quot;:&quot;ServiceAccount&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{&quot;eks.amazonaws.com/role-arn&quot;:&quot;arn:aws:iam::081976408565:role/jeff-test-role&quot;},&quot;name&quot;:&quot;jeff-acc&quot;,&quot;namespace&quot;:&quot;default&quot;}}<br>  creationTimestamp: &quot;2023-01-19T14:13:32Z&quot;<br>  name: jeff-acc<br>  namespace: default<br>  resourceVersion: &quot;319803&quot;<br>  uid: 7ed608a1-f167-463b-9259-08f477179c4b</pre><h3>Modify Trusted Relationships of IAM Role</h3><p>After creating the EKS Service Account, you will need to modify the Trusted Relationships information of the Role created above. The information before modification is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*obwOgx2RbTVAi906" /></figure><p>After modify：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w9mpgFcSCsAwTO-b" /></figure><h3>Test</h3><p>After completing all of the above configurations, we can create a Pod to verify the effect. First, write a test.yaml file, the content is as follows:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  labels:<br>    run: my-pod<br>  name: my-pod<br>  namespace: default<br>spec:<br>  serviceAccountName: jeff-acc<br>  initContainers:<br>  - image: amazon/aws-cli<br>    name: my-aws-cli<br>    command: [&#39;aws&#39;, &#39;s3&#39;, &#39;ls&#39;, &#39;s3://jeff-tpch-test-3/&#39;]<br>  containers:<br>  - image: nginx<br>    name: my-pod<br>    ports:<br>    - containerPort: 80<br>  dnsPolicy: ClusterFirst<br>  restartPolicy: Always<br>status: {}</pre><p>Then execute the command:</p><pre>kubectl apply -f test.yaml</pre><p>Wait for the Pod to be created. Interested readers can also use the following command to view the pod status:</p><pre>[sr@ip-172-31-29-22 sr_eks]$ kubectl describe pod my-pod<br>Name:         my-pod<br>Namespace:    default<br>Priority:     0<br>Node:         ip-192-168-0-127.us-west-2.compute.internal/192.168.0.127<br>Start Time:   Thu, 19 Jan 2023 14:21:40 +0000<br>Labels:       run=my-pod<br>...<br>Containers:<br>  my-pod:<br>    Container ID:   containerd://75435c3e9b5a6dd41dc9e4e871a202c8b478bba360f766aa7a8a1c804986551d<br>    Image:          nginx<br>    Image ID:       docker.io/library/nginx@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e<br>    Port:           80/TCP<br>    Host Port:      0/TCP<br>    State:          Running<br>      Started:      Thu, 19 Jan 2023 14:21:43 +0000<br>    Ready:          True<br>    Restart Count:  0<br>    Environment:<br>      AWS_STS_REGIONAL_ENDPOINTS:   regional<br>      AWS_DEFAULT_REGION:           us-west-2<br>      AWS_REGION:                   us-west-2<br>      AWS_ROLE_ARN:                 arn:aws:iam::081976408565:role/jeff-test-role<br>      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token</pre><p>It can be seen that the Pod has set some environment variables related to AWS authentication, which are necessary for authentication. At the same time, we can further check whether the command “aws ls s3” is executed successfully in the Pod through the following command:</p><pre>[sr@ip-172-31-29-22 sr_eks]$ kubectl logs my-pod my-aws-cli -n default<br>                           PRE 007af5a1-65fd-43a0-8c2c-fc9f5d1037f7/<br>                           PRE 0f205252-9ae3-486b-9603-8f93a10f0a68/<br>                           PRE 25eed49a-fd7d-40ec-91a3-ed01e8d11b3c/<br>                           PRE 27b8a3c7-83e0-47b7-96eb-be5cfa6ae6fe/<br>                           PRE 377f44ce-d984-4b6b-858b-782a22616894/<br>                           PRE 3a7861a6-dcf2-49e4-8c4f-e891af79351a/<br>                           PRE 44e169c9-4c5c-4629-a5fa-edc48c32ec63/<br>                           PRE a43a41f0-b985-4242-a18a-9580f8bfc937/<br>                           PRE b3d3dfff-13cc-420f-aa02-bbac6da013c1/<br>                           PRE c9fcb11e-42ca-4b0f-9acf-85ce2a8882cf/<br>                           PRE e7e2e08e-0961-4685-97a6-0964a1c383f7/<br>                           PRE eed88ce7-c9c4-4ed5-b414-c313796e281b/</pre><p>The above output indicates that the execution was successful, thus verifying that the authentication process is OK.</p><h3>Reference</h3><ul><li><a href="https://levelup.gitconnected.com/using-iam-roles-to-allow-the-pods-in-aws-eks-to-read-the-aws-s3-bucket-be493fbdda84">https://levelup.gitconnected.com/using-iam-roles-to-allow-the-pods-in-aws-eks-to-read-the-aws-s3-bucket-be493fbdda84</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70056a52789b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Four authentication methods StarRocks Cloud Native supports]]></title>
            <link>https://medium.com/starrocks-engineering/four-authentication-methods-starrocks-cloud-native-supports-3de8c4b71a32?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/3de8c4b71a32</guid>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-authentication]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Sat, 31 Aug 2024 09:35:22 GMT</pubDate>
            <atom:updated>2024-08-31T09:35:22.421Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>StarRocks Cloud Native Edition support access to resources on public clouds, so security is a very important aspect. This document provides four authentication methods supported while deploying StarRocks Cloud Native Edition on AWS.</p><h3>Method 1: Using AK / SK (Access Key and Secret Key)</h3><p>This is also the least secure method. By configuring the AK/SK key pair on the FE node , it will distribute the AK/SK key pair to the BE computing node during system operation. The BE node then passes the AK/SK to the AWS SDK to access cloud resources. This is the easiest method to use but has the lowest security factor. Once the AK/SK is leaked, it will have a severe impact and is not recommended to use.</p><h3>Method 2: Using Instance Profile</h3><p>This method involves creating an instance profile for EC2 instances in the cloud environment, and configuring the profile with permissions to access specific resources. In this way, instances with this profile can access the corresponding resources. The steps of this operation are:</p><ol><li><strong>Create a service role with type ‘AWS Service’:</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oLynKyYV73Y5qX8G" /></figure><p><strong>2. Add full access permission to S3 service for role, as shown in the following picture:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yqmlLv7r9RYrhx34" /></figure><p><strong>3. Naming the role:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tUfcCpeZZqrauEi6" /></figure><p><strong>4. You can view the details of the newly created service role on the AWS console:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*88MsnfdMwFNsnQce" /></figure><p><strong>5. Attach the role created above to EC2 that needs access to AWS resources. In the following image, select “Modify IAM role” and in the drop-down box, select the service role just created.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TnlmVzfsdmGzBTLK" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sephj5H4ViNqvqnC" /></figure><p><strong>6. Once completion, you can view the attached profile in the EC2 detail page:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1l3mLVSDYn_3a9QR" /></figure><p>After finishing the above steps, configure the authentication method and enable instance_profile property during creating storage volume:</p><pre>CREATE STORAGE VOLUME jeff_s3_volume<br>TYPE = S3<br>LOCATIONS = s3://jeff_bucket/starrocks/<br>PROPERTIES (<br>  &quot;aws.s3.region&quot; = &quot;&lt;region&gt;&quot;,<br>  &quot;aws.s3.endpoint&quot; = &quot;&lt;endpoint_url&gt;&quot;,<br>  &quot;aws.s3.use_aws_sdk_default_behavior&quot; = &quot;false&quot;,<br>  &quot;aws.s3.use_instance_profile&quot; = &quot;true&quot;,<br>)</pre><p>This method is much more secure than the first method, but there are problems such as abuse of power. Use it with caution and it is not the best way we recommend.</p><h3>Method 3: Using Instance Profile and Assume Role</h3><p>In method two, a service role with high privileges is created and attached to the EC2, giving the EC2 the corresponding privileges, but this is not the best way either. We generally recommend using the Instance Profile plus Assume Role method for authentication and access control.</p><p>The AWS Assume Role mechanism allows a service or user entity to take on a certain role, thereby having the permissions of that role. Because you can dynamically control the types and number of roles played, it is very flexible in actual use and brings the maximum level of security.</p><p>To use this mechanism, you first need to create a role with no permissions. The steps are as follows:</p><ol><li><strong>Create a role with no permissions, as shown in the following image:</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kz1EQGRivz-8poYY" /></figure><p><strong>2. Select permissions, note that at this time no permissions need to be selected. This is one of the differences from method two.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YnmDhLyQrN8312Z_" /></figure><p><strong>3. Give it a name and complete the creation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Xxt7p9RMXdcu6ZD" /></figure><p><strong>4. Next, you can see detailed information of the role on the console:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vi7wkkPC-CP4lzoM" /></figure><p><strong>5. Next, attach the newly created role to EC2 that needs to access to resources, the steps are the same as method two, it will not be repeated here. After the attachment is completed, the effect is as follows:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m7Cq4LcDqwBR-CrY" /></figure><p><strong>6. Then, the newly created role needs to be configured so that assume role operations can be performed. The steps are as follows:</strong></p><ul><li>Add permissions to the newly created role.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BInirtiRwfa4QdCi" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Bgn9sMSALTJ-Db8" /></figure><p>It means allowing the role ‘<em>jeff-test-role-no-policy</em>’ to assume the target role ‘<em>arn:aws:iam::081976408565:role/CELERDATA_Data_Role</em>’, so that it can access resources that the target role has access to. Therefore, you need to create the target role and define its access permissions beforehand. After it is created, the effect will be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fni9BWCVtcqWdkDe" /></figure><ul><li>The next step is to configure the role that will be assumed by ‘<em>jeff-test-role-no-policy</em>’ , specifically, by modifying the following fields:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ImzuxX6Vcl1rNW6h" /></figure><ul><li>Add the ARN of the role that it is allowed to assume, as shown in the blue box in the following picture:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L4wtpY850d3ggSns" /></figure><p>After completing the above configuration, the environment configuration is complete. Then you can create storage volume like this:</p><pre>CREATE STORAGE VOLUME jeff_s3_volume<br>TYPE = S3<br>LOCATIONS = s3://jeff_bucket/starrocks/<br>PROPERTIES (<br>  &quot;aws.s3.region&quot; = &quot;&lt;region&gt;&quot;,<br>  &quot;aws.s3.endpoint&quot; = &quot;&lt;endpoint_url&gt;&quot;,<br>  &quot;aws.s3.use_aws_sdk_default_behavior&quot; = &quot;false&quot;,<br>  &quot;aws.s3.use_instance_profile&quot; = &quot;true&quot;,<br>  &quot;aws.s3.iam_role_arn&quot; = &quot;arn:aws:iam::081976408565:role/CELERDATA_Data_Role&quot;<br>)</pre><p>Please note that the <em>iam_role_arn</em> should be configured as the ARN of the assumed role.</p><h3>Method 4: Web identity</h3><p>This method allows pods in EKS to access S3 objects, in support of deploying StarRocks Cloud Native clusters directly in EKS environments. The main steps of the operation will be describiled in next document.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3de8c4b71a32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starrocks-engineering/four-authentication-methods-starrocks-cloud-native-supports-3de8c4b71a32">Four authentication methods StarRocks Cloud Native supports</a> was originally published in <a href="https://medium.com/starrocks-engineering">StarRocks Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hierarchy of files in StarRocks Cloud Native]]></title>
            <link>https://medium.com/starrocks-engineering/introduction-47d1eea48b41?source=rss-cc243c061c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/47d1eea48b41</guid>
            <category><![CDATA[starrocks]]></category>
            <category><![CDATA[storage-engine]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[olap]]></category>
            <category><![CDATA[lakehouse-architecture]]></category>
            <dc:creator><![CDATA[Jeff Ding]]></dc:creator>
            <pubDate>Tue, 13 Aug 2024 06:12:27 GMT</pubDate>
            <atom:updated>2024-08-14T19:48:49.361Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the product of StarRocks’ cloud native, the data file organization format has been redesigned to better adapt to the characteristics of object storage. This article will use SSB lineorder table as an example to illustrate how data files are organized on object storage.</p><h3>Table Storage Path</h3><p>When creating a table in a cloud native cluster, you can specify a Storage Volume in the parameters (for more information on Storage Volume, please refer to ). The Storage Volume specifies the storage bucket and sub-path. Once the table is successfully created, all its data will be stored under the specified storage path in the Storage Volume.</p><p>The specific storage path rules for the table are as follows:</p><ul><li>For older versions (3.1.3 and earlier), the storage path for table data is organized in the following format:</li></ul><pre>s3://${storage_volume_location}/${cluster_id}/${table_id}/</pre><ul><li>In intermediate versions (3.1.4 and after 3.2.0), the storage path for table data is organized in the following format:</li></ul><pre>s3://${storage_volume_location}/${cluster_id}/${table_id}/${partition_id}/</pre><ul><li>In the latest versions (3.1.8 and after 3.2.3), the storage path for table data is organized in the following format:</li></ul><pre>s3://${storage_volume_location}/${cluster_id}/db{dbid}/${table_id}/${partition_id}</pre><p>For example, the Storage Volume information used in our testing environment is as follows:</p><pre>mysql&gt; DESC STORAGE VOLUME builtin_storage_volume\G<br>*************************** 1. row ***************************<br>     Name: builtin_storage_volume<br>     Type: S3<br>IsDefault: true<br> Location: s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762<br>   Params: {&quot;aws.s3.region&quot;:&quot;us-west-2&quot;,&quot;aws.s3.use_instance_profile&quot;:&quot;true&quot;,&quot;aws.s3.use_aws_sdk_default_behavior&quot;:&quot;false&quot;,&quot;aws.s3.endpoint&quot;:&quot;xxx&quot;}<br>  Enabled: true</pre><p>The data for this table will be stored in the path <em>s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762.</em></p><p>Next, we will create an SSB lineorder table in this cluster with the following structure:</p><pre>CREATE TABLE `lineorder` (<br>  `lo_orderkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_linenumber` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_custkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_partkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_suppkey` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_orderdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_orderpriority` varchar(16) NOT NULL COMMENT &quot;&quot;,<br>  `lo_shippriority` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_quantity` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_extendedprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_ordtotalprice` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_discount` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_revenue` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_supplycost` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_tax` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_commitdate` int(11) NOT NULL COMMENT &quot;&quot;,<br>  `lo_shipmode` varchar(11) NOT NULL COMMENT &quot;&quot;<br>) ENGINE=OLAP<br>DUPLICATE KEY(`lo_orderkey`)<br>COMMENT &quot;OLAP&quot;<br>PARTITION BY RANGE(`lo_orderdate`)<br>(PARTITION p1 VALUES [(&quot;-2147483648&quot;), (&quot;19930101&quot;)),<br>PARTITION p2 VALUES [(&quot;19930101&quot;), (&quot;19940101&quot;)),<br>PARTITION p3 VALUES [(&quot;19940101&quot;), (&quot;19950101&quot;)),<br>PARTITION p4 VALUES [(&quot;19950101&quot;), (&quot;19960101&quot;)),<br>PARTITION p5 VALUES [(&quot;19960101&quot;), (&quot;19970101&quot;)),<br>PARTITION p6 VALUES [(&quot;19970101&quot;), (&quot;19980101&quot;)),<br>PARTITION p7 VALUES [(&quot;19980101&quot;), (&quot;19990101&quot;)))<br>DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 1<br>PROPERTIES (<br>&quot;replication_num&quot; = &quot;1&quot;<br>);</pre><p>Then use the following command to obtain the storage path of the table on S3 (tmp is the database name):</p><pre>mysql&gt; show proc &#39;/dbs/tmp&#39;;<br>+---------+-----------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------------------------------------------------------------------------------------------------+<br>| TableId | TableName | IndexNum | PartitionColumnName | PartitionNum | State  | Type         | LastConsistencyCheckTime | ReplicaCount | PartitionType | StoragePath                                                                                                            |<br>+---------+-----------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------------------------------------------------------------------------------------------------+<br>| 26019   | lineorder | 1        | lo_orderdate        | 7            | NORMAL | CLOUD_NATIVE | NULL                     | 7            | RANGE         | s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762/632add3b-f7e1-477c-a925-f71aab66bbd2/db26010/26019 |<br>+---------+-----------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------------------------------------------------------------------------------------------------+</pre><p>StoragePath field clearly shows the path of the table on S3:</p><pre>s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762/632add3b-f7e1-477c-a925-f71aab66bbd2/db26010/26019</pre><p>Parsing the path:</p><blockquote>s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762<em>: This part is the prefix path defined in the Storage Volume earlier</em></blockquote><blockquote>632add3b-f7e1–477c-a925-f71aab66bbd2<em>: cluster id, each StarRocks storage and computation separated cluster will generate a globally unique id as part of the path</em></blockquote><blockquote>db26010<em>: db id to which the table belongs</em></blockquote><blockquote>26019<em>: table id</em></blockquote><p>Since the table is a partitioned table with 7 partitions, we can see that there are 7 subdirectories under the table’s storage path:</p><pre>aws s3 ls s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762/632add3b-f7e1-477c-a925-f71aab66bbd2/db26010/26019/<br>                           PRE 26012/<br>                           PRE 26013/<br>                           PRE 26014/<br>                           PRE 26015/<br>                           PRE 26016/<br>                           PRE 26017/<br>                           PRE 26018/</pre><p>Each subdirectory is named after the Partition Id.</p><pre>mysql&gt; show partitions from lineorder;<br>+-------------+---------------+----------------+----------------+-------------+--------+--------------+-------------------------------------------------------------------------+-----------------+---------+----------+----------+-----------------+------------+-------+-------+-------+<br>| PartitionId | PartitionName | CompactVersion | VisibleVersion | NextVersion | State  | PartitionKey | Range                                                                   | DistributionKey | Buckets | DataSize | RowCount | EnableDataCache | AsyncWrite | AvgCS | P50CS | MaxCS |<br>+-------------+---------------+----------------+----------------+-------------+--------+--------------+-------------------------------------------------------------------------+-----------------+---------+----------+----------+-----------------+------------+-------+-------+-------+<br>| 26012       | p1            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [-2147483648]; ..types: [INT]; keys: [19930101]; ) | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>| 26013       | p2            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [19930101]; ..types: [INT]; keys: [19940101]; )    | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>| 26014       | p3            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [19940101]; ..types: [INT]; keys: [19950101]; )    | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>| 26015       | p4            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [19950101]; ..types: [INT]; keys: [19960101]; )    | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>| 26016       | p5            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [19960101]; ..types: [INT]; keys: [19970101]; )    | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>| 26017       | p6            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [19970101]; ..types: [INT]; keys: [19980101]; )    | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>| 26018       | p7            | 0              | 1              | 2           | NORMAL | lo_orderdate | [types: [INT]; keys: [19980101]; ..types: [INT]; keys: [19990101]; )    | lo_orderkey     | 1       | 0B       | 0        | true            | false      | 0.00  | 0.00  | 0.00  |<br>+-------------+---------------+----------------+----------------+-------------+--------+--------------+-------------------------------------------------------------------------+-----------------+---------+----------+----------+-----------------+------------+-------+-------+-------+<br>7 rows in set (0.21 sec)</pre><p>There are also three subdirectories under the above partition path: data/, meta/ and log/, used to store different types of files. The following will explain one by one. Before discussing this, we need to briefly describe the multi-version data technology in cloud native.</p><h3>Multi-Version Data Files</h3><p>Currently, StarRocks storage and computation separated tables use multi-version data technology, and the overall storage structure is as shown in the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LwzGGfRpK0vdCFBE" /></figure><p>In the above diagram, three data import transactions were generated, including:</p><ul><li><strong><em>Load Txn 1</em></strong>: During the transaction data writing phase, new data files file 1 &amp; file 2 were generated, and after this transaction was committed, Tablet Meta V1 was generated, which records the list of visible files for this version as {file-1, file-2}</li><li><strong><em>Load Txn 2</em></strong>: During the transaction data writing phase, new data files file 3 &amp; file 4 were generated. Upon submission, based on the previous version (i.e., Tablet Meta V1) and the new data files generated in this import transaction (file-3 &amp; file-4), a new Tablet Meta V2 was created, and thus the list of visible files for this version is {file-1, file-2, file-3, file-4}</li><li><strong><em>Load Txn 3</em></strong>: During the transaction data writing phase, a new data file file 5 was generated. Upon submission, based on the previous version (i.e., Tablet Meta V2) and the new data file generated in this import transaction (file-5), a new Tablet Meta V3 was created, and thus the list of visible files for this version is {file-1, file-2, file-3, file-4, file-5}</li></ul><p>In addition to user-imported transactions generating new data versions, in cloud native, system background Compaction tasks will also generate new data versions. The purpose of Compaction is twofold: 1) to merge multiple versions of small files into large files, reducing the number of random IO operations during queries, 2) to eliminate duplicate data records, reducing the overall data volume.</p><p>In StarRocks cloud native, each Compaction will also generate a completely new version. For the example above, if a new transaction Txn 4 after Txn 3 is a Compaction task and merges the 4 files file1 ~ file4 into file-6, then upon submission of this transaction, the newly generated Tablet Meta V4 will record the file list as {file-5, file-6}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9z9YjKDVR7e45ij5" /></figure><h3>data file</h3><p>This directory mainly stores Tablet Data files, and the data file format remains consistent with storage and computation together (still Segment files of columnar storage).The data file naming format is:</p><pre>{txn_id}-{uuid}.dat</pre><p>After importing a batch of data into the lineorder table, check the files under the data directory of partition 26012:</p><pre>aws s3 ls s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762/632add3b-f7e1-477c-a925-f71aab66bbd2/db26010/26019/26012/data/<br>2024-05-28 10:59:46   21651388 000000000000271e_225023df-78e4-4d58-89c9-1ec828c44f98.dat</pre><h3>meta file</h3><p>This directory mainly stores Tablet Meta files. As mentioned earlier, this file is mainly used to record the data file paths contained in a specific version. The file naming format is:</p><pre>{tablet_id}_{version}.meta</pre><p>For example, the following shows:</p><pre>aws s3 ls s3://starrocks-common/skr2veiad-jeff_io_optimize_test-1712281975762/632add3b-f7e1-477c-a925-f71aab66bbd2/db26010/26019/26012/meta/<br>2024-05-28 10:14:01        823 00000000000065A5_0000000000000001.meta</pre><h3>log file</h3><p>This directory mainly stores log files generated during the running of import transactions, which mainly record information such as the data file names generated in this import transaction. After the transaction is successfully completed, this file is deleted. As mentioned earlier, this file is mainly used to record the data file paths contained in a specific version. The file naming format is:</p><pre>{tablet_id}_{txn_id}.log</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47d1eea48b41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/starrocks-engineering/introduction-47d1eea48b41">The Hierarchy of files in StarRocks Cloud Native</a> was originally published in <a href="https://medium.com/starrocks-engineering">StarRocks Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>