<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akash Dwivedi on Medium]]></title>
        <description><![CDATA[Stories by Akash Dwivedi on Medium]]></description>
        <link>https://medium.com/@akash-dwivedi?source=rss-30753b40e37d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1hpHpEofCHLrvB5LgItD9w.png</url>
            <title>Stories by Akash Dwivedi on Medium</title>
            <link>https://medium.com/@akash-dwivedi?source=rss-30753b40e37d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:13:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akash-dwivedi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Time-Series Look-Ahead Bias]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://akash-dwivedi.medium.com/time-series-look-ahead-bias-dc1108f468e9?source=rss-30753b40e37d------2"><img src="https://cdn-images-1.medium.com/max/612/1*bsjjH2lVVZkHy8DRIqxO6w.jpeg" width="612"></a></p><p class="medium-feed-snippet">Random shuffling on time-series data is not a valid validation strategy. It is a critical bug that turns prediction into interpolation.</p><p class="medium-feed-link"><a href="https://akash-dwivedi.medium.com/time-series-look-ahead-bias-dc1108f468e9?source=rss-30753b40e37d------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://akash-dwivedi.medium.com/time-series-look-ahead-bias-dc1108f468e9?source=rss-30753b40e37d------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc1108f468e9</guid>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[engineeringanalysis]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Akash Dwivedi]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 09:07:15 GMT</pubDate>
            <atom:updated>2026-01-12T09:07:15.474Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Real-Time CTR: Beyond the Hype and Into the Infrastructure]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://akash-dwivedi.medium.com/real-time-ctr-beyond-the-hype-and-into-the-infrastructure-c793f455d7cb?source=rss-30753b40e37d------2"><img src="https://cdn-images-1.medium.com/max/2233/1*eo6MOzUlefuofZ3OD43DpQ.png" width="2233"></a></p><p class="medium-feed-snippet">Recommenders collapse under mislabeled implicit feedback</p><p class="medium-feed-link"><a href="https://akash-dwivedi.medium.com/real-time-ctr-beyond-the-hype-and-into-the-infrastructure-c793f455d7cb?source=rss-30753b40e37d------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://akash-dwivedi.medium.com/real-time-ctr-beyond-the-hype-and-into-the-infrastructure-c793f455d7cb?source=rss-30753b40e37d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c793f455d7cb</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[real-time-systems]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akash Dwivedi]]></dc:creator>
            <pubDate>Sat, 10 Jan 2026 08:14:29 GMT</pubDate>
            <atom:updated>2026-01-10T08:14:29.779Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Consistency Model in Cassandra]]></title>
            <link>https://blog.dataengineerthings.org/consistency-model-in-cassandra-23af926928c1?source=rss-30753b40e37d------2</link>
            <guid isPermaLink="false">https://medium.com/p/23af926928c1</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Akash Dwivedi]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 04:17:58 GMT</pubDate>
            <atom:updated>2023-06-16T18:21:18.221Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Engineering</h4><h3>Storing Consistent Data in Cassandra</h3><h4>A guide to tune consistency in Cassandra</h4><figure><img alt="Photo by Markus Spiske on Unsplash" src="https://cdn-images-1.medium.com/max/1024/1*lAHwW35SZ2aEZ6TMOcL2Pw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Markus Spiske</a> on <a href="https://unsplash.com/s/photos/technology?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3><strong>What is Cassandra?</strong></h3><p>Cassandra DB is an open-source, distributed NoSQL database that works on the principle of decentralization, meaning every node is identical in Cassandra. There is no notion of master and slave. It works on peer-to-peer architecture and uses a “<em>gossip protocol</em>” for sync and auditing. Cassandra is an elastically scalable database with fault tolerance and high availability. Unlike other distributed NoSQL databases, Cassandra has a striking feature of a tuneable consistency row-oriented database.</p><p>Cassandra inherited distribution design from <em>Amazon </em><a href="https://en.wikipedia.org/wiki/Amazon_DynamoDB"><em>DynamoDB</em></a> and data modeling from <em>Google’s </em><a href="https://en.wikipedia.org/wiki/Bigtable"><em>BigTable</em></a>. It was created in Meta (formerly Facebook) office and is nowadays widely used across the industry to power cloud-scale applications.</p><h4><strong>Where Does Cassandra Fit?</strong></h4><p>In simple terms, you can use Cassandra for a mid to large-scale project where you need outstanding throughput on writes. It provides out-of-the-box support for globally deployed application data by putting it near the user.</p><h4>Consistency Levels</h4><p>The binary nature of black or white doesn’t truly reflect in practice. So instead of having a consistent or non-consistent system, Cassandra offers levels of consistency based on the use case.</p><blockquote>“The consistency level is a setting that clients must specify on every operation and that allows you to decide how many replicas in the cluster must acknowledge a write operation or respond to a read operation in order to be considered successful. That’s the part where Cassandra has pushed the decision for determining consistency out to the client.”</blockquote><blockquote>— Cassandra: The Definitive Guide, 3rd Edition</blockquote><p>A client can specify the consistency level on each read-and-write query.</p><p>The read consistency level specifies how many replica nodes must respond to a read request before returning the data.</p><p><em>For write operations, the consistency level specifies how many replica nodes must respond for the write to be reported as successful to the client.</em></p><h4>Increasing the Consistency levels</h4><p>To Achieve strong consistency in Cassandra</p><blockquote><strong><em>R + W &gt; RF = strong consistency</em></strong></blockquote><p>In this equation,</p><ul><li>R = Read replica count</li><li>W = Write replica count</li><li>RF = Replication factor</li></ul><h4><strong>Write Consistency Levels</strong></h4><p>We can specify the consistency level during the write operation in Cassandra. Higher consistency levels also come with a reduction in availability, as more nodes must be operational for the write to succeed. Once a request is sent to any Cassandra node, that node becomes the coordinator node for that request flow.</p><p><strong>Write-steps in Cassandra :</strong></p><ol><li>Query initiated for coordinator node. This node identifies the partition and all the replica nodes. Throws an error if there are not enough replicas available as per the consistency level.</li><li>The coordinator node sends requests to all replicas. If the cluster spans multiple data centers then forward the write request to the remote coordinator.</li><li>This confirms all the “a<em>live”</em> Nodes will get the data. “D<em>ead”</em> nodes are repaired by one of the anti-entropy mechanisms: <em>hinted-handoff, read repair, anti-entropy repair.</em></li><li>Once a sufficient number of replicas have responded as per consistency level, send an acknowledgment to the client.</li></ol><h4>Read Consistency Levels</h4><p>Cassandra&#39;s reads are slower than writes. It has to read from SSTables and typically has to perform seeks. Cassandra readings have to wait for the response of all replicas as per consistency level and perform read repairs as per the necessity.</p><p><strong>Read-Steps in Cassandra:</strong></p><ol><li>Query initiated for coordinator Node. This node identifies the partition and all the replica nodes. If the coordinator is not included in the replica then it sends the query to the nearest replica node.</li><li>There is a concept of digest request is used in reading from Cassandra. which tells replicas to return a hash of the requested data.</li><li>In this step, the coordinator compares the digests returned from replicas: if digests are consistent and the required consistency level has been met then data from the fastest replica has been returned. Else, if digests are not consistent, then the coordinator must perform a “<em>read repair</em>”.</li></ol><figure><img alt="Read/Write Path" src="https://cdn-images-1.medium.com/max/836/1*CO-viCWFOfI2OCzcHuecHg.png" /><figcaption>Read/Write Path Diagram Source:Author</figcaption></figure><figure><img alt="Consistency Levels in Cassandra" src="https://cdn-images-1.medium.com/max/1024/1*l99htGXAnx1mg01yphd36Q.png" /><figcaption>Consistency Levels in Cassandra Source: Author</figcaption></figure><h4>Consolidating all the above facts</h4><p>The read-and-write consistency levels we can choose to use in our applications are an example of the tunable consistency Cassandra provides.</p><p>We can achieve a strong consistency on reads by using read and write consistency levels whose sum exceeds the replication factor.</p><p>The <strong>QUORUM</strong> consistency level is the ideal level for that</p><p>For example, on a keyspace with a replication factor of 3, <strong>QUORUM</strong> represents a response from 2 out of 3 nodes. Because 2 + 2 is greater than 3, strong consistency is guaranteed.</p><h3>Implementation</h3><p>Let’s get our hands dirty with code!</p><h4><strong>Approach 1</strong></h4><p><strong>Enforcing consistency at the session level</strong></p><p>This Python code establishes a connection to a Cassandra database cluster, authenticates the user, and retrieves data from the database.</p><pre># importing the required modules from the Cassandra Python driver<br>from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT<br>from cassandra.policies import WhiteListRoundRobinPolicy, DowngradingConsistencyRetryPolicy<br>from cassandra.query import tuple_factory<br>from cassandra.auth import PlainTextAuthProvider<br><br># defines a configuration dictionary _CONFIG that contains connection details <br># such as nodes, username, password, and keyspace for the database.<br>_CONFIG = {<br>    &quot;dev&quot;: {<br>        &quot;nodes&quot;: [&quot;127.0.0.1&quot;],<br>        &quot;username&quot;: &quot;test&quot;,<br>        &quot;password&quot;: &quot;&quot;,<br>        &quot;keyspace&quot;: &quot;test&quot;<br>    }<br>}<br><br># creates an instance of the PlainTextAuthProvider class <br># passing in the username and password from the configuration dictionary.<br>auth_provider = PlainTextAuthProvider(<br>            username=_CONFIG[&quot;username&quot;], password=_CONFIG[&quot;password&quot;]<br>        )<br>        <br>cluster_args = []<br><br># defines profile object that specifies the execution profile for the cluster.<br># The execution profile object basically overrides the default behavior of Cassandra.<br><br>profile = ExecutionProfile(<br>    load_balancing_policy=WhiteListRoundRobinPolicy([&#39;127.0.0.1&#39;]),<br>    retry_policy=DowngradingConsistencyRetryPolicy(),<br>    consistency_level=ConsistencyLevel.LOCAL_QUORUM,<br>    serial_consistency_level=ConsistencyLevel.LOCAL_SERIAL,<br>    request_timeout=15,<br>    row_factory=tuple_factory<br>)<br># Add remaining cluster options.<br># setting default EXEC_PROFILE_DEFAULT as custom ExecutionProfile object<br><br>cluster_kwargs = {<br>            &quot;auth_provider&quot;: auth_provider,<br>            &quot;protocol_version&quot;: 4,<br>            &quot;control_connection_timeout&quot; : 20,<br>            &quot;connect_timeout&quot; : 20,<br>            &quot;execution_profiles&quot; = {EXEC_PROFILE_DEFAULT: profile}<br>        }<br># Connecting from Cassandra cluster <br><br>cluster = Cluster(*cluster_args, **cluster_kwargs)<br>session = cluster.connect()<br><br># Exxecuting Query <br>print(session.execute(&quot;SELECT name, age, email FROM users&quot;).one())<br><br></pre><p><strong>Enforcing consistency at the query level</strong></p><pre># importing the required modules from the Cassandra Python driver<br>from cassandra import ConsistencyLevel<br>from cassandra.query import SimpleStatement<br>from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT<br>from cassandra.policies import WhiteListRoundRobinPolicy, DowngradingConsistencyRetryPolicy<br>from cassandra.query import tuple_factory<br>from cassandra.auth import PlainTextAuthProvider<br><br># defines a configuration dictionary _CONFIG that contains connection details <br># such as nodes, username, password, and keyspace for the database.<br>_CONFIG = {<br>    &quot;dev&quot;: {<br>        &quot;nodes&quot;: [&quot;127.0.0.1&quot;],<br>        &quot;username&quot;: &quot;test&quot;,<br>        &quot;password&quot;: &quot;&quot;,<br>        &quot;keyspace&quot;: &quot;test&quot;<br>    }<br>}<br><br># creates an instance of the PlainTextAuthProvider class <br># passing in the username and password from the configuration dictionary.<br>auth_provider = PlainTextAuthProvider(<br>            username=_CONFIG[&quot;username&quot;], password=_CONFIG[&quot;password&quot;]<br>        )<br>        <br>cluster_args = []<br><br># Add remaining cluster options.<br>cluster_kwargs = {<br>            &quot;auth_provider&quot;: auth_provider,<br>            &quot;protocol_version&quot;: 4,<br>            &quot;control_connection_timeout&quot; : 20,<br>            &quot;connect_timeout&quot; : 20<br>        }<br># Connecting from Cassandra cluster<br><br>cluster = Cluster(*cluster_args, **cluster_kwargs)<br>session = cluster.connect()<br><br># Creating a SimpleStatement object and specifying the result consistency level<br>query = SimpleStatement(<br>    &quot;INSERT INTO users (name, age) VALUES (%s, %s)&quot;,<br>    consistency_level=ConsistencyLevel.QUORUM)<br><br># Executing the query<br>session.execute(query, (&#39;John&#39;, 42))</pre><h4><strong>Approach 2</strong></h4><p>Although approach 1 uses Cassandra’s ability to tune consistency levels, it comes with the cost of a reduction in high availability. Most of the time this reduction seems negligible in a cluster with more than 5 nodes by implementing proper node caching and tuning the queries. <br>Approach 2 explains a logical design to achieve the same response without varying the consistency levels in Cassandra.</p><figure><img alt="Logical design to achieve strong consistency" src="https://cdn-images-1.medium.com/max/670/1*E0PqVX4Cq-tA0zKhc3kCrw.png" /><figcaption>Logical design to achieve strong consistency Source: Author</figcaption></figure><p><strong>Implementation</strong></p><p>We introduced an in-memory data store in between the client query and Cassandra DB.</p><p>Writing Data:</p><ol><li>Create an object and write it in both MemCache and Cassandra.</li><li>MemCache record will have a TTL associated with it. Theoretically, TTL should be greater than <em>CrossNodeLatency</em> time.</li></ol><p>Reading Data:</p><ol><li>The client application will fire the query. At first, it will look out for data in MemCache.</li><li>If the record in MemCache is not expired return the record.</li><li>If MemCache has expired or no record is found in MemCache, query the Database and return to the Client</li></ol><h3>Conclusions</h3><p>Cassandra is a distributed NoSQL database that provides tunable consistency levels to balance between strong consistency and high availability. Consistency levels specify how many replica nodes must acknowledge read or write operations for the operation to be considered successful.</p><p>While Cassandra’s consistency levels offer the flexibility to establish the right balance between consistency and availability, they will not solve every consistency problem. Developers will still need to design data storage systems with their own consistency models that ensure strong consistency for critical fields. Setting the right consistency level is a balancing act. Data architects need to ensure that their application’s data model is optimized for strong consistency without sacrificing availability.</p><p>The LOCAL_QUORUM consistency level is ideal for achieving strong consistency, as it guarantees a response from a majority of replica nodes. Cassandra’s write consistency levels come with a reduction in availability, as more nodes must be operational for the write to succeed.</p><h3><strong>References</strong></h3><ul><li><a href="https://docs.datastax.com/en/developer/python-driver/3.21/getting_started/">DataStax Documentation</a></li><li><a href="https://www.oreilly.com/library/view/cassandra-the-definitive/9781098115159/">Cassandra: The Definitive Guide, 3rd Edition</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23af926928c1" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/consistency-model-in-cassandra-23af926928c1">Consistency Model in Cassandra</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sort-Merge-Join in Spark]]></title>
            <link>https://akash-dwivedi.medium.com/sort-merge-join-in-spark-9ebf40436bd3?source=rss-30753b40e37d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ebf40436bd3</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Akash Dwivedi]]></dc:creator>
            <pubDate>Mon, 14 Sep 2020 04:27:32 GMT</pubDate>
            <atom:updated>2026-01-16T03:45:52.294Z</atom:updated>
            <content:encoded><![CDATA[<p>Working with huge datasets and performing joins is always a bottleneck. Most of the time job gets failed due to <em>java.lang.OutOfMemoryError </em>. In this blog I am trying to provide a join method that is robust enough to handle huge datasets and if used smartly then fast enough to provide the desired results.</p><p>Sort-Merge-Bucket Join is a combination of two steps. In the first step it orders the joined data-sets. The second operation is the merge of sorted data into a single place by simply iterating over the elements and assembling the rows having the same value for the join key.</p><p>In SPARK Broadcast Hash join seems to be most performant join strategy , it is applicable to a small set of scenarios where datasets are able to fit in hash partition memory. Shuffle Hash Join &amp; Sort Merge Join are the true work-horses of Spark SQL.</p><p>The property which leads to setting the Sort-Merge Join :</p><pre><em>spark.sql.join.preferSortMergeJoin</em></pre><p>The class involved in sort-merge join we should mention</p><pre>org.apache.spark.sql.execution.joins.SortMergeJoinExec</pre><p>Below is the simple script which shows you how Sort-Merge-Join works.</p><pre>from pyspark.sql import SparkSession<br> <br> spark = SparkSession.builder\<br>         .appName(&quot;sort-merge-analysis&quot;)\<br>         .master(&quot;yarn&quot;)\<br>         .config(&quot;spark.sql.join.preferSortMergeJoin&quot;, &quot;true&quot;)\<br>         .config(&quot;spark.sql.autoBroadcastJoinThreshold&quot;, &quot;1&quot;)\<br>         .config(&quot;spark.sql.defaultSizeInBytes&quot;, &quot;100000&quot;)\<br>         .enableHiveSupport()\<br>         .getOrCreate()<br> <br> orders = spark.read.json(&#39;retail_db_json/orders&#39;)<br> order_item = spark.read.json(&#39;retail_db_json/order_items&#39;)<br> wide_table = order_item.join(orders , orders.order_id==order_item.order_item_order_id)<br> wide_table.show()</pre><p>Here is the query Execution plan:</p><pre>== Physical Plan ==<br> <strong>*SortMergeJoin</strong> [order_item_order_id#26L], [order_id#10L], Inner<br> :- *Sort [order_item_order_id#26L ASC NULLS FIRST], false, 0<br> :  +- Exchange hashpartitioning(order_item_order_id#26L, 200)<br> :     +- *Project [order_item_id#25L, order_item_order_id#26L, order_item_product_id#27L, order_item_product_price#28, order_item_quantity#29L, order_item_subtotal#30]<br> :        +- *Filter isnotnull(order_item_order_id#26L)<br> :           +- *FileScan json</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/21/0*EU6IwUcrkHU6zai3" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*qirs0En-1qd6jOAT.png" /></figure><h3>Deep Dive</h3><p>The Spark SQL planner chooses to implement the join operation using ‘SortMergeJoin’. The precedence order for equi-join implementations (as in Spark 2.2.0) is as follows:</p><ol><li>Broadcast Hash Join</li><li>Shuffle Hash Join: if the average size of a single partition is small enough to build a hash table.</li><li>Sort Merge: if the matching join keys are sortable.</li></ol><p>Next thing which requires attention is Bucketing.</p><p>Bucketing is one of the famous optimization technique which is used to avoid data shuffle.</p><p>S-M-B joins gives its best performance when the datasets use buckets to store the data.</p><p>Bucketing is used exclusively in <strong>FileSourceScanExec</strong> physical operator (when it is requested for the input RDD and to determine the partitioning and ordering of the output).</p><p>In the above DAG if we inspect properly, we can clearly see there is an <strong>Exchange </strong>stage present. When we use <strong>pre-shuffled bucketed tables</strong> on join key, we take the advantage of offset provided by bucketing.</p><p>Below I will try to run the same operation and this time the two tables will be pre-shuffled bucketed on join key.</p><p>Here is the code snippet</p><pre>from pyspark.sql import SparkSession<br> <br> spark = SparkSession.builder\<br>         .appName(&quot;sort-merge-bucket&quot;)\<br>         .master(&quot;yarn&quot;)\<br>         .config(&quot;spark.sql.join.preferSortMergeJoin&quot;, &quot;true&quot;)\<br>         .config(&quot;spark.sql.autoBroadcastJoinThreshold&quot;, &quot;1&quot;)\<br>         .config(&quot;spark.sql.defaultSizeInBytes&quot;, &quot;100000&quot;)\<br>         .enableHiveSupport()\<br>         .getOrCreate()orders = spark.sql(&quot;select * from orders_bucketed&quot;)<br>order_item = spark.sql(&quot;select * from order_item_bucketed&quot;)wide_table = order_item.join(orders , orders.order_id==order_item.order_item_order_id)<br>wide_table.show()</pre><p>DAG generated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/27/0*ygFfoyCMIeMCvgD1" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*SQvAXUZZ9sC8z2aS.png" /></figure><p>As we can clearly see there is no exchange step present in DAG</p><p><strong>Few Observations:</strong></p><ul><li>The number of buckets has to be between 0 and 100000 exclusive or Spark SQL throws an<em> AnalysisException</em></li><li>The number of partitions on both sides of a join must be exactly the same.</li><li>As of <a href="https://issues.apache.org/jira/browse/SPARK-23803">Spark 2.4</a>, Spark SQL supports <strong>bucket pruning</strong> to optimize filtering on bucketed column (by reducing the number of bucket files to scan).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ebf40436bd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>