<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aasha Medhi on Medium]]></title>
        <description><![CDATA[Stories by Aasha Medhi on Medium]]></description>
        <link>https://medium.com/@aashamedhi?source=rss-bc66bfa2727b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*L33QN2H32z63-acc.jpg</url>
            <title>Stories by Aasha Medhi on Medium</title>
            <link>https://medium.com/@aashamedhi?source=rss-bc66bfa2727b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:51:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aashamedhi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Hive 3 Replication : Orchestration]]></title>
            <link>https://aashamedhi.medium.com/apache-hive-3-replication-orchestration-8ad65e23ba62?source=rss-bc66bfa2727b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ad65e23ba62</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[replication]]></category>
            <dc:creator><![CDATA[Aasha Medhi]]></dc:creator>
            <pubDate>Tue, 15 Jun 2021 07:42:07 GMT</pubDate>
            <atom:updated>2021-06-15T07:42:07.297Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Hive 3 Replication : Orchestration</h3><p>Scheduling a Hive replication policy has become even more simple now. Hive 3 allows the orchestration within Hive. Now you can schedule replication policies using Hive CLI commands. Hive has its own scheduler and supports scheduled queries. Replication uses the hive scheduler to schedule replication queries.</p><p>Like I talked about the 2 steps of replication in my <a href="https://aashamedhi.medium.com/hive-3-replication-b7c639f9198a">previous blog</a> lets see how do we schedule replication using hive commands.</p><p>Lets take an example here. We will replicate sales db from primary cluster to DR</p><p><strong>Primary Cluster </strong>: Schedule repl dump command. Every 15 minutes a repl dump command will be triggered which will dump the data/metadata in the staging directory. If the previous dump is not loaded in the DR cluster yet, it will skip the dump. The load acknowledgement file will help here to determine if the dump is loaded or not.</p><ul><li>create database sales;</li><li>use sales;</li><li>create table items(id int);</li><li>insert into items(1);</li><li>create scheduled query every 15 minutes as repl dump sales with (‘hive.repl.rootdir’ = ‘&lt;common staging location&gt;’)</li></ul><p><strong>DR Cluster </strong>: Schedule repl load command. Every 15 minutes a repl load command will be triggered which will load the data/metadata from the staging directory. If the dump is not completed yet, subsequent load will be skipped. The dump acknowledgement file will help here to determine if the dump is complete or not.</p><ul><li>create scheduled query every 15 minutes as repl load sales into sales_bkup with (‘hive.repl.rootdir’ = ‘&lt;common staging location&gt;’)</li></ul><p><strong>Validation of DR Cluster </strong>: Once a complete run of replication is done(one dump and load), check the DR cluster for the replicated DB. It should have all the data and metadata for the DB.</p><ul><li>use sales_bkup;</li><li>select * from items; -&gt; should give a single row with value 1</li></ul><p><strong>Monitoring the replication </strong>: The scheduled queries allow you to monitor the status of the schedules.</p><ul><li>information_schema.scheduled_queries : This table lists down all the scheduled queries.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TdsynyGevPQbTvDX" /></figure><ul><li>sys.scheduled_executions : This table lists down all the scheduled executions. This also shows in which executor was the query scheduled. Last 10 executions for a scheduled query shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B5KxmNy7I3hTYK4a" /></figure><ul><li>sys.replication_metrics : This lists the replication status and metrics. This can also be used to tie the load and dump pair. The scheduled_execution_id is used to get the metrics for a particular run.The dump_execution_id is the scheduled_execution_id of the corresponding repl dump policy running on source(0 in case of dump metrics)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ouKOSyXM4fj8kHo4" /></figure><ul><li>All the above tables are present on both the clusters. The dump metrics can be obtained from the Primary Cluster and the load metrics can be obtained from the DR cluster.</li><li>Any non recoverable error details are also present as part of the replication_metrics table.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ad65e23ba62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Hive 3 Replication]]></title>
            <link>https://aashamedhi.medium.com/hive-3-replication-b7c639f9198a?source=rss-bc66bfa2727b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7c639f9198a</guid>
            <category><![CDATA[hivereplication]]></category>
            <category><![CDATA[replication]]></category>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[line-of-sight]]></category>
            <category><![CDATA[apache-hive]]></category>
            <dc:creator><![CDATA[Aasha Medhi]]></dc:creator>
            <pubDate>Tue, 15 Jun 2021 05:19:50 GMT</pubDate>
            <atom:updated>2021-06-15T05:48:47.029Z</atom:updated>
            <content:encoded><![CDATA[<p>No more line of sight requirement</p><p>Hive 2 Replication needed line of sight for replication. The source HS2 would talk to both source and target HMS and vice-versa. This would require HMS ports to be open. Specifically for on-prem to cloud replication, generally accessing on-prem from cloud is not a good idea. This violates the security requirements for customers. To overcome this, came the Hive3 replication without any line of sight requirement. It communicates through a shared staging location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wpbuw28imbq9wDE5e2T0UQ.png" /></figure><p><strong>Staging based Replication</strong></p><p>Replication in Hive is a 2 step process.</p><ul><li>Repl Dump — On the primary cluster the data and metadata is dumped to the staging location.</li><li>Repl Load — On the DR where the data needs to be replicated, data and metadata are loaded from the staging location.</li></ul><p>Staging location is shared between the Primary and DR cluster. Both the clusters needs write access to this location. This location can be in any one of the clusters or it could also be in any other FS.</p><p><strong>FS based Acknowlegement</strong></p><ul><li>Dump Acknowlegement : Once the dump is done, a dump acknowledgement file is kept at the staging location. The load on the DR cluster will only start once it sees the dump acknowledgment file.</li><li>Load Acknowlegement : Once the load is done, a load acknowledgement file is kept at the staging location. The next(future) dump on the Primary cluster will only start once it sees the load acknowledgment file.</li><li>Metadata Files : The metadata files will include the event id till which its replicated. This way the source HS2 doesn’t need to interact with the target HMS to get the last event till which its replicated. The other important metadata information like checkpointing etc is also stored as part of the staging location.</li></ul><p><strong>Why is this better?</strong></p><ul><li>No more line of sight requirement. The HMS ports on both the clusters need not be exposed. No more security concern.</li><li>Staging location can be on any shared File System. Should be writeable by both thee clusters.</li><li>No need to set up 2 way trust in case of secure clusters.</li><li>Ability to only dump and load metadata from staging location. Data can be copied directly from HDFS location on Primary clusters. This can be used for on-prem to on-prem replication where there is no issue of trust set up or security concern.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7c639f9198a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>