<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brajesh Pandey on Medium]]></title>
        <description><![CDATA[Stories by Brajesh Pandey on Medium]]></description>
        <link>https://medium.com/@pandey.brajesh?source=rss-1138fbc842e6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Brajesh Pandey on Medium</title>
            <link>https://medium.com/@pandey.brajesh?source=rss-1138fbc842e6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:16:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pandey.brajesh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Netezza on Cloud (NPSaaS) integration with Tableau]]></title>
            <link>https://medium.com/modern-data-warehouse/netezza-on-cloud-npsaas-integration-with-tableau-5d3a683dc86e?source=rss-1138fbc842e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d3a683dc86e</guid>
            <dc:creator><![CDATA[Brajesh Pandey]]></dc:creator>
            <pubDate>Sat, 18 May 2024 00:54:43 GMT</pubDate>
            <atom:updated>2024-05-18T00:54:43.645Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*v8vaQlcfATnZJjNbpxmFMQ.png" /><figcaption><strong>Netezza + Tableau</strong></figcaption></figure><h3><strong>Tableau</strong></h3><p>A visual analytics platform to help people see and understand data and to explore and manage data, and faster to discover and share insights that can change businesses and the world. For more details — refer to <a href="https://www.tableau.com/">Tableau documentation</a>.</p><h3><strong>NPSaaS</strong></h3><p>A high performance petabyte scale fully managed pay-as-you-go cloud data warehouse service on AWS and Azure. IBM Netezza Performance Server provides a high-performance, cloud-native data warehouse designed for scalable analytics and insights accessible in a single platform bringing decades of innovation in data governance and security, in-database analytics and ML and hybrid-columnar processing to your data in the cloud. For more details — refer to <a href="https://www.ibm.com/products/netezza">Netezza documentation</a>.</p><blockquote><strong>Time Travel Feature</strong></blockquote><blockquote>NPSaaS time travel feature enables database users to retrieve and analyze historical data, and reconstruct deleted data. It supports users to run query using system time with AS OF, BEFORE, BETWEEN and FROM time expression to travel the temporal table as per the use case.</blockquote><h3><strong>Use-case: Anomaly detection in retail warehouse</strong></h3><p>In the following section, we will go over a “retail” use case where we will use Tableau to visually analyze warehouse inventory, find anomaly in ETL batch jobs and then reconstruct the data to the previous valid state using NPSaaS’s time travel feature.</p><p>To connect “Tableau” to “NPSaaS”, one would need to navigate to <em>“Netezza connector”</em> in Tableau desktop and input details to make connection as given below.</p><h4><strong>IBM Netezza Connector</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q-fZbL1DgxFQ-hhtwwI-pw.png" /></figure><h3><strong>Let’s start with the demo.</strong></h3><p>As you can see below, once connected “Retails” database, there are 2 tables listed under it.</p><p><strong>“items”</strong> — A table to store information about items. This table don’t get updated frequently (considered a regular Netezza table).</p><p><strong>“warehouse”</strong> — A time travel table to store items inventory. This table gets updated periodically through a batch job every day.</p><p><strong>Data exploration:</strong> Tableau provides a great visual tool to explore the data. For example, I was able to simply drag / drop both the tables and it automatically detected a relation between the tables for join, very neat. I was also able to view the result and understand the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lZPVWfDbipAF29olHWdZPQ.png" /><figcaption><strong>Data exploration</strong></figcaption></figure><p>To make the demo simpler and easier to understand, I limited warehouse item to only “Bike” and “Helmet”. ETL batch job runs every day and updates inventory for “Bike” and “Helmet” proportionally (1:1) in the warehouse.</p><h4><strong>Day 1 — ETL first batch</strong></h4><p>Let’s look at the view of the data (in Tableau sheet) when the first batch was inserted to “warehouse” table below. As you can see, bike and helmet counts were updated proportionally. For example, both “Bike-kids” and associated “Helmet-kids” had the same quantity i.e 25.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gr0EtHwEQ4EByQmzBj-7tA.png" /><figcaption><strong>First Batch</strong></figcaption></figure><h4><strong>Day 2 — ETL second batch</strong></h4><p>After a successful first batch, inventory in the warehouse was updated with second batch the next day.</p><p>Let’s look at the data below and see if bike and helmet counts were updated proportionally for respective associated pairs.</p><p>Data looked correct. For ex, 100 items were added to “Bike-girls” since the last batch and the same quantity was added to associated “Helmet-girls” as well, it was proportional.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5x451Ey6gSU16Ow72Fm2w.png" /><figcaption><strong>Second Batch</strong></figcaption></figure><h4><strong>Day 3 — ETL third batch</strong></h4><p>Let’s took a look at the data after 3rd batch update to the warehouse. As you see quantity for most of the bike-helmet pairs got updated proportionally, for example, 25 quantities were added to “Bike-men” and the same for “Helmet-men” since the last batch. However, <strong>there was a discrepancy</strong> between “Bike-kids” and “Helmet-kids” pair where “Bike-kids” product had total 50 items and “Helmet-kids” had 75. That pair was not updated proportionally as analyzed by the charts below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eooS73K4oE0TfQZQwc_DYQ.png" /><figcaption><strong>Third Batch — anomaly detected</strong></figcaption></figure><p>After further batch analysis, it was found that there was no items received for “Bike-kids” and “Helmet-kids” pair in the warehouse for third batch, But by mistake total count for “helmet-kids” was updated incorrectly.</p><p><strong>To correct the anomaly</strong> for “helmet-kids” item, NPSaaS time travel feature syntax <strong><em>“FOR SYSTEM_TIME AS OF &lt;day 2 batch timestamp&gt;”</em></strong> was used to find the previous valid value. Resolution was to update warehouse table with that previous value for “Helmet-kids”.</p><p>After correcting “Helmet-kids“, in the chart below, quantities for respective bike and helmet pairs were updated correctly and proportionally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_ZU4WSdBLRbyfxE57D5eA.png" /><figcaption><strong>Third Batch — Corrected</strong></figcaption></figure><p>Tableau provides a neat option to <strong>create a dashboard</strong> where you can bring all the sheets together to visualize whole story in a single view as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fycn4FGfb4tYdGGLn_1kAg.png" /></figure><h4><strong>To Summarize</strong></h4><p>In this blog, we talked about how Tableau can easily be integrated with NPSaaS data warehouse to analyze and visualize data together. It further extended with a real life ETL use-case and how NPSaaS time travel feature can help solve the anomaly problem easily.</p><h4><strong>References</strong></h4><p><a href="https://www.ibm.com/products/netezza">Netezza Performance Server | IBM</a></p><p><a href="https://www.tableau.com/">https://www.tableau.com</a></p><ul><li><a href="https://cloud.ibm.com/docs/netezza?topic=netezza-introducing_tt">IBM Cloud Docs</a></li><li><a href="https://medium.com/@akbg/time-travel-in-netezza-performance-server-8390cf125fcc">Time Travel in Netezza Performance Server</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d3a683dc86e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/modern-data-warehouse/netezza-on-cloud-npsaas-integration-with-tableau-5d3a683dc86e">Netezza on Cloud (NPSaaS) integration with Tableau</a> was originally published in <a href="https://medium.com/modern-data-warehouse">Modern data warehouse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A java toolkit for Apache Iceberg open table format]]></title>
            <link>https://medium.com/@pandey.brajesh/a-java-toolkit-for-apache-iceberg-open-table-format-64c329d6c719?source=rss-1138fbc842e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/64c329d6c719</guid>
            <dc:creator><![CDATA[Brajesh Pandey]]></dc:creator>
            <pubDate>Sun, 14 May 2023 01:55:06 GMT</pubDate>
            <atom:updated>2023-07-27T00:08:45.797Z</atom:updated>
            <content:encoded><![CDATA[<p>( <a href="https://medium.com/u/1138fbc842e6">Brajesh Pandey</a>, <a href="https://medium.com/u/3709fcd201e8">Shabana Baig</a>)</p><h3><strong>Introduction</strong></h3><p>Data and AI industry is trending towards adapting Open Data Lakehouse ecosystem using technology such as Apache Iceberg etc. However, there are not enough open-source toolkits available to access Open data lakehouse.</p><p>“Java-iceberg-toolkit” is a Java implementation for performing operations on Apache Iceberg and Hive tables to enable open data lakehouse access to developers, data scientists and DB users.</p><p>The toolkit aims at overcoming one of the major challenges which was to find a unified tool that can help interact with iceberg tables.</p><h3><strong>Apache Iceberg — Java toolkit</strong></h3><p>If you don’t know about “Apache Iceberg”, you are not alone :) you can find more information about that here (<a href="https://iceberg.apache.org/">https://iceberg.apache.org/</a>). Basically it is <strong><em>an open table format </em></strong>for huge analytic datasets.</p><p>“Apache Iceberg” provides many functionalities one that caught my eyes was <strong><em>ACID</em></strong><em> (</em><strong><em>atomicity, consistency, isolation, and durability</em></strong><em>) </em>support for tables. Considering the boom in open datasets (Parquet, AVRO, ORC etc), we wanted to explore this.</p><p>When we started exploring Iceberg we quickly realized that we could not find enough APIs or example to perform some of the basic table level operations. To help us and others, we created an <strong><em>“OpenSource toolkit” called java-iceberg-toolkit (</em></strong><a href="https://github.com/IBM/java-iceberg-toolkit">https://github.com/IBM/java-iceberg-toolkit</a>).</p><p><strong>— This toolkit supports the following operations —</strong></p><ul><li>Provides a simple and easy-to-use interface to interact with Apache Iceberg and Hive tables.</li><li>Provides APIs to perform operations on Iceberg tables and Hive tables.</li><li>Supports all primitive data types.</li><li>Supports most of the operations on Iceberg and Hive tables:</li><li>Create namespace</li><li>Create table</li><li>Get schema of a plan</li><li>Get plan task</li><li>Read table</li><li>Write table</li></ul><p>(more details: <a href="https://github.com/IBM/java-iceberg-toolkit#supported-operations">https://github.com/IBM/java-iceberg-toolkit#supported-operations</a>)</p><h4><strong>java-iceberg-toolkit provides the following interfaces</strong></h4><p><strong>A CLI </strong>which is ready to use when the code is packaged and all configurations are in place. (<a href="https://github.com/IBM/java-iceberg-toolkit#cli-2">https://github.com/IBM/java-iceberg-toolkit#cli-2</a>)</p><pre>$ java -jar &lt;jar_name&gt; --help<br>usage: java -jar &lt;jar_name&gt; [options] command [args]<br>    --format &lt;iceberg|hive&gt;       The format of the table we want to<br>                                  display<br> -h,--help                        Show this help message<br> -o,--output &lt;console|csv|json&gt;   Show output in this format<br>    --snapshot &lt;snapshot ID&gt;      Snapshot ID to use<br> -u,--uri &lt;value&gt;                 Hive metastore to use<br> -w,--warehouse &lt;value&gt;           Table location<br><br>Commands:<br>  drop                 Drop a table or a namespace<br>  schema               Fetch schema of a table<br>  read                 Read from a table<br>  commit               Commit file(s) to a table<br>  list                 List tables or namespaces<br>  type                 Fetch table type<br>  uuid                 Fetch uuid of a table<br>  spec                 Fetch partition spec of a table<br>  rename               Rename a table a table<br>  create               Create a table or a namespace<br>  files                List data files of a table<br>  location             Fetch table location<br>  describe             Get details of a table or a namespace<br>  write                Write to a table<br>  snapshot             Fetch latest or all snapshot(s) of a table</pre><p><strong>Programmable APIs</strong></p><p>For ex,</p><ul><li><strong>Get schema of a Hive table:</strong></li></ul><pre>import iceberg.HiveConnector;<br><br>HiveConnector connector = new HiveConnector(uri, warehouse, namespace, table);<br>Schema schema = connector.getTableSchema();</pre><ul><li><strong>Get schema of an Iceberg table:</strong></li></ul><pre>import iceberg.IcebergConnector;<br>import org.apache.iceberg.Schema;<br><br>IcebergConnector connector = new IcebergConnector(uri, warehouse, namespace, table);<br><br>Schema schema = connector.getTableSchema();</pre><ul><li><strong>Create an unpartitioned Iceberg table</strong></li></ul><pre>import iceberg.IcebergConnector;<br><br>import org.apache.iceberg.PartitionSpec;<br>import org.apache.iceberg.Schema;<br>import org.apache.iceberg.types.Types;<br><br>IcebergConnector connector = new IcebergConnector(uri, warehouse, namespace, null);<br>Schema schema = new Schema(<br>            Types.NestedField.required(1, &quot;ID&quot;, Types.IntegerType.get()),<br>            Types.NestedField.required(2, &quot;Name&quot;, Types.StringType.get()),<br>            Types.NestedField.required(3, &quot;Price&quot;, Types.DoubleType.get()),<br>            Types.NestedField.required(4, &quot;Purchase_date&quot;, Types.TimestampType.withoutZone())<br>            )<br>        );<br>PartitionSpec spec = PartitionSpec.unpartitioned();<br>boolean overwrite = false;<br>connector.createTable(schema, spec, overwrite);</pre><p>These APIs are really powerful to solve some use-cases, for example, bulk load ingestion of files directly into Iceberg table.</p><h3><strong>Use-Case</strong></h3><p><strong>Description : </strong>Bulk copy of parquet files to data lake in Iceberg table format.</p><p><strong>Steps:</strong></p><ul><li><strong><em>Create an Iceberg table</em></strong></li></ul><pre>import iceberg.IcebergConnector;<br><br>import org.apache.iceberg.PartitionSpec;<br>import org.apache.iceberg.Schema;<br>import org.apache.iceberg.types.Types;<br><br>IcebergConnector connector = new IcebergConnector(uri, warehouse, namespace, null);<br>Schema schema = new Schema(<br>            Types.NestedField.required(1, &quot;ID&quot;, Types.IntegerType.get()),<br>            Types.NestedField.required(2, &quot;Name&quot;, Types.StringType.get()),<br>            Types.NestedField.required(3, &quot;Price&quot;, Types.DoubleType.get()),<br>            Types.NestedField.required(4, &quot;Purchase_date&quot;, Types.TimestampType.withoutZone())<br>            )<br>        );<br>PartitionSpec spec = PartitionSpec.builderFor(schema)<br>      .year(&quot;hour&quot;)<br>      .build();<br>boolean overwrite = false;<br>connector.createTable(schema, spec, overwrite);</pre><ul><li><strong><em>Write data to Iceberg table</em></strong></li></ul><pre>import iceberg.IcebergConnector;<br><br>IcebergConnector connector = new IcebergConnector(uri, warehouse, namespace, table);<br><br>String record = &#39;{&quot;records&quot;:[{&quot;ID&quot;:1,&quot;Name&quot;:&quot;Testing&quot;,&quot;Price&quot;: 1000,&quot;Purchase_date&quot;:&quot;2022-11-09T12:13:54.480&quot;}]}&#39;;<br>String outputFile = null;<br>String dataFiles = connector.writeTable(record, outputFile);<br></pre><p>Note: If you already have data files stored somewhere, you can move to the next commit phase.</p><ul><li><strong><em>Commit to iceberg table</em></strong></li></ul><pre>import iceberg.IcebergConnector;<br><br>IcebergConnector connector = new IcebergConnector(uri, warehouse, namespace, table);<br>String dataFiles = &#39;{&quot;files&quot;:[{&quot;file_path&quot;:&quot;path_a&quot;}]}&#39;;<br>connector.commitTable(dataFiles);</pre><ul><li><strong><em>Read the iceberg table</em></strong></li></ul><pre>import iceberg.IcebergConnector;<br><br>import java.util.List;<br><br>IcebergConnector connector = new IcebergConnector(uri, warehouse, namespace, table);<br>List&lt;List&lt;String&gt;&gt; records = connector.readTable();<br>for (List&lt;String&gt; record : records) {<br>    for (int x = 0; x &lt; record.size(); x++) {<br>        String comma = x == record.size() - 1 ? &quot;&quot; : &quot;, &quot;;<br>        System.out.print(record.get(x) + comma);<br>    }<br>    System.out.println();<br>}</pre><h3>Conclusion</h3><p>Open source “java-iceberg-toolkit” is a Java implementation for performing operations on Apache Iceberg and Hive tables. It was created to help developers understand low level Iceberg APIs and can be used seamlessly to integrate with other products.</p><p><em>Please stay tuned for the next blog on “How to write a Java App Server to improve performance of Iceberg APIs”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*3Nk-uVBnuJ5LnSjjIx0z-A.jpeg" /><figcaption>watsonx.data-Netezza-Iceberg</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64c329d6c719" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Method of configuring local storage solution for legacy data warehouse]]></title>
            <link>https://medium.com/modern-data-warehouse/method-of-configuring-local-storage-solution-for-legacy-data-warehouse-4da54035bea8?source=rss-1138fbc842e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4da54035bea8</guid>
            <dc:creator><![CDATA[Brajesh Pandey]]></dc:creator>
            <pubDate>Thu, 09 Jun 2022 22:36:43 GMT</pubDate>
            <atom:updated>2022-06-10T00:47:37.184Z</atom:updated>
            <content:encoded><![CDATA[<p>Traditional legacy analytic mpp data warehouse applications were optimized to discard unnecessary rows before sending them over the network to perform work, but modern cloud storage solutions negate this optimization by performing their own distribution of the data. This limitation plus the inability to schedule data partitions where the data resides (in case of failures) are big factors in decision making for an organization to move their legacy data warehouse workloads to a Cloud Native Platform. One of the solution to solve this in my opinion is mentioned as follows.</p><p>This document mainly focuses on -</p><ul><li>A high performant storage solution that can be deployed on a Container Orchestration (such as OpenShift) running on on-prem with locally attached drives and provides close to bare metal drives performance.</li><li>Solution supports resiliency and redundancy; provides an interface to select supported redundancy profile.</li></ul><p>Legacy analytic data warehouse mainly cares about performance and wants to run workloads as far as possible very close to bare metal layer having full control and knowledge about I/O stack. This provides a deployable high performant optimized storage layout tailored for legacy databases on a Cloud Native Platform that provides a bare metal performance with offering enough controlling / configurable flexibility to data warehouse application.</p><p>We call this solution Horcrux going forward in the document.</p><ul><li>The solution uses LVM, RAID1, DM, MD and iSCSI technologies based on an optimized storage topology to create a faster and resilient storage layout over local storage and remote iSCSI devices.</li><li>This solution gets deployed using Kubernetes CSI driver onto a Cloud Native Platform such as Kubernetes or OpenShift.</li><li>Storage topology is integrated with Kubernetes Scheduler to take an optimized decision while placing a data partition application pod to a node.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*Gnh_zdjqcJvLZiY3nZgogA.png" /><figcaption>High Level block diagram</figcaption></figure><p>Storage layout ensures resiliency and redundancy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*uPqKNIwG1wr0BstI_Kdakw.png" /><figcaption>Storage layout</figcaption></figure><p><strong>HA LVM Cluster (availability zone)</strong></p><ul><li>HA-LVM chosen over CLVMD because of active/passive nature and performance</li><li>Depending on total deployable nodes, there could be more than one HA LVM Cluster.</li><li>By default, 7 nodes per HA LVM cluster. There will be an interface provided to change it.</li><li>Each HA LVM cluster can have maximum 16 nodes.</li><li>One storage controller pod to manager all HA LVM clusters.</li><li>Pods deployed as a daemonset across cluster</li><li>Pods expose all drives as iscsi targets</li><li>Pods mount all remote drives as iscsi initatior</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*VQThvr2ZfXhza8kYY3tH0A.png" /><figcaption>Storage cluster using HA LVM</figcaption></figure><h4>Kubernetes CSI controller</h4><ul><li>It is responsible for <em>handling a PVC create request</em> (storage allocation request).</li><li>It is <em>an extension of Kubernetes provided CSI apis</em> to support an external storage provisioner.</li></ul><p><strong>CreateVolume</strong>(ctx context.Context, req *csi.CreateVolumeRequest)</p><p><strong>DeleteVolume</strong>(ctx context.Context, req *csi.DeleteVolumeRequest)</p><p><strong>ControllerExpandVolume</strong>(ctx context.Context, req *csi.ControllerExpandVolumeRequest)</p><p><strong>CreateSnapshot</strong>(ctx context.Context, req *csi.CreateSnapshotRequest)</p><p><strong>DeleteSnapshot</strong>(ctx context.Context, req *csi.DeleteSnapshotRequest)</p><ul><li>It’s running as a <em>“deployment” (a K8S term) </em>on one of the OpenShift master node.</li><li>It will be deployed by Openshift operator.</li></ul><h4>CSI Node plugin</h4><ul><li>It is responsible for attaching / mounting / binding a PVC to an application.</li><li>It is <em>an extension of Kubernetes provided CSI apis</em> to support an external storage provisioner.</li></ul><p><strong>NodeStageVolume</strong>(ctx context.Context, req *csi.NodeStageVolumeRequest)</p><p><strong>NodeUnstageVolume</strong>(ctx context.Context, req *csi.NodeUnstageVolumeRequest)</p><p><strong>NodePublishVolume</strong>(ctx context.Context, req *csi.NodePublishVolumeRequest)</p><ul><li>It is running as “Daemonset” (K8S term). Basically, on all horcrux labelled worker (by default all worker nodes).</li><li>It will be deployed by Openshift operator.</li></ul><h4>Storage Node pod</h4><ul><li>Runs on all Horcrux labelled worker nodes (by default all worker nodes).</li><li>Runs as a “Statefulset” (K8S term).</li><li>Responsible for setting up iSCSI initiators and targets.</li><li>It has a light weight REST server running. REST is used to handle communication to/from storage node, CSI pods and storage controller.</li><li>On demand, provide provisioning status and local volume status to Storage controller pod.</li><li>It will be deployed by Openshift operator.</li></ul><h4>Storage controller pod</h4><ul><li>Running on one of the Openshift master node.</li><li>Handles storage layout provisioning by keeping track of “storage pod” provisioning status.</li><li>Initiate a REST request to one of “storage node” to create VG for that cluster. If “storage node” is not quorum leader (pacemaker), it forwards that request to storage pod that’s also packemaker leader.</li><li>Monitoring health of storage HA cluster(s) and can initiate resync operation if spare capacity is allocated for a cluster.</li><li>It will be deployed by Openshift operator as a deployment kubernetes object.</li></ul><h4>Scheduler</h4><ul><li>A custom kubernetes scheduler will be installed which will deploy pods to the baremetal node that the primary mirror volume lives on.</li><li>If the node/disk where the primary mirror volume lives is inaccessible, the scheduler will deploy the pod on the node containing the secondary mirror volume.</li><li>The scheduler application will need to communicate with the storage rest server in order to determine the correct node to deploy a pod onto.</li></ul><h4>Operator</h4><ul><li>Horcrux uses “Openshift Operator” to deploy all its elements such as storage node pod, storage controller pod, node plugin pod, and controller plugin pod.</li><li>It’s written using operator-sdk and golang.</li><li>It uses Openshift internal registry to pull and push horcrux images.</li><li>Horcrux CRD (Custom Resource Definition) interface provides the following configurable options</li><li>NUM_NODES_PER_CLUSTER (by default: RAID7)</li><li>SPARE_CAPACITY_PER_CLUSTER (by default: 0)</li><li>PROFILE (RAID1, RAID5 ) (by default: RAID1)</li><li>HORCRUX_SCHEDULER (by default: enabled)</li></ul><h4>Operator deployment flow</h4><p>Horcrux operator controls deployment. Once Horcrux operator and CRD deployed, it’s reconciled loop kicks in and deploys other pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pvYJGej_aYeu5rsWu5jfAA.png" /></figure><h4>Cluster Provisioning flow</h4><p>Once all storage node pods deployed, storage controller pod will takeover to setup Horcrux cluster with storage layout. In this example, let’s say there are 3 storage nodes. It basically goes through 3 states.</p><p><strong>state:NodeDiscovering</strong></p><ul><li>Storage controller pod tries to discover all storage node pods.</li></ul><p><strong>state:NodeReady</strong></p><ul><li>Storage controller pod discovered storage node pods and they are up and running.</li><li>At the point in time, storage node pods are ready to accept REST request.</li></ul><p><strong>state:DevicesDiscovered</strong></p><ul><li>Local and iSCSI remote devices discovered</li><li>Pacemaker cluster is setup</li></ul><p><strong>state:Running</strong></p><ul><li>HA LVM storage layout is setup.</li></ul><p>There is an intermediate ‘ING’ state for the above 2 states as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*ljRGivxYBa13Ct9f4gQyig.png" /></figure><h4>Storage Volume Claim</h4><h4>Availability-zone</h4><p>Defines a boundary for number of nodes and persistent Volumes.</p><p>We will have a storage class for each availability zone, and while creating a PVC an appropriate one should be used.</p><p>PVC will be provisioned from a given availability zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*Q0vJ6AtMbfPZISDNWLZ-fA.png" /></figure><h4>Drive failure</h4><p>In the event of a drive failure, the pods running on the physical node will not be evicted. They’ll continue to run, however the storage volumes attached to these pods will now be read from their mirrors that are stored on a remote node. In the background, the storage system should begin replicating the volumes from the failed drive evenly across the cluster to protect against data loss. A disk replacement procedure will be followed, and the primary mirror volumes will be restored to their original drive. At this time performance will return to normal for the user with zero downtime. In the event of a second drive failure, the behaviour will be the same as single disk failure as long as there are replicas of all the failed volumes. If a second drive fails and contains the only complete replica of a volume (for example if replication of a failed volume has not yet completed), the storage system won’t be able to provide access to the storage, and the pods will need to be evicted. Data may be lost, and an admin may need to manually try to recover data.</p><h4>Node failure</h4><p>In the event of a node failure, pods which run on that node will be forcibly evicted. The custom scheduler will deploy the pods on the nodes where their volume’s secondary mirror partitions live. Once the pods are moved the system will run with degraded performance, since some physical nodes will now be oversubscribed with work. A node replacement procedure will be followed, and once the new node is added the data will be replicated back to the new node. At this point, even though they are running in a degraded state, we don’t want to evict pods to the new node since this will cause a service interruption. The admin will be able to run a command during scheduled maintenance which will evict necessary pods and schedule them back to the new node.</p><p>co-author: <a href="https://medium.com/u/71a42ed65e83">Mike DeRoy</a> <a href="https://medium.com/u/7e04cebff4c5">Aniket Kulkarni</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4da54035bea8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/modern-data-warehouse/method-of-configuring-local-storage-solution-for-legacy-data-warehouse-4da54035bea8">Method of configuring local storage solution for legacy data warehouse</a> was originally published in <a href="https://medium.com/modern-data-warehouse">Modern data warehouse</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>