<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Itamar Syn-Hershko on Medium]]></title>
        <description><![CDATA[Stories by Itamar Syn-Hershko on Medium]]></description>
        <link>https://medium.com/@synhershko?source=rss-1289b2f838da------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_Bt6ec-b_cgRmXLtWxAt1g.png</url>
            <title>Stories by Itamar Syn-Hershko on Medium</title>
            <link>https://medium.com/@synhershko?source=rss-1289b2f838da------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:23:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@synhershko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Who’s Minding the Database? The Coming Crisis of the Generalist Engineer]]></title>
            <link>https://medium.com/@synhershko/whos-minding-the-database-the-coming-crisis-of-the-generalist-engineer-712cb9772237?source=rss-1289b2f838da------2</link>
            <guid isPermaLink="false">https://medium.com/p/712cb9772237</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Itamar Syn-Hershko]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 15:34:28 GMT</pubDate>
            <atom:updated>2026-03-01T15:34:28.582Z</atom:updated>
            <content:encoded><![CDATA[<p>The database market will reach $203.6 billion by 2027. <a href="https://www.gartner.com/en/newsroom/press-releases/2026-1-15-gartner-says-worldwide-ai-spending-will-total-2-point-5-trillion-dollars-in-2026">AI spending will hit $2.5 trillion in 2026</a>. And the people managing these systems are being told to stop specializing.</p><p>Three forces are creating a tension most engineering organizations haven’t reckoned with. Data platforms are exploding — the <a href="https://www.mordorintelligence.com/industry-reports/database-market">DBMS market</a> hit $119.7 billion in 2024 and is growing at 13–16% annually, with <a href="https://www.fortunebusinessinsights.com/cloud-database-market-110240">cloud databases at 19.9% CAGR</a>, graph databases at 26.4%, and <a href="https://www.marketsandmarkets.com/Market-Reports/vector-database-market-112683895.html">vector databases projected to surpass $8.9 billion by 2030</a>. AI workloads are the primary engine. Meanwhile, engineers are becoming generalists — Wix’s xEngineer, the “product engineer” trend, merged guilds, end-to-end ownership. Vibe-coded features suddenly gaining traction. AI amplifying breadth over deep stack specialization. And then there’s the third force: products that hit product-market fit now scale overnight. A fintech startup goes from prototype to 200 billion rows. An AI agent platform spawns a million database instances in three months. Everyone needs the platform reliable <em>yesterday</em>.</p><p>The industry is training engineers to own more and move faster while knowing less about each layer — at exactly the moment when data platforms are becoming more complex, more critical, and less forgiving.</p><h3>Data Platforms Are Eating Everything</h3><p>Gartner projects the DBMS market will grow from <a href="https://www.gartner.com/en/newsroom/press-releases/2026-02-03-gartner-forecasts-worldwide-it-spending-to-grow-10-point-8-percent-in-2026-totaling-6-point-15-trillion-dollars">$137 billion in 2025 to $203.6 billion by 2027</a> at a 16.8% CAGR. <a href="https://www.mordorintelligence.com/industry-reports/cloud-database-and-dbaas-market">Cloud databases already account for 64% of all DBMS spending</a>. The <a href="https://www.mordorintelligence.com/industry-reports/database-automation-market">database automation market</a> is on a tear — 24.38% CAGR, from $2.92 billion to $8.70 billion by 2030. The <a href="https://www.precedenceresearch.com/ai-data-management-market">AI data management market is projected to reach $234.95 billion by 2034</a>. None of this is incremental.</p><p>AI workloads are creating entirely new demands on database infrastructure. Vector search, embeddings, RAG pipelines — all need sub-millisecond access to fresh data. Agentic AI is changing who creates databases in the first place. PingCAP reports that <a href="https://www.pingcap.com/blog/agentic-ai-database-trends-that-will-define-2026/">more than 90% of new TiDB Cloud clusters are now created by AI agents</a>, not humans. One AI agent platform, Manus, created close to a million database tenants in three months. Gartner predicts <a href="https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025">40% of enterprise applications will embed task-specific AI agents by the end of 2026</a>, up from less than 5% in 2025. Multi-model requirements are becoming the norm: relational, document, vector, and graph — all at once.</p><p>Whether they planned for it or not, every company is becoming a data platform company. The question isn’t <em>if</em> you’ll need serious database infrastructure. It’s <em>when</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKOeaxTVgQU8AD5s5p4MEQ.png" /><figcaption>AI DBAs to the rescue of the xEngineer and Vibe Coder</figcaption></figure><h3>When Success Becomes an Ops Crisis</h3><p>Products that find product-market fit in this era don’t grow linearly. They explode. Look at OpenAI. A single PostgreSQL primary serves <a href="https://openai.com/index/scaling-postgresql/">800 million ChatGPT users, handling millions of queries per second</a> across nearly 50 read replicas. When ImageGen launched and went viral, write traffic surged more than 10x. The response required hacking PostgreSQL’s system catalogs to disable indexes — an undocumented internal operation — decomposing 12-table ORM-generated joins at the application layer, implementing cache-locking to prevent stampedes, and enforcing 5-second timeouts on all schema changes. They eventually offloaded write-heavy workloads to Cosmos DB entirely because PostgreSQL’s single-primary architecture couldn’t absorb the growth.</p><p>The person who led this work, Bohan Zhang, worked with Andy Pavlo at CMU’s Database Group and co-founded OtterTune, a database tuning startup. Not a generalist engineer figuring things out over a weekend — deep specialization applied under extreme pressure. As one <a href="https://pigsty.io/blog/db/openai-pg/">external analysis</a> put it: “Even with top-tier cloud database services, the customer needs to have sufficient knowledge and skill on the application and operations side.”</p><p>That’s the gap. When you’re processing financial data, healthcare records, or real-time trading signals, “we’ll figure it out” doesn’t cut it. Platforms backing large-scale services need reliability now — not in six months after you’ve hired a specialist. Hiring a senior DBA takes months. Training an engineer on ClickHouse internals or Elasticsearch shard allocation takes years. The platform doesn’t wait.</p><h3>The Brilliant Generalist, The Dangerous Operator</h3><p>Wix made the trend explicit in early 2026 when they <a href="https://www.aviransplace.com/post/embracing-the-ai-revolution-introducing-the-xengineer-at-wix">introduced the xEngineer</a> — a “design-first, not code-first” engineer who leverages AI as foundational infrastructure. They merged Frontend, Backend, and Mobile guilds into a single Engineering Guild. The reasoning: if AI abstracts away syntax, the <a href="https://medium.com/@natishalom/the-rewiring-of-the-engineering-org-lessons-from-wix-linearb-and-latent-space-974099efee9f">barriers between “Frontend” and “Backend” are artificial</a>. Engineers should own outcomes end-to-end, from design to deployment to operations. The North Star: AI generates the majority of production code, freeing engineers to focus on software design and architectural thinking.</p><p>Wix isn’t alone. Linear hires “Product Engineers” who own features from design to delivery. Vercel’s “Design Engineers” <a href="https://vercel.com/blog/design-engineering-at-vercel">design, build, and ship solutions autonomously</a>. Incident.io expects engineers to speak directly with customers and care more about outcomes than tools. T-shaped engineers, AI-native full-stack developers, cross-functional teams owning entire product slices — the industry is moving fast in this direction. In practice, it’s a massive shift-left. Fewer people owning more at higher scale. Engineers own their databases, their infrastructure, their observability, their deployment pipelines, their business outcomes.</p><p>But an xEngineer who owns a ClickHouse pipeline doesn’t know what nats_handle_error_mode = &#39;stream&#39; does to materialized views. A product engineer who owns an Elasticsearch cluster doesn&#39;t understand shard allocation when nodes go red. A full-stack developer who provisioned a PostgreSQL database doesn&#39;t know when to use CONCURRENTLY for index creation or how to tune autovacuum under heavy write loads. Effective managers and executors, yes. Database specialists, no.</p><p>The DBA skills gap compounds the problem. The <a href="https://www.mordorintelligence.com/industry-reports/database-automation-market">database automation market is growing at 24.38% CAGR</a> partly because skilled database administrators are scarce and organizations are turning to automation out of necessity. <a href="https://www.solarwinds.com/company/newsroom/press-releases/survey-finds-one-in-three-dbas-eye-career-move-as-demands-on-role-increase">SolarWinds reports</a> one in three DBAs is considering a career move due to increasing demands, with 75% reporting alert fatigue and 27 of every 40 work hours spent reacting to incidents rather than doing proactive work. The DBAs who remain are <a href="https://www.infoworld.com/article/3818889/the-future-is-far-from-doom-and-gloom-for-database-administrators.html">evolving into strategic architects</a> — approaching optimization with the entire data pipeline in mind. New ones aren’t being trained at anywhere near the rate the market demands.</p><h3>Monitoring Won’t Save You</h3><p>Most teams rely on Datadog, New Relic, or Dynatrace to catch problems. These tools collect metrics. They don’t explain <em>why</em> something is happening or <em>what to do about it</em>. Facts without a story.</p><p>What teams actually need isn’t “your CPU spiked.” They need: “You deployed changes three days ago. Today is prime time in one of the countries you serve. You received many more requests than usual. The index isn’t being used because the new query pattern bypasses it. CPU spiked because the database is doing full scans instead of index lookups. Add this index to fix it.” That’s the distance between a metric and an actionable diagnosis.</p><p>APM has matured for applications. Database observability hasn’t kept pace. Three areas where things consistently go wrong: code changes that produce different SQL patterns, schema migrations with unexpected side effects, and runtime behavior shifts under changing workloads. Developers need database guardrails, not dashboards. Call it the Grafana paradox: Grafana would show a table growing, Prometheus would show insert rates climbing, but nothing would flag that the data being inserted is 100% garbage from a misconfigured pipeline. Metrics tell you <em>what</em> is happening. They don’t tell you <em>what it means</em>.</p><h3>The Specialist That Scales</h3><p>If modern engineers are generalists by design, and data platforms require deep specialists by necessity, and the DBA talent pool is shrinking — the specialist must be AI.</p><p>Not a general-purpose AI assistant giving textbook answers. A specialized AI DBA with the encoded knowledge of a senior database administrator — system vitals collection, hand-curated playbooks, adaptive investigation chains, safety guardrails built for a specific database technology. The difference between a general-purpose AI SRE and a specialized AI DBA comes down to tools, not intelligence. A general AI engine will tell you to check your slow query log. An AI DBA with deep domain-specific tooling finds 95,000 ghost rows nobody saw — default-value rows inserted every five minutes because 38 of 39 materialized views were missing a WHERE clause. No metric catches this; the AI looked at the actual data. It reviews schemas before they reach production, catching wrong engines, missing partitioning, backwards ORDER BY, floating-point types for monetary values, redundant indexes — every one of which would surface months later the hard way. It maps 68 tables across 9 pipelines in 30 seconds for a new engineer who’d otherwise spend days reading code. It diagnoses that the alert system itself is the bug, not the database, and provides specific settings to fix the merge race condition. It catches a DROP TABLE stuck in an infinite retry loop, spamming errors every five seconds. No dashboard tracks that.</p><p>General-purpose AI SRE platforms sift through logs to find system-wide patterns. The good ones can point at a database as the source of a problem. But then what? When you’re dealing with <a href="https://bigdataboutique.com/services/postgresql-consulting">PostgreSQL</a>, <a href="https://bigdataboutique.com/services/opensearch-consulting">OpenSearch</a>, <a href="https://bigdataboutique.com/what-is-clickhouse">ClickHouse</a>, or other high-throughput database deployments at the heart of a system holding all its data, you need a proactive AI DBA that specializes in that technology. Not scripts — adaptive diagnostic strategies. Not generic advice — root cause analysis with evidence.</p><p>The new stack for the modern engineer is taking shape. The xEngineer owns the outcome. AI coding assistants — Cursor, Claude Code, Copilot — handle implementation. AI specialists handle the infrastructure. The human focuses on design, architecture, and business decisions. The industry is creating brilliant generalists while building ever-more-complex data platforms, and the way forward isn’t back to silos. The xEngineer model is right: engineers should own outcomes end-to-end. But someone needs to be the master. If it can’t be every engineer, it needs to be AI — deeply specialized, always available, proactively watching. “AI will not replace DBAs, but DBAs who use AI will replace other DBAs.” Companies that don’t rely on specialized AI DBA platforms for the data technologies forming their backbone will carry unnecessary risk. And they won’t be able to match — in speed or in cost — the companies that do.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=712cb9772237" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon OpenSearch Monitoring — Choosing the Right Tool]]></title>
            <link>https://medium.com/@synhershko/amazon-opensearch-monitoring-choosing-the-right-tool-aae559f6f605?source=rss-1289b2f838da------2</link>
            <guid isPermaLink="false">https://medium.com/p/aae559f6f605</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[opensearch]]></category>
            <dc:creator><![CDATA[Itamar Syn-Hershko]]></dc:creator>
            <pubDate>Tue, 15 Oct 2024 10:18:04 GMT</pubDate>
            <atom:updated>2025-01-16T20:31:07.828Z</atom:updated>
            <content:encoded><![CDATA[<h3>Amazon OpenSearch Monitoring — Choosing the Right Tool</h3><blockquote><em>In this blog post, we’ll review the available ways to monitor your Amazon OpenSearch Service cluster, aiming to offer a comprehensive guide to help you choose the perfect tool.</em></blockquote><h3>Introduction</h3><p>OpenSearch is one of the most popular software tools in the industry. It’s revered for its multifaceted functionality spanning Search, as well as Observability, Security Information and Event Management (SIEM), and — in recent OpenSearch versions — even as a Vector Database. As a result, OpenSearch has become a critical part of many companies’ software stacks.</p><p>This is why maintaining the stability and peak performance of OpenSearch clusters is paramount. Achieving this goal calls for robust monitoring solutions tailored specifically for OpenSearch.</p><p>In this blog post, we’ll review and dive into the monitoring tools available, aiming to provide a detailed roadmap for selecting the ideal tool for monitoring your OpenSearch clusters.</p><h3>What Makes a Good OpenSearch Monitoring Tool?</h3><p>Before diving into the evaluation of OpenSearch monitoring tools, it’s essential to delineate the key attributes that define an ideal monitoring solution for OpenSearch clusters:</p><ol><li><strong>Comprehensive Monitoring Scope</strong>: Your monitoring tool should provide insights into the OpenSearch process, the underlying operating system, and the Java Virtual Machine (JVM) hosting OpenSearch. This ensures a holistic understanding of the cluster’s health and performance.</li><li><strong>Feature-Rich Capabilities</strong>: The ideal monitoring tool should offer a wide array of features, including the collection of operating system metrics such as CPU and RAM usage, JVM metrics like heap usage and Garbage Collection (GC) total count, as well as OpenSearch cluster metrics such as query response times and index sizes. Additionally, the tool should let you create alerts, visualizations, and dashboards for comprehensive monitoring.</li><li><strong>Scalability and Cost-Effective</strong>: Scalability is crucial to accommodate the growth of OpenSearch clusters, while cost-effectiveness ensures that monitoring solutions remain viable for organizations of all sizes.</li></ol><p>Now, let’s explore some of the prominent monitoring tools available in the market:</p><h3>6 OpenSearch Monitoring Tools To Consider</h3><h3>1. CloudWatch</h3><p><a href="https://aws.amazon.com/cloudwatch/">CloudWatch</a> is a monitoring and observability service from Amazon Web Services (AWS) that offers seamless integration with OpenSearch. It lets users collect and track metrics, monitor log files, set alarms, and automatically react to changes in their OpenSearch clusters.</p><p>Amazon OpenSearch Service Domains or clusters gets a monitoring dashboard for key metrics. Those are powered by CloudWatch and can be extended as needed.</p><p>The built-in dashboards provide detailed insights into OpenSearch cluster health, performance, and stability. Additionally, users can create custom dashboards for specialized metrics and set up alerts to receive notifications of any potential issues.</p><p>Pros:</p><ul><li>Provided out of the box.</li><li>Seamless integration with other AWS services and specifically Amazon OpenSearch Service.</li><li>Some service-specific metrics are only available there (for example, EBS disk stats and cluster-wide service blocks).</li></ul><p>Cons:</p><ul><li>Built-in dashboards are insufficient, and you need to graph many additional metrics.</li><li>Relatively pricey for any metrics beyond what’s provided out-of-the-box.</li><li>Limited customization compared to some standalone monitoring tools.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AUiknpQtpKXJ94v-.png" /></figure><h3>2. Grafana and Prometheus</h3><p><a href="https://grafana.com/grafana/">Grafana</a> is an open-source tool for monitoring and visualizing metric data. It seamlessly integrates with various sources and is usually paired with <a href="https://prometheus.io/">Prometheus</a>, an open-source metrics collection and storage tool. This is a great combination for effectively monitoring and visualizing metric data.</p><p>The adaptability and customizable features of Grafana let you craft personalized dashboards and alerts drawing from diverse data sources. You can freely install Grafana as an open-source version for self-maintenance of your cluster health. Alternatively, Grafana Labs provides a hosted version, offering a basic free tier and paid plans catering to increased time series data and storage requirements.</p><p>While Grafana may offer a limited array of built-in integrations for alerting, it provides a plugin system that enables users to install plugins that facilitate support for popular alert system targets such as Slack, Teams, PagerDuty, and ServiceNow.</p><p>Grafana boasts powerful visualization features, but its integration with Prometheus can be a steep learning curve for some users.</p><p>Proficiency in Grafana requires domain expertise to maximize its capabilities and familiarity with integrated systems. For instance, using Prometheus with Grafana involves collecting and exporting metrics, while setting up alerts in Grafana requires knowledge of <a href="https://bigdataboutique.com/prometheus.io/docs/prometheus/latest/querying/basics/">PromQL</a> syntax, adding complexity to the learning curve.</p><p>Pros:</p><ul><li>Flexible dashboarding and visualization.</li><li>Extensive plugin ecosystem.</li><li>Open-source with both self-hosted and hosted options. Cons:</li><li>Steep learning curve, particularly when integrating with Prometheus.</li><li>Requires managing multiple tools for metrics collection and visualization.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W0UJI7yaGGcGuU8y.png" /></figure><h3>3. Datadog</h3><p><a href="https://docs.datadoghq.com/integrations/amazon_es">Datadog</a> is a robust enterprise-grade observability tool that offers comprehensive insights into OpenSearch metrics and logs. It supports various integrations for monitoring, visualization, dashboards, tags, and alerts. A notable feature is its templating support, which allows quick access to pre-configured templates for dashboards and reports, simplifying setup and customization. Moreover, Datadog was designed to handle large-scale deployments, making it suitable for enterprises with massive OpenSearch clusters.</p><p>The Amazon <a href="https://docs.datadoghq.com/integrations/amazon_es/">OpenSearch Service integration</a> requires ingesting CloudWatch metrics and logs into Datadog. This allows for a decent view of cluster metrics and issues. However, not all important metrics are ingested — for example, hot threads and shard balancing data aren’t available on Datadog.</p><p>Datadog’s main drawback is its high cost, making it one of the priciest monitoring solutions available. It also offers limited customization compared to some alternatives.</p><p>In addition, there is a steep learning curve for advanced features. While the interface is user-friendly, mastering some of the more advanced capabilities may require extra training. Despite this, it remains a strong choice for those needing comprehensive monitoring alongside other infrastructure and application monitoring.</p><p>Pros:</p><ul><li>User-friendly interface many users know and love.</li><li>Extensive out-of-the-box integrations.</li><li>Allows you to monitor your entire stack in one place, not just OpenSearch. Cons: High cost.</li><li>Limited customization compared to some alternatives.</li><li>Steep learning curve for advanced features.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VMCAabs8PxFN2ryC.png" /></figure><h3>4. New Relic</h3><p><a href="https://forum.newrelic.com/s/hubtopic/aAXPh0000001OgDOAU/opensearch-integration">New Relic</a> is a popular monitoring platform with powerful tools and dashboards. You can use it for tracking Amazon OpenSearch Service performance and health by following the <a href="https://docs.newrelic.com/docs/infrastructure/amazon-integrations/aws-integrations-list/aws-elasticsearch-monitoring-integration/">monitoring integration guide</a>.</p><p>New Relic provides real-time observability and detailed insights into your infrastructure, with customizable dashboards and alerts to streamline OpenSearch monitoring. One of its standout features is the comprehensive APM (Application Performance Monitoring) integration. It allows teams to view OpenSearch metrics alongside application performance data, giving a holistic view of system health. New Relic also supports distributed tracing, helping track queries and transactions across complex systems.</p><p>On the downside, New Relic’s pricing model is based on data volume, which can quickly become costly for organizations with large OpenSearch clusters or significant volumes of log and trace data. Additionally, while New Relic offers powerful tools, the depth of features may be overwhelming for smaller teams or those with simpler monitoring needs.</p><p>Pros:</p><ul><li>Real-time insights and customizable dashboards.</li><li>Comprehensive APM integration for a holistic view of system health.</li><li>Supports distributed tracing for deeper analysis.</li><li>Monitor your entire stack, including OpenSearch, in one place.</li></ul><p>Cons:</p><ul><li>Pricing can escalate quickly with larger data volumes.</li><li>Feature-rich platform may feel overwhelming for smaller teams or less complex environments.</li><li>Outdated integration that lacks maintenance, no focus on monitoring OpenSearch clusters.</li></ul><h3>5. Amazon OpenSearch Service Monitor</h3><p>AWS has provided a step-by-step guide to setting up a monitoring stack for Amazon OpenSearch Service domains and Amazon OpenSearch Serverless collections across all specified regions.</p><p>You can configure monitoring for Amazon OpenSearch Service domains and OpenSearch Serverless collections which will fetch the Cloudwatch Metrics and Cloudwatch logs from all domains/collections at a regular interval. This example also comes with pre-built OpenSearch dashboards and Alerts.</p><p>Available in <a href="https://github.com/aws-samples/amazon-opensearch-service-monitor">this repository</a>.</p><p>Pros:</p><p>Cons:</p><ul><li>Very basic monitoring.</li><li>Very simple and basic alerts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZOvQa9oDjiIsBTYk.png" /></figure><h3>6. Pulse</h3><p><a href="https://pulse.support/">Pulse</a> is an OpenSearch monitoring solution designed to simplify the management of OpenSearch clusters while efficiently addressing critical issues.</p><p>Developed by engineers at <a href="https://bigdataboutique.com/">BigData Boutique</a>, Pulse learns from your cluster’s configuration, unlike tools that use generic thresholds. This means tailored monitoring, dashboards, anomaly detection, and alerts designed to prevent issues before they cause downtime. Pulse focuses on actionable insights, minimizing alert fatigue so you only get notified when it matters.</p><p>With insights built from years of hands-on consulting experience, Pulse provides comprehensive visibility into all aspects of your OpenSearch deployment and cluster operations. Its intuitive dashboards showcase cluster, node, and index health, along with pertinent operating system metrics-all prioritized to highlight the most critical information.</p><p>Pulse features <a href="https://bigdataboutique.com/blog/pulse-query-analytics-for-elasticsearch-and-opensearch-529bd8">Query Analytics</a>, empowering users to fine-tune search performance and enhance user experience. For users on premium plans, Pulse includes expert support from experienced OpenSearch engineers, complete with industry-standard SLAs, ensuring rapid resolution of any issues. It also offer broad support, so whether you’re debating <a href="https://bigdataboutique.com/blog/opensearch-vs-elasticsearch-an-up-to-date-comparison-5c1c71">OpenSearch vs Elasticsearch</a> or managed solutions like <a href="https://bigdataboutique.com/blog/elastic-cloud-vs-aws-elasticsearch-opensearch-service-9051b1">Elastic Cloud vs AWS OpenSearch</a>, Pulse monitoring will still work for you.</p><p>Pros:</p><ul><li>Powerful and flexible dashboards for tracking cluster health.</li><li>Provides actionable insights and anomaly detection to prevent future issues.</li><li>Simple to set up and use.</li><li>Expert support as part of the product offering.</li></ul><p>Cons:</p><ul><li>No free or open-source option.</li><li>Limited to OpenSearch and relevant OS metrics.</li><li>Does not integrate with other monitoring tools.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WBkAi2J6lVkn2dwo.png" /></figure><h3>How to Choose the Best OpenSearch Monitoring Tool</h3><p>Excellent solutions are available for OpenSearch monitoring, ranging from free options to commercial licenses. Choosing the right one depends on personal preference and specific requirements.</p><p>Due to our extensive experience with OpenSearch and after using many different tools over the years, we developed and currently use Pulse for most use cases. We recommend Pulse for its comprehensive monitoring and alerting capabilities that provide actionable insights for OpenSearch cluster management.</p><p>Alternatively, CloudWatch offers seamless integration with OpenSearch for users already leveraging AWS services, providing a robust monitoring solution within the AWS ecosystem.</p><p>Grafana is an excellent free and open-source monitoring solution for those prioritizing cost. However, if budget is not a concern and you seek a comprehensive observability platform covering OpenSearch clusters, applications, logs, and metrics, consider Datadog.</p><h3>Wrap up</h3><p>We hope this guide helps you make the best decision regarding your OpenSearch monitoring needs, both today and in the future. If you happen to be looking for Elasticsearch monitoring, here’s a comprehensive guide to the available tooling: <a href="https://bigdataboutique.com/blog/elasticsearch-monitoring-selecting-the-ideal-tool-4c5d17">Elasticsearch Monitoring: Selecting the Ideal Tool</a></p><p>Supporting both Elasticsearch and OpenSearch, <a href="https://pulse.support/">Pulse</a> optimizes performance, reduces costs, and aligns with your requirements, helping your team save time managing clusters and focus on innovation instead.</p><p><em>Originally published at </em><a href="https://bigdataboutique.com/blog/amazon-opensearch-monitoring-choosing-the-right-tool-7c973b"><em>https://bigdataboutique.com</em></a><em> on October 15, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aae559f6f605" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Lake by Databricks — an Introduction]]></title>
            <link>https://medium.com/@synhershko/delta-lake-by-databricks-an-introduction-2cb8df8e4235?source=rss-1289b2f838da------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cb8df8e4235</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[delta-lake]]></category>
            <dc:creator><![CDATA[Itamar Syn-Hershko]]></dc:creator>
            <pubDate>Wed, 03 May 2023 14:37:57 GMT</pubDate>
            <atom:updated>2023-07-04T07:14:54.732Z</atom:updated>
            <content:encoded><![CDATA[<h3>Delta Lake by Databricks — an Introduction</h3><blockquote><em>Delta Lake (originally developed by Databricks) has become a household name when it comes to modern cloud table formats. But is it the right one for you? Learn more in this detailed review.</em></blockquote><p>In the latest edition of our modern data platform post series, we are going to put the Delta Lake project under the microscope, looking first at its history, before delving into its architecture and features. Delta Lake solves similar challenges to the other cloud table formats we’re exploring in this series ( <a href="https://blog.bigdataboutique.com/2023/03/introduction-to-apache-hudi-c83367">Apache Hudi</a> and Iceberg). In many ways it’s an evolution of Apache Hudi, but has a lot of additional features and is making fast advancements in this space. It’s absolutely a technology to consider , whether you are looking to upgrade an older <a href="https://blog.bigdataboutique.com/2022/09/hive-tables-and-whats-next-for-modern-data-platforms-1xts1m">Hive-based solution</a>, or create a greenfield Data Lakehouse solution.</p><h3>What is Delta Lake?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcejKQPpAA5Rj1Si1qKXJA.png" /><figcaption>Delta Lake</figcaption></figure><p><a href="https://delta.io/">Delta Lake</a> was launched in early 2019 by Databricks, an Apache Spark company, as a cloud table format built on open standards and partially open-source.</p><p>The idea behind Delta Lake was to support the very frequently requested features in the modern data platforms ecosystem, or should we say Big Data — data mutability, point in time view of data (aka <a href="https://delta.io/blog/2023-02-01-delta-lake-time-travel/">Time Travel</a>), ACID guarantees and concurrent writes, and more. We discussed those features and why they are so requested and were hard to get right in <a href="https://blog.bigdataboutique.com/2022/08/architectures-of-a-modern-data-platform-aaw0xs">our previous posts in the series</a>.</p><p>At the time of its launch, Delta Lake was released with some features behind a “paywall” and you had to be a Databricks customer to use them in their <a href="https://bigdataboutique.com/services/databricks-consulting">Databricks Platform</a>. It was only in 2022 that Databricks agreed to fully open source the entire product and contribute future enhancements to the Linux Foundation.</p><p>Even as an open source platform, Databricks are investing and aligning Delta Lake features so it aligns with their other products and cloud (and understandably so). Additionally, because it’s maintained by a large and well-known commercial entity, Delta Lake is likely to have more development effort committed than other data platforms technologies (esp. that it is now an integral part of Delta Live Tables that is yet another hot product from Databricks). However, Delta Lake isn’t alone as a commercial open source product in the data lake space — Apache Iceberg and Apache Hudi have been doing this for some time already, too.</p><p>Delta Lake is mostly a Java and Apache Spark effort, but it is indeed developed through an open standard so there are, and will be, integrations from other languages and tools, for example the <a href="https://github.com/delta-io/delta-rs">Delta Lake Rust interface</a>.</p><h3>Delta Lake Architecture</h3><p>Like Parquet, ORC, CSV and JSON, Delta Lake is a file-based format or to be rather more exact — a table format. The main idea has always been to make listing files in cloud blob storages faster (and cheaper at the same time).</p><p>When building Delta Lake, Databricks took the same concepts of Hive table format that we’ve seen before such as partitioning through paths, and implemented a lot of improvements and fixes based on lessons learned from Apache Hudi and Apache Iceberg (nota bene, many if not most changes to the latest version of the Connector API in Apache Spark 3 have been contributed by the former Netflix engineer, Ryan Blue who created a commercial entity behind Apache Iceberg, Tabular).</p><p>Delta Lake absolutely satisfies many items on our <a href="https://blog.bigdataboutique.com/2022/09/hive-tables-and-whats-next-for-modern-data-platforms-1xts1m#our-wishlist-for-a-modern-data-platforms">wishlist for modern data architectures</a>. These are things like ACID transactions, mutable data, a point-in-time view (“time travel”), integration with many tools and technologies, and more.</p><p>Below are some highlights and architectural nuances unique to Delta Lake.</p><h3>Hive-like Table Format</h3><p>Similar to <a href="https://blog.bigdataboutique.com/2023/03/introduction-to-apache-hudi-c83367">Hudi’s Hive format</a>, Delta Lake also creates a transaction log. This transaction log maintains a record of all operations requested on data, and the table metadata, as well as pointers to the data files themselves (Parquet) and their statistics (in JSON-encoded metadata files). Those statistics are important and deliver huge benefits for query performance. It also maintains a record of table versions which facilitates Delta Lake’s point-in-time view, called <a href="https://delta.io/blog/2023-02-01-delta-lake-time-travel/">Delta Time Travel</a>.</p><p>The statistics recorded by the transaction log are the maximum and minimum values for each column found in the actual data files, null counts, etc that allow for metadata-only queries like counts or data skipping (skipping data files at load time if they do not meet the requirements of a query). This means that complex queries can be processed in much less time and cheaper (we’re in a cloud environment after all where every operation counts). This is known as data skipping.</p><p>Most importantly, accessing the transaction log (“manifest files”) using Apache Spark, PrestoDB, Trino and many other supported query engines is very straightforward. Querying and writing to a Delta Lake powered data platform is very easy today and does not require any specific knowledge of the inner workings of Delta Lake. The query engines will do all that for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z_5AhftbOAMXgDlY.png" /><figcaption>Delta Lake File Format</figcaption></figure><p>This table format allows to support updates and deletes, therefore CDC (Change Data Capture) is fully supported to sync OLTP databases into the data lake; and also shallow clones, which helps with creating derivative tables easily — cloning PBs of data easily and cheaply just like a “git branch” operation.</p><h3>Medallion Architecture (Bronze, Silver, and Gold Tables)</h3><p>Databricks, the company behind Delta Lake, promotes a data maintenance strategy often referred to as Medallion Architecture (Bronze-Silver-Gold). Those are conceptual, logical tiers of data which helps categorize data maturity and availability to querying and processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rmUg0Jv4N7rS5h-x.png" /><figcaption>Delta Lake’s Medalion Architecture</figcaption></figure><p>Standard query engines and processing techniques can be used on any of the tiers, and also for moving data between the tiers. The main idea behind this approach is to differentiate between different data qualities, and to create higher tiers of data by cleaning, enriching and aggregating data from lower tiers.</p><p>The Bronze Tier consists of tables that hold raw data, immediately-post ingest. This data will most likely be from various sources, but usually it will be unaltered from its source. Data can be ingested into this tier via either streaming or batch processes.</p><p>Data held in the Silver tier has been refined, either via ETL processes or data preparation. Typically Delta Lake tables in this tier will have been enriched and possibly joined with other tables and sources to create a more customized and insightful view of the data.</p><p>Gold tier data is full of meaningful and actionable insights, mainly because the Delta Lake tables on that tier are fully prepared. These tables act as your feature store and represent actionable data, and are directly queryable by analysts and even end users.</p><p>Data in any of the tiers is queryable, but Silver and Gold tiers is data that is really ready for consumption.</p><h3>Copy on Write</h3><p>Delta Lake only offers Copy on Write (CoW) with no support for Merge on Read, (although this capability could be available in future releases). CoW with Delta Lake sees data stored in a Parquet format; every time the data is updated, a new version of the updated data is created (leading to even more parquet part data files under the table’s data directory). General opinion and common sense dictate that CoW is far better for query-heavy use-cases.</p><h3>Delta Lake Schema Architecture</h3><p>Delta Lake was designed to mitigate data loss and optimize performance. As such, there are several elements of schema architecture that are unique to it as a platform. Here are the main cornerstones you need to be familiar with.</p><h3>Schema Evolution</h3><p>Schema Evolution on Delta Lake allows users to change a table’s existing schema to accommodate data that has changed over time or is added to the table. This is important for tables in the Gold tier as it’s most commonly needed for append and overwrite operations. This allows for the creation of additional columns that were not previously described in the schema.</p><p>It’s also possible to have an implicit schema definition with Delta Lake by using the mergeSchema option in a query. This will transfer all of the columns to the target table on write, and can also add nested fields. Caution must be exercised while using MERGESCHEMA because you could end up with duplicate data and unnecessary columns without proper planning.</p><h3>Schema Enforcement</h3><p>The Delta Lake architecture contains a built-in failsafe to guarantee data quality, which is called Schema Enforcement. This prevents writes to a table that do not match its schema and can be altered to enforce against different conditions such as differences in columns or differences in data types. If data is outside of the pre-existing schema, then it is simply recorded as a null value.</p><p>Schema Enforcement is useful for data analytics, machine learning, and business intelligence as it gives a uniformity of data to prevent existing algorithms or dashboards from erroring.</p><h3>Delta Lake Features</h3><p>As with the architectural section, Delta Lake shares common features with other data lake products of equal repute. However, Delta Lake also delivers some stand-out features which differentiate it from the crowd. We’ll explore these below.</p><h3>Delta Live Tables</h3><p>Delta Live Tables is a framework which allows to declare transformations on datasets and specify how records are processed through query logic. It is used for building reliable, maintainable, and testable data processing pipelines. As stated by the documentation — “Instead of defining your data pipelines using a series of separate Apache Spark tasks, you define streaming tables and materialized views that the system should create and keep up to date. Delta Live Tables manages how your data is transformed based on queries you define for each processing step. You can also enforce data quality with Delta Live Tables expectations, which allow you to define expected data quality and specify how to handle records that fail those expectations.”</p><p>You can draft a data quality constraint using either Python or SQL. The constraint consists of a descriptor, an invariant, and an action to take for any record that fails the invariant. It is based on the data expectations feature in Delta Lake, see the <a href="https://books.japila.pl/delta-lake-internals/check-constraints/">CHECK constraint</a> for example. Actions that can be taken on records that don’t meet the invariant are drop (prevention of procession of a given record), fail (stops any process as soon as a record fails to meet an invariant, with table updates being rolled back), and retain (keep the records which fail to meet the invariant).</p><p>Delta Live Tables also respects any preexisting dependencies which may exist between tables to prevent you from having to rewrite any ongoing queries. This is a great feature for developing data pipelines and ensuring that as your data progresses through Bronze, Silver, and Gold tiers (see above) it improves in quality. It also allows for automated testing and can be run in either batch or streaming mode.</p><h3>Generated Columns</h3><p>Whilst generated columns are nothing new for databases, having them in a Data Lake solution is rare. They are columns whose values are generated by Delta Lake based on functions defined by end-users. Typically you would use the Generated Columns feature in your Gold tier of data, as it serves to provide precise insights from a range of sources and then writes that insight to a table.</p><p>Identity Columns, a type of Generated Columns that assign a unique value for each row, are also available and supported in Spark 3.4, that opened up the feature to other cloud table formats.</p><p>You can create Generated Columns on Delta Lake using either SQL or the DeltaTableBuilder API.</p><h3>Cloud Agnostic</h3><p>Delta Lake is just an open-standard, and then an implementation of it, and as such it isn’t constrained to a cloud platform. Delta Lake is available on Databricks’s platform on all major cloud platforms, but it’s also possible to leverage Delta Lake without requiring a Databricks installation, or on any niche cloud or private data center.</p><h3>Data Skipping and Compaction</h3><p>As mentioned earlier, Delta Lake’s data skipping is made possible by the use of a transaction log. This narrows down the searching time of a given query, as all of the records are described in the log and can be quickly narrowed down based on their suitability for the query. If that feature wasn’t enough, Delta Lake further enhances performance via compaction.</p><p>Compaction is triggered by running a simple OPTIMIZE command on Delta Lake. This feature condenses numerous small Parquet files into a single large Parquet file, as it’s faster to query one large file than many small files.</p><p>What’s more, because of Delta Lake’s ACID compliance, the OPTIMIZE command can be run and executed at the same time that users are querying the table. To top it all off, this function can be run automatically to automatically regulate Parquet size and continuously optimize query performance.</p><p>Data Skipping is a major feature of any modern data lake and a lot of effort is put in order to innovate and find more ways to do this efficiently, for various use-cases. For example, in Delta Lake <a href="https://docs.databricks.com/delta/data-skipping.html">z-order indexes</a> are used for data skipping as well.</p><h3>Delta Sharing</h3><p>As a general rule, you can’t access Delta Lake tables outside of Databricks runtime because ingestion depends on the Delta ACID API. However, Databricks have introduced Delta Sharing, which might change all that. The goal of the Delta Sharing feature is to allow the sharing of live data without having to make a copy, across a variety of clients whilst maintaining total security.</p><p>At its core, Delta Sharing is a REST protocol that allows limited access to part of a cloud-hosted database. It uses traditional cloud-based storage, like AWS S3 or Google Cloud Storage, to facilitate the sharing or transfer of datasets. Security is governed by token-based authentication, although other verification methods are under development (e.g. Databricks Unity Catalog).</p><p>One of the most interesting aspects of Delta Sharing is that you as the data provider can choose the version or how much of the dataset you wish to share. This sharing capability is significantly more cost-effective than traditional data transfers, and also allows for global collaboration on big data between organizations. Use-cases in financial services, vaccine development, and space exploration are already being seen.</p><h3>Conclusion</h3><p>There’s no arguing that Delta Lake is a fantastic tool with some incredible features. Unfortunately, there are some aspects like the lack of support for Merge on Read that let it down. But overall, its unique features are its greatest differentiator. Delta Sharing and Data Skipping do not exist with the same scalability elsewhere on the market. These combined with Databricks’ enterprise support organization makes Delta Lake a great option for large companies.</p><p>While Delta Lake is indeed open-source, its most common usage is through the Databricks cloud product. Regardless, it is very easy to start with and the fast development pace is very promising.</p><p>Stay tuned for our next installment in this series — reviewing Apache Iceberg, and then comparison of Apache Hudi, Apache Iceberg, and Delta Lake.</p><p><em>Originally published at </em><a href="https://bigdataboutique.com/blog/delta-lake-by-databricks-an-introduction-f78147"><em>https://bigdataboutique.com</em></a><em> on May 3, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cb8df8e4235" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Apache Hudi]]></title>
            <link>https://medium.com/@synhershko/introduction-to-apache-hudi-27422fda1c1b?source=rss-1289b2f838da------2</link>
            <guid isPermaLink="false">https://medium.com/p/27422fda1c1b</guid>
            <category><![CDATA[apache-hudi]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Itamar Syn-Hershko]]></dc:creator>
            <pubDate>Fri, 17 Mar 2023 09:31:41 GMT</pubDate>
            <atom:updated>2023-07-03T05:55:43.427Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Apache Hudi introduced easy updates and deletes to S3-based Data Lake architectures, and native CDC ingestion patterns. Here is all you need to know about it.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Xwb299oGZYCjPZTidjUuA.png" /><figcaption>Introduction to Apache Hudi</figcaption></figure><p><a href="https://hudi.apache.org/">Apache Hudi</a> is a data Lake technology that has been in use since 2016. Originally built by Uber, Hudi is the first of the three data platforms that we’re going to examine in detail. The series has in-depth reviews of Delta Lake and Apache Iceberg, and will end with a comparison between those three popular technologies.</p><p>This is the third post in our series, <a href="https://blog.bigdataboutique.com/series/architectures%20of%20a%20modern%20data%20platform">Architectures Of A Modern Data Platform</a></p><p>In this post, we are going to provide an overview of Apache Hudi, before diving in and examining its key strengths and how they match up against the requirements we set out in our previous article.</p><h3>A short history of Apache Hudi</h3><p>Hudi was originally developed at Uber in the early 2010s and went into production in 2016. Uber had a requirement for a cloud-based data lake to power and manage the 100PB of data that underpin key business functions for trips, riders, and customers. Uber’s data engineering team had a specific need to feed changes from relational SQL databases via either change data capture (CDC) or binlogs into a data lake.</p><p>Uber submitted Hudi to the Apache software foundation in 2019, releasing it as Open-Source software and forming a community around it. Since Uber created Hudi, other large data-rich organizations like Walmart and Disney have adopted it as their primary data platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*xXPOgjhMOlB6aZJ8.png" /></figure><p>As covered in <a href="https://!%5Bfile%5D(https://bdbq-assets.storage.googleapis.com/bdbq-blog/c83367_5bdf27c0-c4a6-11ed-a277-d5f7a2219b1f.png).bigdataboutique.com/2022/08/architectures-of-a-modern-data-platform-aaw0xs">our introductory post</a>, we aren’t trying to differentiate between data lakes or data warehouses. When implemented properly, their architectures are likely to be similar on multiple levels. Hence the term ‘data platforms’. With these modern architectures come a variety of fairly standard requirements, and we are going to evaluate how Hudi measures up.</p><h3>The Challenges Hudi Solves</h3><p>Hudi originated from the requirement to feed changes from traditional, relational SQL databases into a data lake platform for long retention and to support analytical queries at scale.</p><p>Uber faced challenges with their ever-growing data platform requirements, such as incremental data processing, data versioning, and efficient data ingestion.</p><p>At the core of the Hudi project is the capability to connect any relational database to a data lake storage layer such as S3 and HDFS via CDC or binlog, which are the standard ways of capturing data changes in various SQL databases. Since those change notification streams do not contain only append-events, but they can signal a row update or delete as well, this is not a trivial task.</p><p>During the years Hudi was conceived and developed, there were no other easy solutions to support data updates on S3 — users had to innovate ways to do it, and it was rather a difficult and rigorous task to do, and many just decided to avoid it altogether. That is why Hudi is so important, it paved the way for modern data lakes as we know them today.</p><p>Nowadays, any commonly used query engine supports querying Hudi tables — Trino, PrestoDB, Spark and more.</p><h3>Common Apache Hudi Use Cases</h3><p>Given Uber’s initial in-house development of Hudi, it addresses several use cases, some are niche but some are very popular, where there are gaps in traditional data warehouse and data lake platforms.</p><h3>Real-time Data Ingestion</h3><p>Hudi doesn’t quite provide real-time data ingestion, but it’s closer than other data lake platforms. Traditionally when ingesting data from OLTP sources, users require an array of tools to handle the sources in a timely fashion. Some Hudi functions (like Upsert for RDBMS) provide resource-efficient alternatives to bulk-upload ingests. Ultimately, it’s Hudi’s DeltaStreamer tooling that allows easy scale-out to include more and more sources.</p><h3>Incremental Processing</h3><p>Incremental Processing is an important use case as traditional data architectures often ignore data that arrives late. As the complexity of ETL pipelines increases, the more likely it is for processing and event times to deviate. Hudi deals with this issue by providing a dedicated table for consuming new data. A separate table is also used to reconcile any granular changes for individual reads. Thus, processing schedules can be significantly reduced to much shorter increments.</p><h3>New Batch and Streaming Architecture</h3><p>Hudi introduces data streaming principles to data lake storage, which allows data to be ingested significantly faster than traditional architectures. It also allows for the development of incremental processing pipelines (see above) which are hugely faster than traditional batch processing jobs. Additionally, because Hudi doesn’t require server resources up front, it provides greater analytics performance with less operational cost.</p><h3>Schema Evolution</h3><p>As we covered in our introductory article, the ability to ad-hoc alter the data schema is key. Hudi leverages Avro to dynamically alter table schemas, using schema on write. This is important for consistency as it ensures data pipelines don’t fall over as a result of failed backward compatibility from schema changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pK8vuPU90Jb3odmz.png" /><figcaption>Overview of Apache Hudi</figcaption></figure><p>As mentioned in <a href="https://blog.bigdataboutique.com/2022/08/architectures-of-a-modern-data-platform-aaw0xs">our introductory piece</a>, we believe that a modern data platform should provide robust support for updates and deletes even when data resides on a cheaper storage layer like S3 or HDFS. Luckily, Apache Hudi has both of the above.</p><h3>Deletes</h3><p>Hudi allows two types of deletes to be used in its tables for various compliance and privacy requirements.</p><p>The first is ‘soft delete’, which retains a record but sets the value of the data to null. Provided the fields you are seeking to delete can be set to null, as documented in the schema, this is a good option for standard deletes not related to privacy or compliance. Soft deletes can be rolled back using the record that is maintained.</p><p>‘Hard deletes’, on the other hand, are permanent deletes to erase any trace of the field from the table. There are three ways of doing this, using either DataSource and changing OPERATION_OPT_KEY; using DataSource and changing PAYLOAD_CLASS_OPT_KEY; or by using Delta Streamer and adding a new column indicating which records are to be deleted.</p><p>Hudi comes with two options when it comes to updating, which shouldn’t be used interchangeably as they behave very differently.</p><p>‘Insert’ is a fast operation that can be used to update or insert new values to records that are tagged. It is exceptionally fast and is best used in circumstances where you can tolerate multiple duplicates in a table. There is also a BULK_INSERT operation, which is a scalable version of insert capable of handling hundreds of terabytes of load.</p><p>‘Upsert’ is the alternative to insert, as is configured as the default in Hudi. It is largely the same as the insert operation, but it performs an index lookup before any updates. This is slower than insert but is better for storage optimization and rightsizing of files. This operation will not generate any duplicates.</p><h3>Hudi’s Table Format</h3><p>In order to support those use-cases and features, Hudi uses a similar partitioned directory structure to that of a traditional <a href="https://blog.bigdataboutique.com/2022/09/hive-tables-and-whats-next-for-modern-data-platforms-1xts1m">Hive table format</a> but introduces the concept of a commit log, or a “timeline”, which allows it to do a lot more than what Apache Hive ever supported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JW6F_6D98qlpRYnq.png" /><figcaption>Apache Hudi Table Format</figcaption></figure><p>This table format and how data is being written to it are rigorously explained in the Hudi docs: <a href="https://hudi.apache.org/docs/file_layouts/">file layout</a> , <a href="https://hudi.apache.org/docs/write_operations/#writing-path">writing path</a>. What’s more important for us here is how this data is queried, and the different query types offered by Hudi.</p><h3>Apache Hudi Query Types</h3><p>Copy on write and merge on read are two different approaches to handling updates in data storage systems. With Hudi, you can choose whether to use a Copy on Write (CoW) or Merge on Read (MoR) on a table basis (“Dataset”), to optimize either reads or writes for any given table or use-case. Let’s see how that’s done.</p><h3>Copy on Write</h3><p>A Copy on Write is a strategy where data is not modified directly in place, but instead a copy of the data is made when a modification is requested. This ensures that the original data remains intact and can be used by other processes or users, but also means updates and deletes are applied during write. The updated data is written to the new copy of the data, and the old copy is discarded.</p><p>CoW should be used when there is a read-intensive workload in use, as well as data with a lower rate of change. Hudi defaults to CoW unless it’s manually changed to MoR. Every time that an update or delete is requested, a new version of the updated files is created in a write.</p><h3>Merge on Read</h3><p>Merge on read, on the other hand, is a strategy where data is updated on read time and not immediately reflected in the data on storage.. Instead, the updates are stored separately and are merged with the original data when it is read.</p><p>MoR is best suited to data with a high change rate, or write-heavy data. Unlike CoW, MoR uses a mix of Parquet and Avro data formats. Any data updates are logged at a row-level to allow for column updates. Hudi also creates two tables in Hive — one is optimized for reading and the other is a real-time view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wNWvM-k_54v9zchh.png" /><figcaption>Merge on Read vs Copy on Write</figcaption></figure><p>Hudi supports different query types to balance freshness of data and query performance. Merge on read will demonstrate higher write throughput but queries will have to work harder thus be slower. Copy on write will exhibit slower writes, but queries will not have to waste time doing extra work. Ingest speeds also matter for some use-cases, so assuming control over that is a great feature by Hudi.</p><h3>Apache Hudi: A Stream-based approach</h3><p>Hudi differentiates itself from other modern data platforms with its own ingestion tool called DeltaStreamer. This enables Hudi to ingest data from a variety of sources, whilst giving them shared capabilities. A unique capability of Hudi is that it allows you to manage data at an individual record level in object storage, whilst DeltaStreamer handles the ingestion. This makes CDC (Change Data Capture) and data streaming much more straightforward.</p><h3>DeltaStreamer</h3><p>DeltaStreamer sits separately from Hudi as a standalone function. It is capable of ingesting data from DFS, database change logs, and Kafka. It also has both one-off and continuous ingestion modes.</p><p>Run-once is an incremental pull of events and ingesting them into Hudi, and for MoR tables compaction is automatically enabled. Continuous mode creates an infinite loop of ingestion, the frequency of which can be configured. If you enable continuous mode, you must allocate appropriate resources as compaction and ingestion will be running concurrently.</p><h3>Object Store Access</h3><p>Because Hudi allows you to manage all of your data at a record level within object storage, it gives you a vast range of query tools and storage options. Hudi will read and write datasets on Azure Blob, Google Cloud Storage, or AWS S3. It also has specific inbuilt capabilities such as consistency checking on S3. It’s recommended to use Hive, Spark, Presto, or Flink to query Hudi data.</p><p>There’s no doubt that Flink has changed data processing and data architectures. It is both a processing engine and a framework for performing computations at in-memory speed over bounded and unbounded data streams. Bounded data streams have a pre-defined start point and end, whilst unbounded do not have a defined endpoint.</p><p>Where Hudi is concerned, Flink provides a stream processing engine that returns queries in close to real time. This is known as continuous queries.</p><p>The introduction of Flink continuous queries to DeltaStreamer has built up a strong and unique capability. It allows users to perform stream analysis whilst data is ingested, and relies upon Hudi’s native CDC support in object storage. Any message changes in the data stream are retained and then consumed, which Fink then analyzes in real-time, which massively cuts down on latency in an ETL pipeline.</p><h3>Summary</h3><p>There’s no doubt that Hudi is a powerful tool that has set the scene for modern data lake and warehouse platforms. In the next posts in the series, we are going to look at other technologies which took inspiration from Hudi no doubt, and then evolved further.</p><p><em>Originally published at </em><a href="https://bigdataboutique.com/blog/introduction-to-apache-hudi-c83367"><em>https://bigdataboutique.com</em></a><em> on March 17, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27422fda1c1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hive Tables and What’s Next for Modern Data Platforms]]></title>
            <link>https://medium.com/@synhershko/hive-tables-and-whats-next-for-modern-data-platforms-84526872e2f?source=rss-1289b2f838da------2</link>
            <guid isPermaLink="false">https://medium.com/p/84526872e2f</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-hive]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Itamar Syn-Hershko]]></dc:creator>
            <pubDate>Thu, 15 Sep 2022 12:03:35 GMT</pubDate>
            <atom:updated>2023-07-01T18:28:22.408Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Apache Hive defined the standard of persisting data on object storage technologies to enable querying. We are now starting to hit the limits of Hive Tables and it’s probably time to discuss what’s next. What are the features we’d want to see in modern Data Lake architectures?</em></blockquote><p>In our <a href="https://blog.bigdataboutique.com/2022/08/architectures-of-a-modern-data-platform-aaw0xs">introductory post</a> we discussed the typical structure and usual components of a modern data platform.</p><p>A very common component of any Data Lake and Data Warehouse implementation is what we often call the “Cold Storage” tier. This is where, or rather how, the vast majority of data is persisted in a cheaper storage solution, ideally in a way that still allows it to be queried. Putting data in “cold storage” assists in reducing the total costs of the data platform and trading off query speed with significantly lower costs.</p><p>In this post we are going to focus on that layer of cold storage, understand how it usually works, and try to map the gaps that exist today and offer our wishlist for more modern implementations which will serve the future versions of data platforms.</p><p>The best place to start is with the convention most large data platforms in use today are using, often referred to as “Hive Tables”, or the Hive Table Format.</p><h3>The Hive Table Format</h3><p><a href="https://hive.apache.org/">Apache Hive</a> is a SQL engine to query Big Data. It was originally built to translate SQL statements into Hadoop MapReduce jobs, and continued to evolve along with the original Hadoop project, eventually graduating to exist on its own. While Hive as a query engine is mostly dead today, having been replaced by several other more advanced technologies, the key concepts it brought with it are very much still with us today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ik-bidCi-Q8E-JcrsFvJmg.png" /><figcaption>Apache Hive powering a not-so-modern data warehouse</figcaption></figure><p>To understand the problem Hive needed to solve, we have to remember it is being used in systems where data is persisted on Object Storage technologies such as HDFS, S3, Azure Blob Storage and others. In those technologies, there aren’t any indices or any other clues that can help indicate which file (or blob) contains records of interest. Data just resides in many large binary files on storage.</p><p>The key idea behind Hive tables is to organize data in a directory structure, so we can read only relevant files based on some partition key. The folder name would be the partition key, and using some table metadata stored in the Hive Metastore a querying engine could only access the “interesting” partitions effectively “pruning” the other partitions from the query execution plan. That way we essentially create sort of an index on the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/0*X61yaP2MUE3sx1iB.jpg" /></figure><p>In addition to partitioning, the files in each partition can be broken down further into “buckets”, thus allowing to create some inner sorting of data within a partition, thus potentially further reducing the amount of data files that need to be read and scanned during a query execution.</p><h3>File Formats</h3><p>To make querying Big Data efficient it is important to use the right file format on storage. While the naive approach of storing many JSON or CSV files on object storage might sound reasonable and is in fact whats done by most unaware developers, it is the worst possible format to use. A better format would allow to more easily scan the data on queries — and that is what columnar file formats like Parquet and ORC do.</p><p>A columnar file format stores all values of a certain column in the same chunks of data on disk. That means reading a column is very very cheap, and this is important because filtering and aggregations are done on columns (for instance using the SQL WHERE operator). This also greatly improves compression, especially for columns with low cardinality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WMaCRhxVRWPPY0UP.png" /></figure><p>Additional features of the file format like sorting, saving stats on headers, bloom filters even — all help with speeding up scans and queries, since irrelevant files can be skipped over much faster, and sometimes entire query results can be provided from the headers alone.</p><h3>Hive Tables Everywhere</h3><p>To summarize, Hive Tables allow effective querying on Big Data by applying a few simple but powerful concepts that make querying data at rest very efficient:</p><ol><li>Partitioning and Bucketing which enable filtering the files being retrieved by the query engine.</li><li>Using columnar file formats, such as ORC and Parquet, to greatly reduce the time needed to scan the files for filtering and aggregating data. Files are both smaller, and better arranged for what query engines need.</li><li>Lastly, optimizing for the right files sizes so the query engine can pull less files that are larger in size. This helps balance parallelism with the actual amount of resources needed to execute a query.</li></ol><p>Hive Tables proved to be so effective in querying even Petabytes of data on cold storage, that it became the convention in many systems. As Hive aged, query engines such as Spark, Presto and Trino, were built and eventually replaced Hive as the actual engines being used to query this data..</p><h3>Where Hive Tables Fail</h3><p>The very design that makes Hive efficient also makes it fall short in quite a few areas, and those are very well known and documented.</p><p>Since data is persisted in compressed files on a cheaper storage tier that doesn’t allow random seeks, updates are deletes are not possible to do, at least not easily. This also affects freshness — if you want to keep files optimized for querying, you have to batch writes thus new data may take some time until it is visible to queries.</p><p>The external metastore instance (the Hive Metastore) is required as a database of file pointers. It was originally built to assist with keeping track of large file repositories and to help with housekeeping work, so it is an important addition. However, besides the fact it requires maintenance on its own, it sometimes goes out of sync and requires to actively perform directory listing and refreshes that can become costly processes to perform.</p><p>Synchronizing between the file-system and the metastore can’t be done atomically, too. This means it is not possible to synchronize writes, and also another reason why updates and deletes are not natively supported.</p><p>Lastly, the partitioning scheme has shortcomings as well.</p><p>The layout dictated by Hive supports only one hierarchy of partitions. A very common way to partition data with Hive is based on dates and time of day. If you wanted to add another partition such as tenant ID for example you’d have to either use dates first then tenant ID, or tenant ID then dates. Due to the folder-like partitioning scheme, it is not possible to create partitions that are not in a hierarchy without duplicating the data.</p><p>Since partitions are physically on storage, meaning files are actually persisted using a file path, changing a partition scheme is very costly. Repartitioning will involve copying all data and distributing it again on disk. More often than not, partitioning is hard to get right and will usually require changes down the road, even if subtle, and this is one of the biggest pains with Hive today for mature systems.</p><h3>Our Wishlist for a Modern Data Platforms</h3><p>Putting the Hive table format aside, lets ask ourselves how the ideal Data at Rest solution would look like?</p><p>We collected a list of features and must-haves we believe are crucial for any modern Data Lake and Data Warehouse solution. Since Data Lakes are more frequently dealing with changing data and schemas we</p><p>The ability to UPDATE and DELETE records within a fileset is critically important. This functionality is often mandated by compliance requirements, such as Europe’s <a href="https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/">GDPR</a> or California’s <a href="https://oag.ca.gov/privacy/ccpa">CCPA</a>, where customers or users may request their data be removed from a system.</p><p>Another very common scenario where updating and deleting records is very handy and a much desired feature is to to fix individual records that have wrong or defect data due to some system bug or missing cleaning procedures.</p><p>When using columnar file formats this is going to prove a huge challenge for multiple reasons:</p><ul><li>It’s impossible to just delete or update a single record easily, and a whole file rewrite is going to be required. This is going to prove very hard and costly to do at scale, and will require a lot of time and computational power.</li><li>For those reasons, supporting UPDATE and DELETE operations concurrently is also close to impossible. Due to the nature of the underlying storage and technologies involved, running more than one update process will most likely lead to data corruption or at least inconsistent results; and facilitating locking is very hard to get right.</li><li>No isolation support on the underlying storage systems (object storage) and also file formats, means updates on data while it’s being queried will either fail queries or render unusable results. At the very least that means you can’t run such update procedures as a normal operation.</li></ul><h3>Support for late arrivals</h3><p>Pretty much every data platform has data records arriving not in order and sometimes days and months in delay. Late arriving data needs to be added to the right partition on arrival, while still keeping the files and partitions in an optimal state for efficient querying. A modern data platform should overcome this.</p><h3>Record Deduplication</h3><p>Often in data architectures, deduplication of data is carried out at the data streaming level as data is being ingested. It can also be written into ETL jobs. However, assuming late arrivals are to be expected in the platform, and efficient DELETE and UPDATE operations exist, depending on the amount of compute resources available, it could prove to be much more efficient for deduplication to be carried out on the storage layer.</p><h3>More Flexible Data Partitioning</h3><p>Partition Pruning is the process of completely ignoring full partitions of data during querying, so only data suspected to be relevant to the query is scanned. This is a crucial feature for efficient queries on Data Warehouse and Data Lake platforms.</p><p>Oftentimes the partitioning scheme is defined early in application life, much before the real challenges and even real data are encountered. Once the application is deployed and used, the weaknesses in the selected partitioning scheme start to surface, and in an ideal data platform we’d like them to be much easier to change. Ideally, changing a partitioning scheme should just be a matter of executing a simple ALTER command without significant side effects.</p><h3>Non-hierarchical partitioning schemes</h3><p>Today, partitions in most systems are usually hierarchical (year, month, day; sometimes tenant ID, year, month, day; and so on). But in truth, the data model for most systems is much more sophisticated than that, and there could be many other useful partitions that could be used if it was possible to do so efficiently.</p><p>As long as the partitioning scheme is relying on the Hive Table Format’s approach, meaning doing it in a folder-like structure, adding another partition means either adding another partition level when the data is orthogonal, resulting in smaller files and requiring more parallelism; or copying the entire data several times to introduce the several partitions on the storage layer.</p><h3>Schema Evolution</h3><p>The ideal modern data platform should allow for the data schema to non-disruptively evolve as time progresses. This may include, for example, adding/dropping/changing columns. The coupling of data and metadata on the Hive Table Format’s approach means many changes to the table structure have to result in rewriting the actual data files on the object storage.</p><p>This is especially important when executing some operations that an be considered “unsafe”, e.g. when multiple partitions are involved in a single operation, or when changing a column position or its data type. Without proper handling of the table metadata, which is handled in a safe and transactional manner, existing code and queries will break.</p><h3>Data Validation and Quality Controls</h3><p>A typical issue with all data platforms is garbage-in garbage-out — the lack of proper data cleansing and validation often leads to wrong answers to queries and even wrong decisions being made.</p><p>Data errors can of course also expose your entire pipeline to crashes and unexpected behaviors, so it’s also a matter of stability and operations peace of mind.</p><p>The ultimate data platform will be sophisticated enough to provide controls for data validation and quality assurance. We could expect it to enforce rules on fields (RegEx validation; timestamp fields should not have 80 year-old date; etc) , and even avoid regulatory issues by preventing PII data from being stored except from in designated fields.</p><h3>Point-in-time View</h3><p>Having a point-in-time view of data is another capability that is important for change management or disaster recovery. The ability to “go back in time” and query and view data as it looked like on a specific date and time, including data that has since been deleted, can significantly improve resilience and software development pace.</p><h3>Views and Materialized Views</h3><p>Like with any database, the ability to create Views and Materialized Views so query parts can be reused, and even better — precomputed, is a significant boost to many workloads. The better the support for those, the better and more performant the data platform will be.</p><h3>Great Operations Story</h3><p>Multi-cloud (and multi-region within the same cloud), easy maintenance, low-touch for operations, good monitoring for any housekeeping tasks — are all important to ensure smooth operations and good overall health. Those are more than just nice to have — but the list is too long to put it all here. A great operations story is critical for any technology to succeed, especially one that is going to be in charge of huge amounts of business critical data.</p><h3>Summary</h3><p>The Hive Table Format is a great entry point for an in-depth discussion on data storage formats applicable for Big Data platforms, which can also enable efficient querying.</p><p>Hive defined a lot of how we do things at present day, but it has aged and requirements have evolved over the years. It is probably a good time to find a suitable replacement, and keep pushing the envelope.</p><p>In the next posts we will dive into several other table formats which come as successors to the Hive table format. Stay tuned!</p><p><em>Originally published at </em><a href="https://bigdataboutique.com/blog/hive-tables-and-whats-next-for-modern-data-platforms-1xts1m"><em>https://bigdataboutique.com</em></a><em> on September 15, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84526872e2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architectures of a Modern Data Platform]]></title>
            <link>https://medium.com/@synhershko/architectures-of-a-modern-data-platform-ac6e28f4a7c4?source=rss-1289b2f838da------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac6e28f4a7c4</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Itamar Syn-Hershko]]></dc:creator>
            <pubDate>Tue, 02 Aug 2022 20:42:53 GMT</pubDate>
            <atom:updated>2023-06-29T12:39:44.092Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>In this post, first in a series, we will look at how a typical Data Warehouse and Data Lake architectures are designed and built, and the technologies involved.</em></blockquote><p>We live in an era of data. Data is in every organization’s strategy, every engineer’s job description, and every CIO’s dreams (or nightmares). Day in, day out, more data collectors and more data generators are being built. The collectors are observability tools and data platforms, and the generators are users of applications and websites, IoT devices and sensors, and diverse levels of infrastructure.</p><p>This presents a challenge. The data itself presents untold opportunities and insights, but organizations consuming this data struggle to ingest, process, and query it in its vast volume. Data velocity is also an increasing problem for organizations, as they may not have the platforms or infrastructure in place to handle the data at the speed it’s generated or streamed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v1cJ2f500YXH51cyKC2bIA.png" /><figcaption>Blueprint for a modern data platform — Data warehouse, data lake and ETLs</figcaption></figure><p>Whilst many software solutions exist to address the challenges posed by such quantities of data, this in itself poses one of two challenges. One challenge is that the existing data platform uses legacy architectures or technologies, and is unable to scale, patch, and accommodate new requirements effectively. The other challenge is that organizations are embracing new technologies, but there is a pervasive lack of knowledge or experience. Some technologies are more common and have market dominance over others, but frequently organizations will use varying technologies and methodologies without clear direction and strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*0WwDGEYI0-eXEkO2.png" /></figure><p>As the tongue-in-cheek tweet above cleverly points out, the definition of big data is vague. However, the challenges caused by big data are well known and well documented. Perhaps the biggest challenge of all is the most obvious: how do you deal with it?</p><p>In this series of blog posts, we’re going to examine the challenges faced by organizations and the methodologies and architectures available to them to address big data problems. Today these problems for most organizations are storing and querying large volumes of data at rest, analyzing and deriving insights from data in real-time as it streams through the data pipelines, and all of that while maintaining economic efficiency.</p><p>Let’s look at how one can build a platform that helps deriving important insights from precious data, while keeping the cost-of-insight low so there is good return on investment to the business.</p><h3>OLTP vs OLAP</h3><p>A modern data platform is designed to accommodate OLAP systems, whilst those systems in turn partially rely upon OLTP systems. Before we dive into details, let’s examine the differences between the two and will explain exactly why you need both.</p><h3>OLTP</h3><p>The first part of a modern data platform is there to support online transaction processing (OLTP). This is normally the single source of truth for the system, where data entities are created, altered, and mutated. The most important characteristic of an OLTP system is its <a href="https://en.wikipedia.org/wiki/ACID">ACID guarantees</a>. Since it’s supposed to maintain transactions, it is ought to provide Atomicity, Isolation and Consistency guarantees. The Durability requirement goes without saying, as this database is now your single source of truth.</p><p>Naturally, traditional SQL databases are often used as the OLTP part of the data platform, as they are fully ACID compliant and battle tested over decades of operation.</p><p>OLTP systems are used across multiple industries across the world to maintain a record of transaction data. Their role in a modern data platform is to store, record, and retrieve that data when queried. OLTP systems are fundamentally able to provide simple insights to end-users in a straightforward way. Because the data in data platforms is constantly evolving, mutable data (such as that stored in OLTP systems) needs to be reliable, such as with ACID guarantees.</p><h3>Challenges with OLTP Systems</h3><p>Whilst OLTP systems are perfect for serving as your single source of truth, they do have limitations. The biggest challenge with SQL databases which are frequently used for OLTP is schema management, ORMs and mismatch between objects and the normalized forms SQL databases are using. But those are well known and well documented challenges.</p><p>OLTP systems are designed to manage data mutations efficiently and consistently, and to safe-keep data. They will struggle when used as a serving layer or for analytics. Those usage-patterns will incur a large number of concurrent queries, and those queries will often require Join between tables and other costly read-time operations</p><p>Additionally, OLTP systems are not designed to handle “what if” queries or any type of complex decision-making.</p><p>Because of the above challenges, OLTP systems should always be used in conjunction with an OLAP system in a modern data platform. The OLAP system can support deep and costly analytics, usually be cheaper to maintain, and be used either as a serving layer in some occasions, or in addition to a separate serving layer.</p><h3>Characteristics of OLAP Systems</h3><p>Online Analytical Processing systems (OLAP) are used when you need to scale out analytics beyond basic queries, and when economically you can’t use OLTP systems for such computations. The OLAP system is normally a different, specialized type of database, that is built from the ground up to support efficient analytic-type queries.</p><p>OLAP systems are often used for performing business decisions, data mining, and carrying out complex calculations. Unlike OLTP systems, which act as a single source of truth, OLAP systems use historical data which is out of date. However, OLAP systems will often have additional data that isn’t present in OLTP systems.</p><p>Whilst OLTP systems are normally the data sources for OLAP systems, OLAP systems will often contain more detailed and enriched data fields than those found in OLTP systems. Updates to OLAP data are often pushed through in more lengthy batch operations, as opposed to real-time user-driven updates. The OLAP system usually stores a lot of information that isn’t typically stored on the OLTP layer — such as usage logs, events and a lot of other data that can be later used for behavioral analytics.</p><p>A common challenge with OLAP systems is to reduce the delta between them at OLTP systems, so that the data being analyzed is as close to the single source of truth as possible.</p><p>Queries are generally slower in OLAP systems for two reasons. First, OLAP systems will normally have far more data in them than OLTP systems. Second, OLAP systems will deal with far more complex queries. It also makes more economic sense to put fewer resources into an OLAP system to keep the cost of OLAP systems as low as possible, whilst maintaining a beneficial time to insight.</p><p>In this way, OLAP systems enable organizations to make data-driven decisions. If it weren’t for the ability to reduce cost of storage and compute, organizations may have opted to discard a lot of data that may have been useful later.</p><h3>OLTP and OLAP Systems Together</h3><p>To explain the synergy between OLTP and OLAP systems, take an eCommerce website as an example.</p><p>On the website, a cart will be maintained for each individual user as they progress through their shopping experience. This cart may have items from the eCommerce site’s catalog added or removed from it, and promotions may be applied. When an order is placed, both a record of the order and a record of its items will be created. This all occurs in an OLTP system.</p><p>The eCommerce organization’s OLAP system will maintain a copy of the data in the OLTP database but will contain various additional data fields that were enriched in a latent process while ingested to the OLAP platform. In the case of an eCommerce site, these will be fields that will assist the organization in deriving insights from the data. It’s the enrichment of that data, such as enhanced user profiles, customer spending history, and aggregations of orders, that drives efficient and actionable insights for organizations.</p><p>In addition to all past catalogs, user profiles and the full history of orders and order lines, the OLAP layer will often persist tables of immutable events of items added and removed from the cart, or user behavior on the website, and so on — which will allow to derive sophisticated insights such as alternate products, user reaction to promotions, product features and bugs, and so on.</p><p>When you hear people talking about either Data Warehouse or Data Lake solutions, in fact these are both OLAP systems.</p><h3>Data Warehouse and Data Lake Architectures</h3><p>The use of the term “data platform” is key. Today, the titles “Data Lake” and “Data Warehouse” are often used interchangeably and have slight differences in reality. Those differences are usually around maturity of the data and how well defined the schema used for it is. If done correctly, though, they tend to have similar architecture and both are describing OLAP systems. In this section, we are going to explore the common features of these solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rTw71UU9UHns9kPc.png" /></figure><h3>Data Ingest</h3><p>At the start of a data platform is the data flowing in (depicted on the far left of the diagram above). The data will be flowing in from multiple sources (such as various OLTP databases, IoT devices and sensors, and many other sources) and subject to either real-time ingestions, micro-batch ingestion, or batch ingestion.</p><p>Real-time ingestion (or streaming ingestion) is a hot topic. As mentioned above, the smaller the delta between your data platform and the OLTP system, then the more accurate and useful the insights generated are. Whilst real-time ingestion data will never be truly ‘live’, it’s closer than batch ingestion.</p><h3>Queuing</h3><p>The backbone of a data platform is the queuing system. In the diagram, we’ve depicted Kafka acting as the queuing system and message broker, but it could be a variety of tools such as AWS Kinesis, Google PubSub, Apache Pulsar, or myriad others.</p><p>The queuing technology acts as a message broker between the ingested data and the ETL/ELT engines. The broker buffers data between the two stages, preventing the rest of the pipeline from being overwhelmed if there is a surge of data or an unexpected volume ingested. This is more of a risk with micro-batch and real-time ingestion as they are more prone to unforeseen floods of data being moved into the pipeline. In many cases, it also allows replaying events in case of error or when a new version is deployed.</p><h3>ETL or ELT Engines</h3><p>A vital part of a data platform after data has been ingested into the queue is the ETL or ELT engines. There are turnkey software products available such as <a href="https://www.upsolver.com/">Upsolver</a>, but organizations with more varied data and greater big data challenges often build their own pipeline often based on open-source tools such as <a href="https://airflow.apache.org/">Airflow</a>, <a href="https://nifi.apache.org/">Apache NiFi</a>, and many many others. ETL and ELT engines are both intended for the same purpose. They take data and merge it, and the end product is left at rest in storage for the next stage of the process. How they achieve that end product varies.</p><p><strong>Extract Transform Load (ETL)</strong> is the most common data processing model. An ETL process takes the buffered data from a wide variety of sources and regiments it into structured data. It then writes that converged data to a target, which is typically just storage optimized for querying.</p><p>Unlike ETL, <strong>ELT engines</strong> first move all the data onto the target storage and then carry out the transformative merging and converging, with an interactive approach where the result of each micro-transformation is also stored on the target storage. Whilst this is more costly in terms of storage resource, it means the whole process is auditable.</p><h3>Storage</h3><p>Storage is an important part of a data platform, particularly if you are running an ELT engine (see above). Selecting the correct storage is also important in terms of the jobs, queries, and analytics you plan on running on your data. It might be either object storage like AWS S3, Google Cloud Storage, or Azure Blob Storage, or solutions like Red Hat Ceph which are often used in data platforms in on-premises air-gapped data centers.</p><h3>Stream Processing</h3><p>Stream processing technologies have made huge progress in recent years. Use-cases that require low latency response, such as alerting, anomaly detection and so on can now be built and operated directly on the stream of incoming data and achieve great level of accuracy as well as great performance.</p><p>Traditional use-cases that previously required batch processing are now available for stream processing. This is made possible by tools like <a href="https://flink.apache.org/">Flink</a> and <a href="https://kafka.apache.org/documentation/streams/">Kafka Streams</a>, which computes data in real-time directly on the data streaming from the queue instead of waiting for it to move into storage first.</p><p>Usually executing computations and queries on optimized data on storage will be far more economical than running them on the stream, because of the computational resources required.</p><h3>Batch Jobs and Query Engines</h3><p>Once the data is at rest on storage, you’re able to run either batch jobs, machine learning jobs or ad-hoc queries against it for your analytics.</p><p>Batch jobs are normally analytics and data processing tasks run on a large volume of data at set intervals. In this more traditional approach, data is accumulated and then sent for processing <em>en masse</em>. This method of processing is normally more predictable in terms of compute resources required, hence more cost effective.</p><p>Whilst it has been around for a while, batch processing is still the first choice for use-cases such as generating reports, customer billing information, and order fulfillment. If you don’t require fresh and most recent up to date data, and can wait a bit for the response, then you might be better off running batch jobs.</p><p>This is usually true for Machine Learning and other types of clustering jobs, some ETL process, and so on. The usual go to technology here is <a href="https://spark.apache.org/">Spark</a>, and Flink can also be used for batch jobs just like it is being used for streaming analytics.</p><p>Some other technologies, namely Presto and <a href="https://trino.io/">Trino</a>, were built to allow what we call “interactive queries” also on large volumes of data. That is why we refer to them here as query engines — their purpose is to enable the possibility of just querying the data, without running lengthy jobs. For that to work some things need to happen (storage optimization, proper cluster sizing, and few more).</p><h3>Summary</h3><p>There are a lot of factors to consider when choosing your future data platform. For every component we’ve discussed in this piece, there are a plethora of challenges. Data collection, running and monitoring ETL jobs, storage optimization, query performance, data cleanup, data quality — the list can be endless.</p><p>In the following posts in this series we will address the challenges we consider to be the biggest and most important. We will discuss the technologies that allow us to solve those challenges, and compare various approaches and solutions.</p><p>At <a href="https://bigdataboutique.com/">BigData Boutique</a> we provide consultancy on Big Data architectures and technologies, including hands-on development services. We also act as Architects-as-a-Service for organizations wishing to build or optimize their data platform. This blog and the following series is a summary of our experience in this field. Feel free to join us for the ride.</p><p><em>Originally published at </em><a href="https://bigdataboutique.com/blog/architectures-of-a-modern-data-platform-aaw0xs"><em>https://bigdataboutique.com</em></a><em> on August 2, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac6e28f4a7c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>