<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anup Rawka on Medium]]></title>
        <description><![CDATA[Stories by Anup Rawka on Medium]]></description>
        <link>https://medium.com/@anuprawka?source=rss-16e51b79c6c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d5ePP7-Vy77R5bqS9tCiow.jpeg</url>
            <title>Stories by Anup Rawka on Medium</title>
            <link>https://medium.com/@anuprawka?source=rss-16e51b79c6c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:17:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anuprawka/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond Model Routing: Enterprise AI Agents Need a Cognitive Scheduler]]></title>
            <link>https://anuprawka.medium.com/beyond-model-routing-enterprise-ai-agents-need-a-cognitive-scheduler-8ec9063d82bb?source=rss-16e51b79c6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ec9063d82bb</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[enterprise-architecture]]></category>
            <dc:creator><![CDATA[Anup Rawka]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 05:59:18 GMT</pubDate>
            <atom:updated>2026-09-10T17:14:00.900Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Model routing was only the first optimization layer. The next generation of enterprise agent platforms must decide not only which model to use, but how much reasoning a task deserves, what context should reach it, and whether an LLM should be involved at all. <br></strong>Co-authored by [<a href="https://medium.com/@praveenakuppa">https://medium.com/@praveenakuppa</a>]</p><p>The waste rarely comes from one spectacularly large model call. It accumulates through ordinary choices: reasoning for a status lookup, loading twenty tools when two matter, regenerating a templated document, or asking an LLM to orchestrate a known workflow. At platform scale, those choices shape cost, latency, and reliability.</p><p>Imagine an operations agent supporting a deployment platform. It receives four requests:</p><ul><li>“What version is deployed?”</li><li>“Generate the standard rollout configuration.”</li><li>“Summarize these health-check failures.”</li><li>“Why does this deployment fail intermittently?”</li></ul><p>Same agent, very different workloads: API lookup, governed template, bounded interpretation, and deep reasoning across logs, metrics, configuration, and history.</p><p>Routing all four through the same premium reasoning path is convenient, but architecturally wasteful.</p><p>This architecture grew from a hackathon prototype asking a narrower question: <strong>which model tier does a registered capability need?</strong> The classifier used metadata already available around the capability — tools, examples, prompt structure, output schema, and deterministic or template potential.</p><p>The experiment quickly showed that model recommendation was only part of the answer. Some capabilities could be characterized once and reused, some needed no LLM, and others changed complexity with the request.</p><p>What started as a model-selection problem was already becoming an execution-policy problem.</p><p>By 2026, <a href="https://arxiv.org/abs/2603.04445">dynamic multi-model routing</a> has become a well-developed area spanning difficulty-aware routing, uncertainty, cascades, preferences, and learned routing policies. The more interesting architecture question is therefore broader:</p><p><strong>How should this task execute?</strong></p><p>We call the control layer that answers this a <strong>Cognitive Scheduler</strong>: it determines how much intelligence a task deserves and which execution mechanism should provide it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XfsDKf0NJZZIyJzyKEOJjw.png" /></figure><h3><strong>The Architecture in Brief</strong></h3><p>A Cognitive Scheduler treats reasoning as a managed platform resource.</p><p>For predictable capabilities, execution decisions can happen before runtime: analyze metadata, store a reusable policy, and reserve runtime intelligence for ambiguity and exceptions.</p><p>The control-plane pattern is:</p><p><strong>Analyze once. Store policy. Execute many times.</strong></p><p>The scheduler can choose premium or economical reasoning, retrieval, deterministic rendering, direct tools or workflows, cache, or no LLM at all.</p><h3><strong>Model Routing Was the First Step</strong></h3><p>Model routing solves a real inefficiency: a structured summary should not consume the same inference budget as multi-source incident analysis.</p><p>But it still asks: <strong>Which LLM should I call?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*TuWB1fE0tI-WszyyGq3SQw.png" /></figure><p>The architectural shift is therefore from <strong>model routing to execution-path routing</strong>.</p><h3><strong>The Routing Problem Has Expanded</strong></h3><p>Agent runtimes already make several distinct decisions, whether explicitly modeled or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vm1ISrlu3XvTmMcw7QlVmQ.png" /></figure><p>The optimization surface has expanded because models are no longer the only expensive thing entering the reasoning path.</p><p>OpenAI’s <a href="https://openai.com/index/introducing-gpt-5-4/">GPT-5.4 Tool Search</a> addresses this at the tool layer: rather than placing every tool definition into context upfront, a lightweight representation can be available first and the complete tool definition loaded when needed.</p><p>Microsoft’s <a href="https://learn.microsoft.com/en-us/agent-framework/agents/skills">Agent Skills</a> follows a similar progressive-disclosure pattern: advertise compact capability metadata, load detailed instructions after selection, and bring in supporting resources only when required.</p><p>The architectural shift is to treat these mechanisms as one <strong>versioned, policy-aware scheduling problem</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OT7lKJH1OuBQ0N2Ls5Gcw.png" /><figcaption>The optimization problem has expanded from choosing a model to scheduling the complete execution policy</figcaption></figure><h3><strong>Reasoning Should Become Schedulable Compute</strong></h3><p>Compute schedulers do not place every workload on the largest machine; they allocate resources to workload needs.</p><p>Agent platforms need the same abstraction for intelligence: select a model, constrain reasoning effort, limit context and retrieval, control capability visibility, and choose deterministic or generative execution.</p><p>A status lookup may need no reasoning; a validation summary may need little; remediation comparison needs more; intermittent cross-system diagnosis may justify deep reasoning.</p><p>Reasoning effort is increasingly becoming a controllable execution variable. OpenAI’s <a href="https://openai.com/index/builders-guide-to-gpt-5-6/">production guidance for GPT-5.6</a> discusses using model choice and reasoning effort together rather than treating model identity as the only optimization lever.</p><p>The scheduler may therefore prefer the same model with lower reasoning effort rather than switch models.</p><h3><strong>Do More Work Before Runtime</strong></h3><p>Much of a capability’s execution profile can be inferred before a user sends a request.</p><p>Registered skills often expose tools, examples, prompts, output schemas, and expected behavior.</p><p>Obvious cases are easy: a read-only status API with a fixed schema looks very different from a capability that correlates logs, configuration, and telemetry into an open-ended diagnosis.</p><p><strong>The edge cases matter more.</strong> Tool semantics, output structure, and examples often reveal more than tool count or keywords. A word such as “analyze” can describe either simple dispatch or complex evidence synthesis.</p><p>“Analyze deployment health and recommend next steps.”</p><p>It could mean checking one health endpoint against a threshold and returning a fixed recommendation — or correlating several signals to infer a failure mode and remediation.</p><p>Execution profiles therefore need combined evidence: tool patterns, examples, output structure, deterministic potential, and semantic context.</p><p>For stable capabilities, classify at registration or material change; runtime then performs a policy lookup instead of rediscovering the same characteristics on every invocation.</p><p>The classification is a default, not permanent truth.</p><h3><strong>The Cognitive Execution Profile</strong></h3><p>Treat the <strong>Cognitive Execution Profile</strong> as versioned workload metadata describing how a capability should normally execute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QryNpUhT__MeOkbJAOIc4w.png" /><figcaption>A Congnitive Execution Profile describes how a capability should execute, not mearly which model it should use</figcaption></figure><p>It can capture execution mode, model class, reasoning effort, context and retrieval budgets, tool disclosure, orchestration, caching, escalation, policy constraints, and quality or latency requirements.</p><p>Confidence is part of the profile, not an afterthought.</p><p>Sparse or conflicting metadata should not carry the authority of an obvious classification. <strong>Category records what the platform believes; confidence determines how strongly that belief should drive execution.</strong></p><p>Lower confidence can trigger conservative fallback, human review, richer analysis, or request-level classification.</p><p>Store confidence, rationale, version, and override state with the recommendation.</p><p><strong>Capability Routing vs. Execution Scheduling</strong></p><p>A <strong>Capability Router</strong> decides which skill, agent, workflow, service, or tool should handle a request.</p><p>The <strong>Cognitive Scheduler</strong> decides how it should execute.</p><p>For “Why did this deployment fail?”, routing may select diagnostics; scheduling then chooses reasoning depth, context, tools, and escalation policy.</p><p>Discovery chooses <strong>what</strong>.</p><p>Scheduling determines <strong>how</strong>.</p><p><strong>Enterprise Cognitive Scheduling Architecture</strong></p><p>A production design benefits from separating relatively stable control-plane decisions from high-volume runtime execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_M2Pl_ua1FR5Ffafsdg5Zw.png" /><figcaption>The control plane establishes reusable execution profiles; runtime applies policy, context, discovery, and feedback to select the appropriate execution path.</figcaption></figure><p><strong>Control Plane</strong></p><p>The control plane analyzes registration metadata, produces a Cognitive Execution Profile, and manages policy and versioning.</p><p>Material changes to tools, schema, examples, or behavior trigger reassessment.</p><p><strong>Runtime</strong></p><p><strong>discover capability → select context → apply policy → schedule execution → validate result</strong></p><p>Stable decisions need not be reconsidered on every request.</p><p><strong>Feedback</strong></p><p>Production evidence — quality, latency, escalation, retries, corrections, and execution cost — can indicate that the stored default is no longer appropriate.</p><p>That feedback can trigger recalibration and a new profile version.</p><p><strong>Policy and Governance Come Before Optimization</strong></p><p>An enterprise scheduler must first determine which execution paths policy allows, then optimize within that set.</p><p>For example, summarization may allow several models, while production-changing actions are restricted to approved tools and workflows.</p><p>Conceptually:</p><p><strong>Minimize execution cost</strong></p><p>subject to:</p><p><strong>Quality ≥ required threshold</strong><br> <strong>Latency ≤ required SLO</strong><br> <strong>Reliability ≥ required threshold</strong><br> <strong>Policy = satisfied</strong></p><p>Confidence, overrides, auditability, and quality regression govern how aggressively the profile is applied.</p><p>The platform should be able to reconstruct the active profile, constraints, selected route, and rationale.</p><p><strong>Six Execution Paths, Not Six Model Tiers</strong></p><p>The scheduler is not choosing model tiers; it is selecting different execution mechanisms.</p><p><strong>Complex Reasoning</strong></p><p>Some workloads genuinely justify deep inference: root-cause analysis, ambiguous debugging, architecture evaluation, or multi-source causal investigation.</p><p>“Why does this deployment fail intermittently?” belongs here when answering it requires correlating logs, configuration, health signals, and previous attempts.</p><p><strong>Constrained Reasoning</strong></p><p>Other tasks require interpretation inside a bounded problem.</p><p>“Summarize these health-check failures” may need a model, but not necessarily the most expensive reasoning configuration.</p><p>An economical model, lower reasoning effort, or both may be sufficient.</p><p><strong>Retrieval + Reasoning</strong></p><p>Instead of placing an entire operational catalog into context, retrieve the deployment metadata, health checks, and relevant runbook sections first.</p><p>The model then reasons over a smaller and more intentional context.</p><p><strong>Deterministic Execution</strong></p><p>This became one of the most important lessons from the prototype.</p><p>“Generate the standard rollout configuration” sounds generative. But if the task has predictable inputs, a fixed output structure, a governed template, and authoritative data sources, then the job may really be:</p><p><strong>select template → validate parameters → populate → render</strong></p><p>The prototype had started with a simpler question: which model should a capability use? For tasks like these, model selection was already the wrong question.</p><p>The better question was:</p><p><strong>Why invoke a model at all?</strong></p><p><strong>The cheapest model call is the one the architecture determines it does not need.</strong></p><p>Zero-LLM execution also improves reproducibility. Deterministic artifacts can be versioned, tested, validated, and compared predictably.</p><p><strong>Direct Tool or Workflow Execution</strong></p><p>“What version is deployed?” should usually resolve to a direct system lookup.</p><p>Known workflows deserve similar treatment. A standard rollout sequence should not be rediscovered by an LLM every time it executes.</p><p>Microsoft’s <a href="https://opensource.microsoft.com/blog/2026/05/14/conductor-deterministic-orchestration-for-multi-agent-ai-workflows/">Conductor</a> illustrates the same distinction for multi-agent systems: known orchestration can be declared deterministically, leaving dynamic reasoning for tasks where the path genuinely needs to be discovered.</p><p><strong>Cache and Reuse</strong></p><p>Some work does not need to run again.</p><p>Validated artifacts, stable context bundles, repeated results, and reusable prompt prefixes may be safely reused when identity, freshness, and data-version constraints allow it.</p><p>Cache eligibility therefore belongs in execution policy rather than being an ad hoc application optimization.</p><p><strong>Progressive Disclosure and Context Scheduling</strong></p><p>Large skill and tool catalogs create their own hidden token tax.</p><p>If an agent has access to dozens or hundreds of capabilities, eagerly loading every schema and instruction is expensive even when most are irrelevant.</p><p>A more scalable sequence is:</p><p><strong>Advertise → Select → Load → Retrieve → Execute</strong></p><p>Our deployment agent does not need the complete operations catalog to answer a rollout question. It needs the relevant capability, deployment metadata, and a small set of supporting tools.</p><p>OpenAI’s <a href="https://openai.com/index/introducing-gpt-5-4/">Tool Search</a> and Microsoft’s <a href="https://learn.microsoft.com/en-us/agent-framework/agents/skills">Agent Skills</a> both reflect this move toward metadata-first discovery and on-demand loading.</p><p><strong>Static by Default, Adaptive When Evidence Demands It</strong></p><p>Registration-time classification works well when capability behavior is stable.</p><p>Some capabilities are not.</p><p>Consider a data-analysis skill receiving both:</p><p>“What columns are in this dataset?”</p><p>and:</p><p>“Why did yesterday’s pipeline produce inconsistent results, and which upstream transformation is most likely responsible?”</p><p>The first is largely retrieval. The second may require multi-step investigation.</p><p>The prototype primarily classified capabilities. That worked when a capability had a stable purpose. It became too coarse when the same capability could receive requests with very different reasoning requirements.</p><p>The hybrid design is more practical:</p><p><strong>Stable capability → classify offline</strong></p><p><strong>Mixed-complexity capability → establish an offline default → classify individual requests only when needed</strong></p><p><strong>Keep the common path deterministic. Spend intelligence on exceptions.</strong></p><p>There was a similar tradeoff in the classifier itself.</p><p>The prototype explored deterministic, evidence-weighted rules because they were fast, inexpensive, reproducible, and auditable. They worked well for obvious cases, but became brittle when vocabulary changed, generic keywords collided, or tool usage failed to reveal actual reasoning complexity.</p><p>A model-assisted classifier can interpret richer semantics, but it introduces another inference call, latency, and additional operational complexity.</p><p>The practical conclusion was to use the cheapest decision mechanism that provides enough confidence, and escalate the classification mechanism only when necessary.</p><p>The profile can then evolve through:</p><p><strong>Default Profile → Execute → Observe → Evaluate → Recalibrate → Version New Profile</strong></p><p>There is one more caution. A sophisticated router is not automatically a better router.</p><p><a href="https://aclanthology.org/2026.findings-acl.1881/">LLMRouterBench</a>, published in the Findings of ACL 2026, found substantial model complementarity while also reporting that several sophisticated routing approaches did not consistently outperform a simple baseline under unified evaluation.</p><p>Rules, lightweight classifiers, learned routers, and LLM-based routing should therefore be tested against the organization’s actual workload distribution.</p><p>Routing intelligence has a cost of its own.</p><p><strong>Agent Unit Economics</strong></p><p>Token counts are useful operational telemetry, but they are not a sufficient business metric.</p><p>Suppose an inexpensive route costs less on the first attempt but frequently produces an unacceptable result, retries, and eventually escalates. Another path costs slightly more initially but succeeds reliably.</p><p>Per-call accounting favors the first route. End-to-end economics may favor the second.</p><p>A more useful measure is:</p><p><strong>Cost per Successful Task</strong></p><p><strong>Total execution cost ÷ accepted successful outcomes</strong></p><p>That total can include inference, retrieval, tools, supporting compute, retries, escalation, and validation.</p><p>AWS’s <a href="https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentcost02-bp01.html">Agentic AI Lens</a> makes a similar argument with <strong>cost-per-correct-response</strong>.</p><p>For our deployment agent, the implication is straightforward: a status lookup should not inherit the economics of intermittent-failure diagnosis merely because both happen inside the same agent session.</p><h3><strong>Before vs. After Architecture Thinking</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQ25JYzSdemJJxE87Fvebg.png" /></figure><p><strong>Five Architecture Lessons</strong></p><p><strong>1. Reasoning is a schedulable resource</strong></p><p>Allocate intelligence according to workload requirements rather than consuming it by default.</p><p><strong>2. The model should not be the universal runtime</strong></p><p>Retrieval, APIs, templates, workflows, caches, and deterministic code remain first-class components of an agent platform.</p><p><strong>3. Control-plane intelligence scales better than repeated runtime intelligence</strong></p><p>Stable decisions belong in reusable policy. Runtime reasoning should focus on what genuinely changes from request to request.</p><p><strong>4. Capability routing and execution scheduling are separate concerns</strong></p><p>Selecting what should handle a task and deciding how it should execute should be able to evolve independently.</p><p><strong>5. Optimize successful outcomes, not token counts</strong></p><p>A cheaper first call is not an optimization if it increases retries, quality failures, or escalation.</p><p><strong>Practical Architecture Checklist</strong></p><p>When reviewing an enterprise agent platform, we would ask:</p><ul><li>Can each major capability have a versioned execution profile?</li><li>Are stable capabilities analyzed at registration time?</li><li>Can the platform select a zero-LLM execution path?</li><li>Are skills and tools progressively disclosed?</li><li>Can reasoning effort vary by task?</li><li>Does policy remove ineligible paths before optimization?</li><li>Is request-level classification reserved for mixed or ambiguous workloads?</li><li>Do runtime outcomes feed profile recalibration?</li><li>Is cost measured per successful task?</li><li>Can every scheduling decision be reconstructed for audit?</li></ul><p><strong>Conclusion</strong></p><p>The first generation of enterprise agent platforms mostly asked:</p><p><strong>Can an LLM perform this task?</strong></p><p>Model routing improved that question:</p><p><strong>Which model should perform this task?</strong></p><p>What began as model selection became execution policy: some workloads need deeper reasoning, some need less, some can be characterized before runtime, and some are better served by deterministic software.</p><p>The next question is broader:</p><p><strong>How should this task execute?</strong></p><p>Sometimes the answer is deep reasoning; sometimes the same model with less reasoning.</p><p>Sometimes it is targeted retrieval with a smaller context.</p><p>Sometimes it is a governed template, known workflow, direct API call, or cached result.</p><p>And sometimes the right amount of LLM reasoning is zero.</p><p>Cognitive scheduling brings capability metadata, offline analysis, execution policy, context discipline, model and reasoning selection, deterministic execution, governance, and runtime feedback into one scheduling problem.</p><p>The most scalable agent platform will not necessarily be connected to the smartest model.</p><p>It will know <strong>when intelligence is needed, how much is enough, what context should reach it, and when conventional software is the better execution engine</strong>.</p><p>That is a platform architecture problem.</p><h3>References</h3><ol><li><strong>OpenAI — Introducing GPT-5.4</strong><br> Tool Search and more efficient use of large tool ecosystems. The article explains how tool definitions can be loaded on demand instead of placing the entire tool surface into context upfront. <br> <a href="https://openai.com/index/introducing-gpt-5-4/?utm_source=chatgpt.com">Introducing GPT‑5.4</a></li><li><strong>OpenAI — The Builder’s Guide to GPT-5.6</strong><br> Guidance on model selection, reasoning effort, and balancing capability, latency, and cost in production agent workloads. <br> <a href="https://openai.com/index/builders-guide-to-gpt-5-6/?utm_source=chatgpt.com">The Builder’s Guide to GPT‑5.6</a></li><li><strong>Microsoft — Agent Skills</strong><br> Agent Skills use progressive disclosure so agents initially see compact skill metadata and load instructions and supporting resources only when required. <br> <a href="https://learn.microsoft.com/en-us/agent-framework/agents/skills?utm_source=chatgpt.com">Microsoft Agent Skills documentation</a></li><li><strong>Microsoft — Conductor: Deterministic Orchestration for Multi-Agent AI Workflows</strong><br> An example of separating known workflow orchestration from dynamic LLM reasoning, using deterministic and inspectable workflow definitions. <br> <a href="https://opensource.microsoft.com/blog/2026/05/14/conductor-deterministic-orchestration-for-multi-agent-ai-workflows/?utm_source=chatgpt.com">Conductor: Deterministic orchestration for multi-agent AI workflows</a></li><li><strong>Li et al. — LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing, Findings of ACL 2026</strong><br> A large-scale evaluation of LLM routing across more than 400K instances, 21 datasets, and 33 models. The study finds strong model complementarity, while showing that several sophisticated routing approaches do not reliably outperform simpler baselines. <br> <a href="https://aclanthology.org/2026.findings-acl.1881/?utm_source=chatgpt.com">LLMRouterBench — ACL Anthology</a></li><li><strong>AWS — Agentic AI Lens: Architect Tiered Model Selection for Cost-Performance Optimization</strong><br> AWS recommends task-level complexity tiers, escalation based on confidence, and measuring <strong>cost-per-correct-response</strong> rather than optimizing only for the cost of individual model calls. <br> <a href="https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentcost02-bp01.html?utm_source=chatgpt.com">AWS Agentic AI Lens — Tiered Model Selection</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ec9063d82bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1:Build Your First Spark Debugging & Optimization Agent]]></title>
            <link>https://anuprawka.medium.com/build-your-first-spark-debugging-optimization-agent-3941432450aa?source=rss-16e51b79c6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3941432450aa</guid>
            <dc:creator><![CDATA[Anup Rawka]]></dc:creator>
            <pubDate>Thu, 01 Jan 2026 08:18:08 GMT</pubDate>
            <atom:updated>2026-01-15T06:18:17.596Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A practical guide to automating Spark job analysis with structured metrics and LLM reasoning</em></p><p>Debugging production Spark workloads is a well-known challenge for data engineers. You open Spark UI, inspect stages and tasks, notice shuffle spikes, spills, executor losses, or long-running stages — and then face the hardest part:</p><p><strong>What exactly should be changed to fix this job?</strong><br>Determining the correct configuration change — such as executor memory, shuffle partitions, or adaptive execution — often requires manual analysis and experience. Is the job failing due to data skew, memory pressure, excessive shuffle, or suboptimal configuration?</p><h3>Current Spark debugging workflow and its limitations</h3><p>In most Spark-based data platforms, job debugging follows a familiar pattern: engineers inspect Spark UI to identify slow or failed stages and tasks, analyze metrics such as shuffle volume, spill, executor runtime, and task retries, and then review driver and executor logs to locate errors such as out-of-memory conditions, fetch failures, or timeouts. Based on these observations, engineers form a hypothesis about the underlying issue — such as data skew, memory pressure, excessive shuffle, or suboptimal configuration — and apply one or more configuration changes in a subsequent run to validate that hypothesis.</p><p>While this workflow provides detailed visibility into execution behavior, it leaves a critical gap: Spark surfaces metrics, but it does <strong>not synthesize them, normalize them, or reason across them</strong> to explain <em>why</em> a failure or performance regression occurred or <em>which</em> configuration change is most likely to resolve it.</p><h3>What you will build next</h3><p>In this guide, you will learn how to close that gap by building a <strong>Spark Doctor agent</strong> — a system that systematically aggregates execution metrics, derives structured evidence, and produces explicit <strong>root cause analysis (RCA)</strong> and <strong>Spark configuration recommendations</strong> from the same signals engineers already use today.</p><p>This blog focuses on the practical steps required to design and build a Spark debugging agent, using execution metrics and structured reasoning to automate root-cause analysis and configuration recommendations, using:</p><ul><li>Collecting metrics using SparkListener</li><li>Building a lightweight asynchronous metrics pipeline</li><li>Aggregating and normalizing execution metrics</li><li>Designing a structured LLM prompt for root cause analysis</li></ul><p>By the end of this post, you will understand:</p><ul><li>what metrics you must collect (and which ones can mislead),</li><li>how to aggregate and normalize them correctly,</li><li>how to structure LLM prompts to avoid hallucinations,</li><li>and how to extract actionable RCA and Spark config recommendations instead of vague advice.</li></ul><p>This is the first version of what we call a <strong>Spark Doctor agent</strong>.</p><h3>The core idea: Spark metrics → evidence → reasoning → recommendation</h3><p>Most Spark tools focus on observability: showing <em>what happened</em>. What we want instead is <strong>decision support</strong>: understanding <em>why</em> it happened and <em>what to change</em>.</p><p>Here’s the mental model you should keep throughout this blog:</p><pre>Spark execution metrics<br>         ↓<br>Structured evidence<br>         ↓<br>LLM reasoning (with strict rules)<br>         ↓<br>Concrete Spark config recommendations</pre><p>The LLM is not the star of the system — the metrics pipeline forms the foundation of accurate and repeatable agent reasoning. If the metrics are wrong or misleading, the agent will confidently produce the wrong answer. Our goal is to build the pipeline correctly before calling the LLM.</p><h3>Who should read this</h3><p>This blog is for you if:</p><ul><li>you operate Spark pipelines in production,</li><li>you debug Spark failures or performance regressions,</li><li>you want to automate <em>decision-making</em>, not just monitoring,</li><li>you are curious how agents fit into real-world data engineering workflows.</li></ul><p>You don’t need:</p><ul><li>deep ML knowledge,</li><li>prompt-engineering tricks,</li><li>or exotic infrastructure.</li></ul><p>You <em>do</em> need:</p><ul><li>a firm understanding of Spark execution,</li><li>familiarity with stage/task metrics,</li><li>and willingness to treat LLM output as part of an auditable pipeline.</li></ul><h3>1) Architecture overview (three layers)</h3><p>This system is intentionally modular and opinionated. It has three layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zqeONMiojAlD8p4edv30gA.png" /><figcaption>Spark Doctor Agent Architecture</figcaption></figure><p><strong>Design principles:</strong></p><ul><li>Spark execution must <em>never block</em> on observability.</li><li>Raw metrics are collected unchanged.</li><li>Aggregation and normalization happen <em>outside Spark</em>.</li><li>LLMs only see <strong>LLM-ready evidence</strong>, never raw events.</li></ul><h3>2) Layer 1 — Metrics collection with SparkListener (Scala)</h3><h3>Why SparkListener</h3><p>To reason about Spark behavior, we need metrics from the execution engine — not logs or post-hoc summaries. SparkListener provides:</p><ul><li>task completion metrics</li><li>stage lifecycle events</li><li>failure reasons</li><li>precise timing</li></ul><p>This is the level of granularity required for meaningful RCA.</p><h3>What we collect</h3><p>At <strong>task end</strong>:</p><ul><li>executor run time</li><li>input/output bytes</li><li>shuffle read/write bytes</li><li>memory and disk spill</li><li>peak execution memory</li><li>task failure reason (if any)</li></ul><p>At <strong>stage completion</strong>:</p><ul><li>submission and completion times</li><li>number of tasks</li><li>failed task count</li><li>stage-level failure reason</li></ul><p>Below is a simplified example of a SparkListener implementation that captures task-level execution metrics in a structured way:</p><pre>import org.apache.spark.scheduler._<br>import java.time.Instant<br>import scala.concurrent.{ExecutionContext, Future}<br>import ExecutionContext.Implicits.global<br>final case class TaskMetricEvent(<br>  appId: String,<br>  stageId: Int,<br>  taskId: Long,<br>  attemptId: Int,<br>  executorRunTimeMs: Long,<br>  inputBytes: Long,<br>  outputBytes: Long,<br>  shuffleReadBytes: Long,<br>  shuffleWriteBytes: Long,<br>  memoryBytesSpilled: Long,<br>  diskBytesSpilled: Long,<br>  peakExecutionMemory: Long,<br>  failureReason: Option[String],<br>  createdAt: Long<br>)<br>class SparkDoctorListener(publisher: TaskMetricEvent =&gt; Unit)<br>  extends SparkListener {<br>  override def onTaskEnd(e: SparkListenerTaskEnd): Unit = {<br>    val m = e.taskMetrics<br>    if (m == null) return <br>      val event = TaskMetricEvent(<br>      appId = e.taskInfo.appId,<br>      stageId = e.stageId,<br>      taskId = e.taskInfo.taskId,<br>      attemptId = e.taskInfo.attemptNumber(),<br>      executorRunTimeMs = m.executorRunTime,<br>      inputBytes = Option(m.inputMetrics).map(_.bytesRead).getOrElse(0L),<br>      outputBytes = Option(m.outputMetrics).map(_.bytesWritten).getOrElse(0L),<br>      shuffleReadBytes = Option(m.shuffleReadMetrics).map(_.totalBytesRead).getOrElse(0L),<br>      shuffleWriteBytes = Option(m.shuffleWriteMetrics).map(_.bytesWritten).getOrElse(0L),<br>      memoryBytesSpilled = m.memoryBytesSpilled,<br>      diskBytesSpilled = m.diskBytesSpilled,<br>      peakExecutionMemory = m.peakExecutionMemory,<br>      failureReason = Option(e.reason).map(_.toString),<br>      createdAt = Instant.now().toEpochMilli<br>    )<br>  // publish asynchronously<br>    Future { publisher(event) }<br>  }<br>}</pre><h3>3) Layer 2 — Aggregation and normalization</h3><h3>Why raw Spark bytes mislead agents</h3><p>Metrics like inputBytes, shuffleReadBytes, and shuffleWriteBytes often reflect <strong>compressed bytes</strong> (e.g., Parquet/ORC on disk or shuffle compression). Two jobs with identical logical data may report very different physical metrics, leading to incorrect reasoning if treated at face value.</p><h3>Physical vs logical data size (with a simple example)</h3><pre>Logical data size (uncompressed)<br>            ┌─────────────────────────────┐<br>            │        200 GB               │<br>            └─────────────┬───────────────┘<br>                          │ compression (Parquet/ORC)<br>                          ▼<br>     Physical data size (on disk / network)<br>            ┌─────────────────────────────┐<br>            │         50 GB               │<br>            └─────────────────────────────┘</pre><p>A Spark job that reads 200 GB of data may report only 50 GB read due to compression. If this distinction is ignored, memory pressure could be misdiagnosed and recommendations will be suboptimal.</p><p>For example, consider a simple scenario where Spark reads compressed input data:</p><h4>A realistic illustration</h4><ul><li>Logical input: <strong>200 GB</strong></li><li>Physical bytes on disk: <strong>50 GB</strong></li><li>Shuffle (compressed): <strong>120 GB</strong></li></ul><p>Spark may report:</p><ul><li>inputBytes = 50 GB</li><li>shuffleReadBytes = 120 GB</li></ul><p>But for memory reasoning, the agent must understand the <strong>logical 200 GB</strong> is the real driver of memory pressure.</p><h3>4) Layer 3 — Agentic layer (how metrics become reasoning)</h3><p>The LLM never sees raw task events. It receives:</p><ul><li>aggregated and normalized metrics</li><li>failure summaries</li><li>stage hotspots</li><li>runtime Spark configuration</li><li>explicit metric semantics</li></ul><p>Here’s a <strong>sample structured prompt</strong> that you can use to drive RCA and configuration recommendations (where all config values come from the failed job’s own settings):</p><p><strong>Note:</strong><br>All Spark configuration values shown below are <strong>dynamically populated from the failed job’s runtime configuration</strong> collected during execution. They are not hardcoded.</p><h3>System Prompt (agent contract)</h3><p>The key to making the agent reliable is enforcing a structured reasoning flow. Below is an example prompt that explicitly constrains how the agent analyzes metrics and derives recommendations:</p><pre>You are a Spark debugging and optimization agent.<br>You will receive execution metrics, failure signals, and the runtime Spark<br>configuration of a failed Spark job.<br>Your task is to:<br>- identify the most likely root cause of the failure,<br>- explain the reasoning using the provided metrics,<br>- and recommend updated Spark configuration values.<br>Rules:<br>- Base all conclusions strictly on the provided data.<br>- Do not assume default Spark settings.<br>- Treat the provided Spark configuration as the actual runtime configuration.<br>- Output MUST be valid JSON only.</pre><h3>User Prompt</h3><pre>Spark Version: 3.3<br>Job Status: FAILED<br>[1] FAILURE SUMMARY<br>- failure_category: MEMORY_ISSUE<br>- failed_tasks: 42<br>- primary_error:<br>  &quot;ExecutorLostFailure: Container killed for exceeding memory limits&quot;<br>[2] AGGREGATED EXECUTION METRICS<br>- logical_input_data_gb: 180<br>- physical_input_bytes_gb: 52<br>- total_shuffle_read_gb: 120<br>- total_shuffle_write_gb: 115<br>- memory_spill_gb: 18<br>- disk_spill_gb: 9<br>- total_executor_run_time_sec: 2400<br>- job_wall_time_sec: 620<br>[3] STAGE HOTSPOTS<br>Stage 14:<br>- logical_input_gb: 95<br>- shuffle_read_gb: 70<br>- memory_spill_gb: 11<br>- executor_run_time_sec: 1100<br>- stage_wall_time_sec: 390<br>- failed_tasks: 28<br>[4] RUNTIME SPARK CONFIGURATION (FROM FAILED JOB)<br>The following Spark configuration values were collected directly<br>from the failed job at runtime:<br>- spark.executor.instances = 20<br>- spark.executor.cores = 4<br>- spark.executor.memory = 8g<br>- spark.executor.memoryOverhead = 2g<br>- spark.sql.shuffle.partitions = 200<br>- spark.sql.adaptive.enabled = false<br>- spark.serializer = org.apache.spark.serializer.JavaSerializer<br>[5] METRIC SEMANTICS<br>- logical_input_data_gb is derived from Parquet file statistics<br>- physical_input_bytes_gb reflects compressed file reads<br>- shuffle bytes reflect compressed shuffle transfer sizes<br>- logical data size should be used for memory pressure analysis<br>[6] TASK<br>Using the provided metrics and runtime configuration:<br>1. Identify the primary root cause of failure<br>2. List contributing factors<br>3. Recommend updated Spark configuration values<br>4. Explain how each recommendation addresses the root cause</pre><h3>Expected Output Schema</h3><pre>{<br>  &quot;rca&quot;: {<br>    &quot;primary_root_cause&quot;: &quot;&quot;,<br>    &quot;contributing_factors&quot;: [],<br>    &quot;failure_classification&quot;: &quot;&quot;<br>  },<br>  &quot;reasoning&quot;: {<br>    &quot;key_observations&quot;: [],<br>    &quot;metric_evidence&quot;: []<br>  },<br>  &quot;recommended_spark_config&quot;: {<br>    &quot;spark.executor.memory&quot;: {<br>      &quot;current&quot;: &quot;&quot;,<br>      &quot;recommended&quot;: &quot;&quot;,<br>      &quot;justification&quot;: &quot;&quot;<br>    },<br>    &quot;spark.executor.memoryOverhead&quot;: {<br>      &quot;current&quot;: &quot;&quot;,<br>      &quot;recommended&quot;: &quot;&quot;,<br>      &quot;justification&quot;: &quot;&quot;<br>    },<br>    &quot;spark.sql.shuffle.partitions&quot;: {<br>      &quot;current&quot;: &quot;&quot;,<br>      &quot;recommended&quot;: &quot;&quot;,<br>      &quot;justification&quot;: &quot;&quot;<br>    },<br>    &quot;spark.sql.adaptive.enabled&quot;: {<br>      &quot;current&quot;: &quot;&quot;,<br>      &quot;recommended&quot;: &quot;&quot;,<br>      &quot;justification&quot;: &quot;&quot;<br>    }<br>  },<br>  &quot;confidence_level&quot;: &quot;LOW | MEDIUM | HIGH&quot;,<br>  &quot;recommended_next_action&quot;: &quot;&quot;<br>}</pre><h3>Sample agent output</h3><p>Below is a <strong>realistic example</strong> of the agent’s JSON response for a failed job with memory pressure:</p><pre>{<br>  &quot;rca&quot;: {<br>    &quot;primary_root_cause&quot;: &quot;Executor memory exhaustion due to under-provisioned executor memory relative to logical input size and shuffle amplification.&quot;,<br>    &quot;contributing_factors&quot;: [<br>      &quot;High logical input data size processed with insufficient executor memory.&quot;,<br>      &quot;Large shuffle volume causing elevated memory pressure.&quot;,<br>      &quot;Memory spill indicating repeated eviction of intermediate data.&quot;<br>    ],<br>    &quot;failure_classification&quot;: &quot;MEMORY_PRESSURE&quot;<br>  },<br>  &quot;recommended_spark_config&quot;: {<br>    &quot;spark.executor.memory&quot;: {<br>      &quot;current&quot;: &quot;8g&quot;,<br>      &quot;recommended&quot;: &quot;12g&quot;,<br>      &quot;justification&quot;: &quot;Increase memory to reduce spill and OOM risk.&quot;<br>    },<br>    &quot;spark.sql.shuffle.partitions&quot;: {<br>      &quot;current&quot;: &quot;200&quot;,<br>      &quot;recommended&quot;: &quot;300&quot;,<br>      &quot;justification&quot;: &quot;More partitions reduce per-task load.&quot;<br>    },<br>    &quot;spark.sql.adaptive.enabled&quot;: {<br>      &quot;current&quot;: &quot;false&quot;,<br>      &quot;recommended&quot;: &quot;true&quot;,<br>      &quot;justification&quot;: &quot;Enable AQE for dynamic partition sizing.&quot;<br>    }<br>  },<br>  &quot;confidence_level&quot;: &quot;HIGH&quot;,<br>  &quot;recommended_next_action&quot;: &quot;Apply recommendations and rerun. Investigate remaining skew if failures persist.&quot;<br>}</pre><p>Ensure:</p><ul><li>your <strong>aggregated metrics database</strong> (Postgres) is populated,</li><li>Kafka consumers are running,</li><li>Spark jobs publish to Kafka via the listener.</li></ul><h3>Conclusion</h3><p>In this first part of the series, you learned how to build a <strong>practical Spark debugging and optimization agent</strong> by focusing on structured metrics and reasoning rather than logs and dashboards alone. We emphasized correctness, transparency, and human-in-the-loop safety. The key insight is that an agent <em>reasoning over normalized execution metrics</em> can provide far more actionable guidance than traditional manual debugging workflows.</p><h3>What’s next in This Series</h3><p>In upcoming parts, we will extend this work in two major directions:</p><ol><li><strong>Stateful agent design:</strong> enabling the agent to learn from historical runs, build context about pipelines, and improve recommendations over time.</li><li><strong>Job type awareness:</strong> Classifying Spark jobs (memory-heavy, compute-heavy, shuffle-heavy) and adapting recommendations accordingly.</li><li>Extending the agent from reactive debugging to proactive optimization and failure prevention.</li></ol><p>These additions will allow the agent to move beyond single-run analysis to adaptive, context-aware optimization and debugging.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3941432450aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Delta Lake quick-start guide with example]]></title>
            <link>https://anuprawka.medium.com/delta-lake-quick-start-with-example-0a66e61dea44?source=rss-16e51b79c6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a66e61dea44</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[feature-store]]></category>
            <dc:creator><![CDATA[Anup Rawka]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 22:34:04 GMT</pubDate>
            <atom:updated>2024-04-13T16:54:43.488Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*2gVSB3KgExE4wRSU.jpg" /></figure><h3>Delta Lake quick-start</h3><p>This Delta Lake quick-start provides an overview of the Delta lake API features. The quick-start guide shows how to perform basic CRUD operations using Delta lake API.</p><p>Please follow the below example and read the references.</p><h4>Dependencies</h4><p>Add delta-core dependency into your project</p><pre>dependencies {<br>  compile group: &#39;io.delta&#39;, name: &#39;delta-core_2.11&#39;, version: &#39;0.6.1&#39;<br>}</pre><h4>Sample code for Delta lake CRUD operations</h4><pre>import com.typesafe.config.ConfigFactory<br>import org.apache.spark.{SparkConf, SparkContext}<br>import org.apache.spark.sql.SparkSession<br>import io.delta.tables._<br>import org.apache.spark.sql.functions._</pre><pre>object TestAppDeltaLakeRunner {<br>  <br>def main(args: Array[String]): Unit = {</pre><pre>    val config = ConfigFactory.parseString(<br>      &quot;&quot;&quot;<br>        di2{<br>            featureStore{<br>                aws{<br>                    awsAccessKeyId = &quot;Redacted&quot;<br>                    awsSecretAccessKey = &quot;Redacted&quot;<br>                    basePath = &quot;s3a://vd-dev-smarttvdata-di2-data-science-virginia/feature-store/test/delta/2021-01-21/&quot;<br>                }<br>            }<br>        }<br>        &quot;&quot;&quot;.stripMargin)</pre><pre>    val conf = new SparkConf()<br>      .setAppName(&quot;DeltaLakeTest&quot;)<br>      .setMaster(&quot;local[2]&quot;)<br>      .set(&quot;spark.driver.bindAddress&quot;, &quot;127.0.0.1&quot;)<br>      .set(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;)<br>      .set(&quot;spark.sql.catalog.spark_catalog&quot;, &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;)<br>      .set(&quot;spark.delta.logStore.class&quot;, &quot;org.apache.spark.sql.delta.storage.S3SingleDriverLogStore&quot;)</pre><pre>    val sparkContext = new SparkContext(conf)</pre><pre>    sparkContext.hadoopConfiguration.set(&quot;fs.s3a.access.key&quot;, config.getString(&quot;di2.featureStore.aws.awsAccessKeyId&quot;))<br>    sparkContext.hadoopConfiguration.set(&quot;fs.s3a.secret.key&quot;, config.getString(&quot;di2.featureStore.aws.awsSecretAccessKey&quot;))<br>    sparkContext.hadoopConfiguration.set(&quot;fs.s3a.endpoint&quot;, &quot;s3.amazonaws.com&quot;)</pre><pre>    val spark = SparkSession.builder.config(sparkContext.getConf).getOrCreate()</pre><pre>    val basePath = config.getString(&quot;di2.featureStore.aws.basePath&quot;)</pre><pre>    //Read data from CSV<br>    //val df_write = spark.read.format(&quot;csv&quot;).option(&quot;header&quot;, &quot;true&quot;).load(config.getString(&quot;di2.featureStore.data.path&quot;))</pre><pre>  import spark.implicits._<br>  val df_write = Seq(<br>    (&quot;100&quot;,&quot;Captain Marvel&quot;,&quot;Superhuman strength&quot;,&quot;2020-01-01&quot;,&quot;2015-01-01T13:51:39.340396Z&quot;),<br>    (&quot;101&quot;,&quot;Gamora&quot;,&quot;Elite combatant&quot;,&quot;2020-01-01&quot;,&quot;2015-01-01T12:14:58.597216Z&quot;),<br>    (&quot;102&quot;,&quot;Thor&quot;,&quot;Lord of Thunder&quot;,&quot;2020-01-01&quot;,&quot;2015-01-01T13:51:40.417052Z&quot;),<br>    (&quot;103&quot;,&quot;Scarlet Witch,Magic&quot;,&quot;2020-01-01&quot;,&quot;2015-01-01T13:51:40.519832Z&quot;),<br>    (&quot;104&quot;,&quot;Groot&quot;,&quot;I am Grrot&quot;,&quot;2020-01-02&quot;,&quot;2015-01-01T12:15:00.512679Z&quot;),<br>    (&quot;105&quot;,&quot;Loki&quot;,&quot;trickster&quot;,&quot;2020-01-02&quot;,&quot;2015-01-01T13:51:42.248818Z&quot;),<br>    (&quot;106&quot;,&quot;Hulk&quot;,&quot;Superhuman strength&quot;,&quot;2020-01-01,2015-01-01T12:14:58.597216Z&quot;),<br>    (&quot;107&quot;,&quot;Tony Stark&quot;,&quot;Genius,2020-01-01&quot;,&quot;2015-01-01T13:51:40.417052Z&quot;),<br>    (&quot;108&quot;,&quot;Peter parker&quot;,&quot;Spider man&quot;,&quot;2020-01-01&quot;,&quot;2015-01-01T13:51:40.519832Z&quot;),<br>    (&quot;109&quot;,&quot;Ant Man&quot;,&quot;Size shifting&quot;,&quot;2020-01-02&quot;,&quot;2015-01-01T12:15:00.512679Z&quot;))<br>    .toDF(&quot;user_id&quot;, &quot;user_name&quot;, &quot;powers&quot;, &quot;creation_time&quot;, &quot;last_login_time&quot;)<br>  <br>  //Create table <br>  df_write.write.mode(&quot;overwrite&quot;).format(&quot;delta&quot;).save(basePath)</pre><pre>    val deltaTable = DeltaTable.forPath(basePath)</pre><pre>    // Update...<br>    // predicate and update expressions using SQL formatted string</pre><pre>    deltaTable.updateExpr(<br>      &quot;user_id = &#39;101&#39;&quot;,<br>      Map((&quot;user_name&quot;) -&gt; (&quot;&#39;Thanos&#39;&quot;), (&quot;powers&quot; -&gt; &quot;&#39;destruction&#39;&quot;), (&quot;creation_time&quot; -&gt; &quot;&#39;2021-01-01&#39;&quot;)))</pre><pre>    // predicate using Spark SQL functions and implicits<br>    deltaTable.update(<br>      col(&quot;user_id&quot;) === &quot;108&quot;,<br>      Map(&quot;powers&quot; -&gt; lit(&quot;reflexes&quot;)));</pre><pre>    // Delete record matching below predicate<br>    deltaTable.delete(condition = expr(&quot;user_id == &#39;104&#39;&quot;))</pre><pre>    import spark.implicits._<br>    val newDF = Seq(<br>      (&quot;105&quot;, &quot;Black Panther&quot;, &quot;Vibranium&quot;, &quot;2020-02-28&quot;, &quot;2020-02-28T13:51:39.340396Z&quot;),<br>      (&quot;110&quot;, &quot;Wonder Woman&quot;, &quot;Hand to Hand Combat&quot;, &quot;2021-01-18&quot;, &quot;2000-01-01T13:51:39.340396Z&quot;))<br>      .toDF(&quot;user_id&quot;, &quot;user_name&quot;, &quot;powers&quot;, &quot;creation_time&quot;, &quot;last_login_time&quot;)</pre><pre>    //Upsert operation<br>    deltaTable.as(&quot;oldData&quot;)<br>      .merge(<br>        newDF.as(&quot;newData&quot;),<br>        &quot;oldData.user_id = newData.user_id&quot;)<br>      .whenMatched<br>      .update(Map(<br>        &quot;user_name&quot; -&gt; col(&quot;newData.user_name&quot;),<br>        &quot;powers&quot; -&gt; col(&quot;newData.powers&quot;)<br>      ))<br>      .whenNotMatched<br>      .insert(Map(<br>        &quot;user_id&quot; -&gt; col(&quot;newData.user_id&quot;),<br>        &quot;user_name&quot; -&gt; col(&quot;newData.user_name&quot;),<br>        &quot;powers&quot; -&gt; col(&quot;newData.powers&quot;),<br>        &quot;creation_time&quot; -&gt; col(&quot;newData.last_login_time&quot;),<br>        &quot;last_login_time&quot; -&gt; col(&quot;newData.last_login_time&quot;)<br>      ))<br>      .execute()</pre><pre>  deltaTable.toDF.show()<br>  val finalSnapshot = spark.read.format(&quot;delta&quot;).load(basePath)<br>  println(&quot; Final snapshot -------------&quot;)<br>  finalSnapshot.show()<br>  /*<br>  Final snapshot of your Delta table:</pre><pre>+-------+--------------+-------------------+--------------------+--------------------+<br>|user_id|     user_name|             powers|       creation_time|     last_login_time|<br>+-------+--------------+-------------------+--------------------+--------------------+<br>|    100|Captain Marvel|Superhuman strength|          2020-01-01|2015-01-01T13:51:...|<br>|    106|          Hulk|Superhuman strength|          2020-01-01|2015-01-01T12:14:...|<br>|    105| Black Panther|          Vibranium|          2020-01-02|2015-01-01T13:51:...|<br>|    109|       Ant Man|      Size shifting|          2020-01-02|2015-01-01T12:15:...|<br>|    108|  Peter parker|           reflexes|          2020-01-01|2015-01-01T13:51:...|<br>|    102|          Thor|    Lord of Thunder|          2020-01-01|2015-01-01T13:51:...|<br>|    103| Scarlet Witch|              Magic|          2020-01-01|2015-01-01T13:51:...|<br>|    101|        Thanos|        destruction|          2021-01-01|2015-01-01T12:14:...|<br>|    107|    Tony Stark|             Genius|          2020-01-01|2015-01-01T13:51:...|<br>|    110|  Wonder Woman|Hand to Hand Combat|2000-01-01T13:51:...|2000-01-01T13:51:...|<br>+-------+--------------+-------------------+--------------------+--------------------+<br> */<br>  <br>  val finalTable = DeltaTable.forPath(spark, basePath)</pre><pre>  val fullHistoryDF = finalTable.history()    // get the full history of the table<br>  <br>  fullHistoryDF.show()</pre><pre>  /*<br>  Metadata about the list of operations performed on target table<br>+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+<br>|version|          timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend|    operationMetrics|<br>+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+<br>|      4|2021-01-29 16:35:36|  null|    null|    MERGE|[predicate -&gt; (ol...|null|    null|     null|          3|          null|        false|[numTargetRowsCop...|<br>|      3|2021-01-29 16:35:08|  null|    null|   DELETE|[predicate -&gt; [&quot;(...|null|    null|     null|          2|          null|        false|[numRemovedFiles ...|<br>|      2|2021-01-29 16:34:55|  null|    null|   UPDATE|[predicate -&gt; (us...|null|    null|     null|          1|          null|        false|[numRemovedFiles ...|<br>|      1|2021-01-29 16:34:42|  null|    null|   UPDATE|[predicate -&gt; (us...|null|    null|     null|          0|          null|        false|[numRemovedFiles ...|<br>|      0|2021-01-29 16:34:25|  null|    null|    WRITE|[mode -&gt; Overwrit...|null|    null|     null|       null|          null|        false|[numFiles -&gt; 1, n...|<br>+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+--------------------+</pre><pre>   */<br>  <br>  val dfBeforeMerge = spark.read<br>    .format(&quot;delta&quot;)<br>    .option(&quot;versionAsOf&quot;, &quot;2&quot;)<br>    .load(basePath)<br>  dfBeforeMerge.show()</pre><pre>  /*<br>Snapshot of the table after operation in version 2, it shows how we can go back in time to get the earlier version<br>of target table and revert it if wrong operation is performed.<br>+-------+--------------+-------------------+-------------+--------------------+<br>|user_id|     user_name|             powers|creation_time|     last_login_time|<br>+-------+--------------+-------------------+-------------+--------------------+<br>|    100|Captain Marvel|Superhuman strength|   2020-01-01|2015-01-01T13:51:...|<br>|    101|        Thanos|        destruction|   2021-01-01|2015-01-01T12:14:...|<br>|    102|          Thor|    Lord of Thunder|   2020-01-01|2015-01-01T13:51:...|<br>|    103| Scarlet Witch|              Magic|   2020-01-01|2015-01-01T13:51:...|<br>|    104|         Groot|         I am Grrot|   2020-01-02|2015-01-01T12:15:...|<br>|    105|          Loki|          trickster|   2020-01-02|2015-01-01T13:51:...|<br>|    106|          Hulk|Superhuman strength|   2020-01-01|2015-01-01T12:14:...|<br>|    107|    Tony Stark|             Genius|   2020-01-01|2015-01-01T13:51:...|<br>|    108|  Peter parker|           reflexes|   2020-01-01|2015-01-01T13:51:...|<br>|    109|       Ant Man|      Size shifting|   2020-01-02|2015-01-01T12:15:...|<br>+-------+--------------+-------------------+-------------+--------------------+</pre><pre>   */<br>  }<br>}</pre><h4>References:</h4><ul><li><a href="https://docs.delta.io/latest/delta-storage.html#amazon-s3">https://docs.delta.io/latest/delta-storage.html#amazon-s3</a></li><li><a href="https://docs.delta.io/0.7.0/quick-start.html">https://docs.delta.io/0.7.0/quick-start.html</a></li><li><a href="https://docs.delta.io/latest/delta-utility.html#language-scala">https://docs.delta.io/latest/delta-utility.html#language-scala</a></li><li><a href="https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html">https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a66e61dea44" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Offline Batch Inferencing with Ray Datasets]]></title>
            <link>https://anuprawka.medium.com/offline-batch-inferencing-with-ray-datasets-3cd1b3d847bd?source=rss-16e51b79c6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd1b3d847bd</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[batch-inferences]]></category>
            <category><![CDATA[ray-framework]]></category>
            <dc:creator><![CDATA[Anup Rawka]]></dc:creator>
            <pubDate>Sat, 16 Dec 2023 09:42:05 GMT</pubDate>
            <atom:updated>2023-12-19T01:29:52.949Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction:</strong></h3><p>In this blog we will look into the how we can use Ray Datasets to build offline scalable batch inferencing pipeline . We will particularly focus on:</p><ul><li>Understand common design patterns for distributed batch inference.</li><li>How to implement scalable batch inferencing using Ray Dataset(BatchPredictor ).</li><li>How to configure the batch size.</li><li>How to configure scaling resources.</li></ul><h3>Why Ray?</h3><ul><li>Easily parallelize and distribute workloads across multiple nodes and GPUs.</li><li>Easily configure to use spot instances (or preemptible instances).</li><li>Pythonic distributed computing primitives for parallelizing and scaling Python applications.</li><li>Friction less transition from Research to Production. <strong>write once. Run everywhere.</strong></li><li>Encourages iterative development and Debugging.</li><li>Fault tolerance–Ensuring tasks complete regardless of inevitable points of failure.</li><li>Auto-scaling–Adjusting the number of resources allocated to dynamic demand.</li><li>It provides easy integration with Airflow for seamlessly moving code from notebook to production.</li></ul><h3>Batch Inference:</h3><p>Batch inference generates predictions over a large number of input data when immediate response is not required or feasible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/0*bk_9Y1Oxpiqqk0EY.png" /><figcaption>Batch processing (photo from Ray documentation)</figcaption></figure><h3><strong>Batch Inference using Ray</strong></h3><p>Ray provides three architectures for performing distributed batch inference:</p><ol><li><strong>Stateless inference with Ray Tasks.</strong></li></ol><p><a href="https://docs.ray.io/en/latest/ray-core/tasks.html#ray-remote-functions">Ray Tasks</a> offer an easy way to distribute inference across a compute cluster. Tasks are Python functions that execute remotely in the cluster, allowing multiple processes to work on different tasks concurrently. Model is replicated across all the CPUs and GPUs and each inference task is executed independently. Since tasks do not store or modify any internal state,<em> it is stateless</em>.</p><p><strong>2. Stateful inference with </strong><a href="https://docs.ray.io/en/latest/ray-core/actors.html#actors"><strong>Ray Actors</strong></a><strong>:</strong></p><p><a href="https://docs.ray.io/en/latest/ray-core/actors.html">Ray Actors</a> are <em>stateful objects</em>, meaning they maintain an internal state. They internally follows the fundamentals of <a href="https://en.wikipedia.org/wiki/Actor_model">Actor Model</a> and they are the building blocks of Ray framework. Because each actor keeps track of an internal state, it can be reused for inference on multiple batches.</p><p>As we are using the Ray Core primitives (Tasks, Actors), it offers a fine-grained control over the behavior of a distributed application and user has to specify <em>how</em> inference should be executed.</p><p><strong>3. Ray AIR </strong><a href="https://docs.ray.io/en/latest/ray-air/predictors.html#batch-prediction"><strong>BatchPredictor</strong></a><strong>:</strong></p><p>Its a utility for large-scale and distributed batch inferencing. It takes a Checkpoint (saved trained model) and a Predictor (class that defined inference logic) to generate predictions on a Ray Dataset. It provides high-level abstractions over Ray tasks and removes the complexity of the challenging aspects of scaling batch inference in exchange for less direct control over the way Ray distributes.</p><p>It has below out-of-the-box features:</p><ul><li>supports various predictors like <a href="https://docs.ray.io/en/latest/ray-air/api/doc/ray.train.torch.TorchPredictor.html#ray.train.torch.TorchPredictor">TorchPredictor</a>, <a href="https://docs.ray.io/en/latest/ray-air/api/doc/ray.train.huggingface.HuggingFacePredictor.html#ray.train.huggingface.HuggingFacePredictor">HuggingFacePredictor</a> or <a href="https://docs.ray.io/en/latest/ray-air/api/doc/ray.train.xgboost.XGBoostPredictor.html#ray.train.xgboost.XGBoostPredictor">XGBoostPredictor</a>)</li><li>it handles framework-native batch conversions</li><li>it has options to resume operations from AIR checkpoint to prediction, selection / keep columns, etc</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4RhIFKwTWTmfkF4CDeEJbw.png" /><figcaption>Source: Ray official documentation</figcaption></figure><p>We will look into how we can use Ray datasets map_batchesapi to build a offline batch inferencing pipeline and also how we can scale it.</p><h3><strong>Implementation:</strong></h3><p>In the below example we will use Ray Dataset API to load source images. <a href="https://docs.ray.io/en/latest/data/dataset.html#datasets">Ray Datasets</a> are the standard way to load and exchange data in Ray. It loads and preprocess data for parallel compute, internally handling operations like batching, pipelining, autoscaling the actor pool, and memory management. Ray Data offers an efficient and scalable solution for batch inference, providing faster execution and cost-effectiveness for deep learning applications. It is well suited for offline batch inference tasks.</p><p>We are using Davis dataset as our data source, <a href="https://davischallenge.org/davis2017/code.html">you can download the dataset from here.</a> We will build a simple image out-painting task using “stable-diffusion-inpainting” model.</p><figure><img alt="Image out-painting flow" src="https://cdn-images-1.medium.com/max/1024/1*E7LWwghniv_9mkM8k8_c2Q.png" /></figure><p>Dependencies:</p><p>You can manually install each of them on your node or put everything into a requirements file as shown below.</p><pre>#requirements.txt<br>urllib3==1.26.6<br>diffusers<br>pillow<br>pyarrow<br>pandas<br>transformers<br>torch<br>accelerate<br>ray[default]<br>ray[data]</pre><p>We can run Ray in a multi node setup or single node setup, for this example we will use single node setup.</p><pre>import subprocess<br>import ray<br>import torch<br>import numpy as np<br>import PIL<br>from PIL import Image<br>from diffusers import StableDiffusionInpaintPipeline<br>import os<br>from ray.air import session<br>from pathlib import Path<br><br># As we are starting Ray on single machine. <br># Calling ray.init() starts a local Ray instance on your machine and it becomes the “head node”.<br>if ray.is_initialized:<br>    ray.shutdown()<br><br>ray.init()<br><br>path = [&quot;/home/ec2-user/in/&quot;]<br><br># Define a Predictor class for inference.<br># Use a class to initialize the model just once in `__init__`<br># and re-use it for inference across multiple batches.<br>class StableDiffusionPipelineClass:<br># You can change the prompt as per your datasets.<br>    def __init__(self, model_name=&quot;runwayml/stable-diffusion-inpainting&quot;,<br>                 prompt=&quot;high resolution, nice background, realistic&quot;, ):<br>        self.device = torch.device(&#39;cuda&#39;)<br>        self.model = StableDiffusionInpaintPipeline.from_pretrained(model_name, local_files_only=True)<br>        self.prompt = prompt<br>        self.batch_counter = 0<br><br>    def __call__(self, batch):<br>        # creating mask, this mask tells where to paint. <br>        # You can play around with the patch size and prompt<br>        left_patch = np.ones((1024, 51, 3), dtype=np.uint8) * 255<br>        right_patch = np.ones((1024, 51, 3), dtype=np.uint8) * 255<br>        middle_patch = np.ones((1024, 922, 3), dtype=np.uint8) * 1<br>        img = np.concatenate((left_patch, middle_patch, right_patch), axis=1)<br>        mask_image = PIL.Image.fromarray(img, &#39;RGB&#39;)<br>        # we are looping through each image in the batch and using its path to create result<br>        # image file name. This will help us in mapping both source and target image together. <br>        for _img, _paths in zip(batch[&#39;image&#39;], batch[&#39;path&#39;]):<br>            file_name = _paths.split(&quot;in&quot;)[1].replace(&quot;/&quot;, &quot;_&quot;)<br>            image = Image.fromarray(_img).resize((512, 512))<br>            model = self.model.to(self.device)<br>            img = model(self.prompt,<br>                        image=image,<br>                        mask_image=mask_image,<br>                        guidance_scale=4.5,<br>                        num_inference_steps=15,<br>                        negative_prompt=&quot;humans, people, faces, low resolution&quot;, ).images[0].resize((512, 512))<br>        img.save(&quot;/home/ec2-user/out/&quot; + file_name)<br><br>        self.batch_counter += 1<br>        return batch<br># Set the batch size to as large possible without running out of memory. <br># If you encounter out-of-memory errors, decreasing batch_size may help.<br>BATCH_SIZE = 10<br># reading all the images in Ray Dataset, when the include_paths argument is set to true<br># ray will read image along with its complete path. <br>ds = ray.data.read_images(&quot;/home/ec2-user/in/&quot;, include_paths=True)<br><br>result_images = ds.map_batches(<br>    StableDiffusionPipelineClass,<br>    # Set the ActorPool size to the number of GPUs in your cluster <br>    # if you are using 1 gpu per actor pool<br>    # In Ray Datasets, transformations can either be carried out by Ray Tasks<br>    # or Actors. While the default compute strategy uses Ray Tasks, <br>    # you can specify an ActorPoolStrategy which dynamically autoscales <br>    # the number of actors between a min and max size to carry out the transforms.<br>    compute=ray.data.ActorPoolStrategy(size=1), <br>    num_gpus=1,  # Specifies how many GPU&#39;s per actior pool strategy will use.<br>    # num_cpus=1,  # Specify N CPUs per model replica.<br>    batch_size=BATCH_SIZE  <br>)<br># Either you can call `materialize()` or call updated_images.take(limit=1) to trigger the inference<br>updated_images.materialize()</pre><h3>Conclusion</h3><p>In this post we have quickly demonstrated how we can use Ray AIR high-level api to build a scalable batch inference pipeline.</p><h3>References:</h3><ol><li><a href="https://docs.ray.io/en/latest/serve/tutorials/batch.html">Official Ray docs and training material</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd1b3d847bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Akka Streams: Our Journey for Fast Data Processing]]></title>
            <link>https://anuprawka.medium.com/akka-streams-our-journey-for-fast-data-processing-30cff2ed941c?source=rss-16e51b79c6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/30cff2ed941c</guid>
            <category><![CDATA[akka-streams]]></category>
            <category><![CDATA[big-data-streaming]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[akka]]></category>
            <dc:creator><![CDATA[Anup Rawka]]></dc:creator>
            <pubDate>Wed, 17 Feb 2021 04:06:08 GMT</pubDate>
            <atom:updated>2024-03-30T10:34:18.032Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Akka Streams: Our Journey for Fast Data Processing and Tech Primer</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*DdqqhNbXaRNtUYqo0AI2Gg.png" /><figcaption>Akka Streams</figcaption></figure><p>In this blog I will walk you through the basics of Akka Streams, along with that we will also dive through the journey of how it helped us in creating a stable Data Ingestion pipeline. I have redacted company and project information from this blog for security reasons. At the end of the blog I have shared github link for Akka Streams Tech Primer, I would recommend to go through the blog before jumping on to the primer for better understanding.</p><h4><strong>Introduction:</strong></h4><p><a href="https://doc.akka.io/docs/akka/current/stream/stream-flows-and-basics.html#core-concepts">Akka Streams is a toolkit for event based processing of streaming data using bounded buffer space.</a> It is an actor based implementation of reactive streams specifications. It not only provides out of the box back-pressuring but also promises type safety at every stage of the streamed pipeline.</p><blockquote><a href="https://developer.lightbend.com/docs/akka-platform-guide/concepts/reactive-streams.html">The main goal of Reactive Streams is to govern the exchange of stream data across an asynchronous boundary (i.e. between producer and consumer in below Flow) — think of it as passing elements on to another thread or thread-pool while ensuring that the receiving side is not forced to buffer arbitrary amounts of data (example, more than N messages). In other words, back pressure is an integral part of this model in order to allow the queues which mediate between threads to be bounded.</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*vlr3FwBPwxgq5Zmf" /><figcaption>Akka Streams Bounded Buffered communication</figcaption></figure><p><strong>How Akka Streams helped us in creating stable and high-performance data ingestion pipeline:</strong></p><h4><strong>The challenge:</strong></h4><p>Create a Data Ingestion Client service (ICS) which can:</p><ul><li>Handle never ending, volume-variable (sometimes in bursts, other times in trickle) flows that run for days without needing any restart.</li><li>Not only process big volumes of data but also process it fast.</li><li>Support both pulling and pushing of messages from diverse sources (Kafka, Flat file, CDC).</li><li>Take data in a variety of formats (Avro, XML, Json, DSV, etc) and convert it to desired format (Avro) if needed.</li><li>Encrypt the data after some validations an</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wb5wyA3MLHrmfWCG" /><figcaption><strong>Ingestion Client Service</strong></figcaption></figure><h4><strong>The Solution:<br>First attempt: Increasing concurrency with Akka Actors</strong></h4><h4><strong>The Basics:</strong></h4><p>Akka is a toolkit for building highly concurrent, distributed and resilient message-driven applications and Actors are basically concurrent processes that communicate by exchanging messages.</p><h4><strong>Implementation:</strong></h4><p>To ensure our new application could scale as our data ingest rates grew, we opted to use Akka Actors, a framework that utilizes the Actor model as its core concurrency abstraction. We chose this because it was proven to scale and we loved the simplicity of the API. This worked well for the cases where data was being pushed to us through different channels (e.g. Kafka) and in our initial tests we reached a performance of 50k events/min (epm) with 10 threads from single threaded flow of 20K epm (2.5KB.event). While the basic actor model provides a simple, scalable concurrency paradigm you still need to think about how work can pile up in your system.</p><p>When data was being pulled (e.g. from File), the actors responsible for pulling the data need to work in tandem with the rest of the actors. However the sole way to interact with an actor is via a message, which means slow actors can have messages queued up in memory while fast actors continue to send them messages creating hot spots. As the load and volume increased we started seeing GC pauses, causing uneven rates of processing for various stages and eventually bringing the application to halt. This leads to unpredictable, over/under-utilization of resources across different components and sometimes eventually crashing the JVM.</p><h4><strong>Second Attempt: Akka -Streams to the rescue The Basics:</strong></h4><p>Akka Streams is a toolkit built on the top of actor-model for processing sequences of elements using bounded buffer space. It removes the low level plumbing that developers have to do to handle streams in case of Akka. Streams are Reactive and Asynchronous.</p><p>Akka’s Streams, that are built from the concepts of Reactive Streams, provided following benefits for the kind of problem we were facing.</p><h4><strong>Back-Pressure</strong></h4><p>Back pressure capabilities are provided transparently and out-of-the-box(back pressuring is the ability to say “hey slow down!”). Hence, where the target system is slower than the source system it makes the pipeline flow controlled and throttled to a maximum and not larger than the survivable speed.</p><h4><strong>Composition</strong></h4><p>Ability to compose small components into any arbitrarily complex pipeline. We can break our big problem into smaller blocks that can be then composed (and reused) to form the solution.</p><h4><strong>Type Safety</strong></h4><p>All the Akka-Streams building blocks are completely type safe: if a component produces/processes/takes a type A, it will be impossible to use a type B by mistake. Compiling type checks dramatically increase the correctness of our code and reduce the need for additional tests. The compiler works more so you do less.</p><p>Here, when we talk of ​<strong><em>building blocks </em></strong>i​n Akka Streams we are referring mainly to the following three:</p><h4><strong>Sources:</strong></h4><blockquote><a href="https://doc.akka.io/docs/akka/current/stream/stream-flows-and-basics.html">An operator with ​<em>exactly one output</em>,​ emitting data elements whenever downstream operators are ready to receive them.</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/0*AMx8P3OZtjyey57Q" /><figcaption>Source</figcaption></figure><h4><strong>Flows:</strong></h4><blockquote><a href="https://doc.akka.io/docs/akka/current/stream/stream-flows-and-basics.html">An operator which has ​<em>exactly one input and output</em>,​ which connects its upstream and downstream by transforming the data elements flowing through it.</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/0*_rJdEvKBQRg5Fy80" /><figcaption>Flow</figcaption></figure><h4><strong>Sinks:</strong></h4><blockquote><a href="https://doc.akka.io/docs/akka/current/stream/stream-flows-and-basics.html">An Operator with​ <em>exactly one input </em>​, requesting and accepting data elements possibly slowing down the upstream producer of elements.</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/0*kd0BPpgC1yQeKZl2" /><figcaption>Sink</figcaption></figure><h4><strong>DSL (in the Scala version):</strong></h4><p>We can work directly with the low level Akka-Streams API to create our graphs, and that will give us complete control over every aspect of the stream. However, Akka-Streams also provides a high-level DSL, that not only provides for a much simpler and idiomatic way of constructing streams, but it is built in such a way that allows us to translate designs into code.</p><p>For example, let’s assume we need to build a simple encryption flow that will take clear text messages, encrypt them according to a certain logic and publish them to a Kafka topic. This processing stream could look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/0*OyO_wNVHxh3CPjGI" /></figure><p>And with the provide DSL the actual code would look like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*EidmSwS2RRT69lUV" /></figure><p>And while the example above is arguably trivial, we can go as elaborated and sophisticated as we want our design and the DSL will still allow us to express it in a simple way.</p><p><strong>Implementation:</strong></p><p>Using the Graph DSL based approach which allows us to segregate logical processing blocks which can be wired together in the end and ensuring type-safety over stage boundaries, we created a separate stage for every challenge listed above and wired them together to create the final workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/0*6twjrWbDVvFR8Ior" /></figure><p>The back pressured communication ensures that neither the producer or consumer end of the workflow can get overwhelmed with the order. This makes all the stages within the ICS work in tandem while removing the bottlenecks and points of contention from it. To keep resource utilization as efficient as possible, Akka Streams enables elastic spawning of actor instances in response to varying load, which then elastically scale back when not needed.</p><p><strong>Results (Apps max memory allocation: 5 GB):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*Etnn8nVXjw3i7I-8ktwC7A.png" /></figure><p><strong>● Improved the performance by multiple folds with no concurrent-programming and distributed-systems problems:<br> </strong>Improved application performance by over 10 folds (25k to 320k, there were other factors along with technology shift which we were able to identify because of the simplicity added by Akka Streams.)</p><p>● <strong>Avoided the risk of expensive downtime in production:<br> </strong>We did the stress testing on Streams based ICS with a volume of 140M records (150GB) and saw consistent throughput (variance of ~10%) with no GC pauses throughout the test.</p><p>● <strong>Improved CPU Utilization:<br> </strong>With Streams the CPU utilization is consistently around 80%.</p><p><strong>● Able to Scale both elastically when needed:</strong></p><p>The Reactive nature of Akka Streams allows us to grow both horizontally and vertically.</p><h3>Tech Primer:</h3><p>Below projects walk you through the steps to build a basic flow with a working example. It will help you in understanding the core concepts of Akka Stream.</p><p><a href="https://github.com/AnupRawka/AkkaStreamsTechPrimer">AnupRawka/AkkaStreamsTechPrimer</a></p><h3><strong>References:</strong></h3><ul><li>Official Akka Streams docs: <a href="https://doc.akka.io/docs/akka/current/stream/index.html">https://doc.akka.io/docs/akka/current/stream/index.html</a></li><li>Maximizing throughput for Akka Streams: <a href="https://blog.colinbreck.com/maximizing-throughput-for-akka-streams/">https://blog.colinbreck.com/maximizing-throughput-for-akka-streams/</a></li><li>Reactive Streams: <a href="https://www.reactive-streams.org/">https://www.reactive-streams.org/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30cff2ed941c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Selecting the Right streaming Engine for your Data Pipeline]]></title>
            <link>https://anuprawka.medium.com/selecting-the-right-streaming-engine-for-your-data-pipeline-e87d76c9dfca?source=rss-16e51b79c6c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e87d76c9dfca</guid>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[akka-streams]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[real-time-streaming-data]]></category>
            <dc:creator><![CDATA[Anup Rawka]]></dc:creator>
            <pubDate>Wed, 26 Aug 2020 00:50:09 GMT</pubDate>
            <atom:updated>2021-03-19T02:56:39.399Z</atom:updated>
            <content:encoded><![CDATA[<p>Until recently most of the data warehouses and data lakes were batch oriented where data was captured in file systems or/and databases and processed in batches. However, now the current technology advancement it is a competitive disadvantage to rely completely on batch-mode. Hence big-data systems are continuously evolving to be more stream oriented where data is processed as it arrives to get the competitive edge over peers.</p><p>Today, there are a number of Open source Streaming frameworks available in the market and almost all of them have been developed in the last few years. As everything is evolving quickly, it’s so easy to get confused and swamped while selecting the right framework for your requirements. These systems need to process infinite data streams continuously and hence they need to be resilient, highly available and scalable as the volume of data grows.</p><p>Let’s briefly go through the various categories of streaming solutions available:</p><p>The streaming processing can be divided into 2 broad categories based on the way the incoming data in processed:</p><ol><li><strong>Micro batching:</strong></li></ol><p>Micro batching is a practice of collecting data in small batches for processing to achieve some of the performance of traditional batch processing without actually increasing latency.</p><p>Example: Spark Streaming</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*qxY-AqZ6gCnr4D-jxWRNwg.png" /><figcaption><strong>Micro batching</strong></figcaption></figure><p>2.<strong> Native Streaming:</strong></p><p>In native stream processing we process each record as soon as it arrives without actually waiting for others.</p><p>Example: Akka Streams, Flink, Apache Storm, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*8ZptcpdZYsdghnM0UOmqHQ.png" /><figcaption><strong>Native Streaming</strong></figcaption></figure><p>The family of Streaming technologies can be broadly divided into 2 major categories:</p><ol><li><strong>Streaming Frameworks:</strong></li></ol><p>Streaming frameworks run as a distributed service to which we have to submit jobs. They can be scaled across clusters easily. Integration with microservices usually requires that you run the framework separately from the microservices and exchange data through Kafka or other means.</p><p>e.g. Apache Spark Structured Streaming, Apache Flink, Kafka connect, etc.</p><p>2. <strong>Streaming Libraries:</strong></p><p>Streaming libraries can be embedded in your microservices, providing greater flexibility in how you integrate analytics with other processing, with very low latency and lower overhead than streaming frameworks.</p><p>E.g. Akka streams, Kafka Streams, AlpAkka, etc.</p><p>Let’s look into the features to consider while selecting your Streaming Engine:</p><ol><li>What is the acceptable Latency?</li></ol><p>a. Do you need to process records within milliseconds? (E.g. Payment processing, Fraud detection, etc)</p><p>b. Is latency in the range of minutes acceptable? (Logs analysis)</p><p>2) What is the data volume to be processed per unit time?</p><p>Data Pipelines nowadays needs to process Billions of records everyday and if it can happen one record at a time over a long period of time then any solution can be made to work however if we are processing Twitter firehouse then we surely need to look towards scaling through distributed processing.</p><p>3) What kind of Data processing is needed:</p><p>If you are simply lifting and shifting your data or running simple ETL logic (filtering and record transformation) then there are many options to go with. If you need to perform complex joins or group by queries then it will constrain your choices.</p><p>4) Preferred Application architecture:</p><p>Libraries can be easily integrated into your microservices providing greater flexibility and control. Frameworks provide very rich analytics and easy scalability as they run jobs on distributed services.</p><p>5) How strong and dynamic the team around the project is:</p><p>Adding new technology into the tech stack is always challenging. How easily people contributing to the project can adapt to the tool/technology and availability of CI/CD processes weighs a lot in decision making.</p><p>Let’s explore some of the best breed streaming solutions available in market based on the above criteria:</p><ol><li><strong>Streaming Frameworks:</strong></li></ol><p>Pick streaming frameworks when your dataset is large enough to require distributed processing. These frameworks handle very high volume of data and can do complex transformations with relatively low latency. However integrating these frameworks with microservices requires exchanging data through some messaging service like Kafka/MQ as these are standalone services.</p><ol><li><strong>Apache Spark Streaming (Structured streaming):</strong></li></ol><p>i. Use it if you are already using it for your batch processing. This transition to spark streaming from spark batch processing is easy.</p><p>ii. Use it if your transformational logic is mainly in SQL or you want to integrate it with Machine learning systems.</p><p><strong>Apache Flink:</strong> Use it if you are not on Spark and need low latency with all the high level APIs that Spark offers. Apache Flink is fairly new in the game and although there is a rise in adoption, its open source community is not as big as Spark.</p><p><strong>Apache Storm:</strong> Use it for streaming analytics, especially when you are using languages like R which are not supported by Spark. Storm is highly scalable and provides lower latency than Spark Streaming.</p><p><strong>Kafka Connect: </strong>Use it for data streaming between Kafka and other systems. It comes with out of the box offset management and auto-recovery features and connectors to a variety of systems.</p><p>2. <strong>Streaming Libraries:</strong><br> Streaming libraries can be directly integrated within your microservices and hence provides low latency. Processing is always faster if your transformation is happening in the same process rather than in separate service. It removes the extra overhead of going to Kafka or other messaging services. Libraries provide better control and flexibility in how applications are deployed and managed.</p><p>Pick them when data partitioning is not required (Volume is manageable within standalone application) or when you want to continue with your microservices based development pattern rather than supporting heterogeneous environments.</p><p><strong>a. Akka Streams:</strong></p><p>i. Use it when you want fine-grained control over your data. Akka Streams follows <a href="https://www.reactivemanifesto.org/">Reactive Manifesto </a>which defines core principles for Reactive programming. It provides an intuitive and safe way to do asynchronous, non-blocking backpressured stream processing.</p><p><strong>b. Kafka Streams:</strong></p><p>Use it when your source and target system is Kafka or you want to run SQL on data available in Kafka.</p><p>Note: Of course, you should also consider how these tools fit into your workloads, infrastructure, and requirements, you may find that the ideal solution is a mixture of the above suggestions. If you needs an hands-on example on<a href="https://anuprawka.medium.com/akka-streams-our-journey-for-fast-data-processing-30cff2ed941c"> Akka stream you can check out my medium blog on it</a>.</p><p><strong>References:</strong></p><ul><li><a href="https://stackoverflow.com/questions/53727872/alpakka-kafka-vs-kafka-streams">https://stackoverflow.com/questions/53727872/alpakka-kafka-vs-kafka-streams</a></li><li><a href="https://medium.com/@chandanbaranwal/spark-streaming-vs-flink-vs-storm-vs-kafka-streams-vs-samza-choose-your-stream-processing-91ea3f04675b">https://medium.com/@chandanbaranwal/spark-streaming-vs-flink-vs-storm-vs-kafka-streams-vs-samza-choose-your-stream-processing-91ea3f04675b</a></li><li><a href="https://stackoverflow.com/questions/37738102/using-apache-flink-for-data-streaming">https://stackoverflow.com/questions/37738102/using-apache-flink-for-data-streaming</a></li><li><a href="https://www.upsolver.com/blog/batch-stream-a-cheat-sheet">https://www.upsolver.com/blog/batch-stream-a-cheat-sheet</a></li><li><a href="https://www.lightbend.com/blog/cloud-native-streaming-data-with-akka-streams-kafka-steams">https://www.lightbend.com/blog/cloud-native-streaming-data-with-akka-streams-kafka-steams</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e87d76c9dfca" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>