<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sumitkrsharma on Medium]]></title>
        <description><![CDATA[Stories by Sumitkrsharma on Medium]]></description>
        <link>https://medium.com/@sumitkrsharma-ai?source=rss-246685f156aa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*M_CwJW15IRdFGqcf</url>
            <title>Stories by Sumitkrsharma on Medium</title>
            <link>https://medium.com/@sumitkrsharma-ai?source=rss-246685f156aa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:47:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sumitkrsharma-ai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agentic Workflow Compilers – Making AI Agents reliable and executable workflows]]></title>
            <link>https://sumitkrsharma-ai.medium.com/agentic-workflow-compilers-making-ai-agents-reliable-and-executable-workflows-6c553eead6b3?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c553eead6b3</guid>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 04:44:23 GMT</pubDate>
            <atom:updated>2026-11-03T04:44:23.525Z</atom:updated>
            <content:encoded><![CDATA[<p>Assume a manufacturing unit that receives an urgent order to produce 5,000 components, inspect their quality, validation of inventory, logistics management , and on time delivery everything within 30 hours. <br>A highly productive employee could handle this task manually. They would check inventory, contact the production team, quality team, calculate supply schedules, and resolution of unexpected events as well. But if that unit receives 100,000 such orders every day, depending upon one employee for decision of every next stage becomes slow, expensive, and inconsistent. Now consider a system that converts each order into an executable plan. That system analyzes the required tasks, determines which tasks can run simultaneously, checks dependencies, assigns the stages along with tools, handles problems, and monitors the complete pipeline. Human supervisors intervene when a decision is risky or ambiguous. This is the primary intuition behind an “<strong>Agentic Workflow Compiler</strong>”. <br>In place of allowing an AI agent to autonomously improvise every action at runtime, the system translates a high-level goal into a structured workflow that can be validated, optimized, and then executed. For example, an insurance company develops this type of system for claims handling where an AI-powered insurance claims system could receive the request: “<em>Investigate this motor insurance claim, verify the policy, inspect the submitted evidence, estimate the potential loss, and prepare a recommendation for an authorized claims officer.” </em>A traditional AI chatbot might explain how to investigate the claim. A basic autonomous agent might call several tools in sequence. But an agentic workflow compiler aims to transform the request into an explicit execution plan with dependencies, validation rules, retry policies, and approval gates. The distinction matters because generating a plausible plan is not the same as executing that plan reliably.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egPh-23SMbzVGrvzHcg5og.png" /></figure><h4>What is an Agentic Workflow Compiler?</h4><p>An Agentic Workflow Compiler is an agentic architecture that transforms a natural-language objective into a structured representation of series of agent actions, dependencies, tools, constraints, and control flow, then validates and prepares that representation for execution by an orchestration runtime.<br>This describes an emerging architectural pattern rather than one universally standardized compiler or a single mandatory technology stack.<br><strong>Agentic workflow compiler combines ideas from three established areas:</strong><br>•	Compiler design: That translates a high-level representation into a lower-level executable form.<br>•	Workflow management: Coordinate tasks, tools, dependencies, retries, and execution stage.<br>•	Agentic AI: use LLM models and other decision-making components to interpret goals, select actions, and handle situations requiring reasoning.<br>A useful conceptual pipeline is:<br>1. <strong>User goal</strong> — natural-language objective with constraints.<br>2. <strong>Planning and intent extraction</strong> — identify tasks, tools, conditions, and goal.<br>3. <strong>Intermediate representation (IR)</strong> — a designed workflow graph along with tools and contracts.<br>4. <strong>Validation and optimization</strong> — check security, data types, dependencies, cost, and scalability.<br>5. <strong>Execution plan</strong> — a validated graph ready for a system manager at runtime.<br>6. <strong>Runtime and feedback</strong> — execute tools, record results, troubleshooting for failures, and request approval.<br>Agentic AI workflow compiler shares analogy with a traditional compiler, but it has a key additional challenge: few decisions depend on uncertain information discovered during runtime. For instance, a claim investigation may reveal at runtime that a document is missing. During execution system must decide whether to request the document, perform an alternative verification, or stop the process. The system may need to branch or be partially recompiled.<br>That is where the ‘agentic AI’ aspect becomes important component.</p><h4>Why do we need this architecture?</h4><p>Conventional AI automation works well when every step is known beforehand. Agentic AI works well when a process requires interpretation, planning, and adaptation. The hurdle is combining both without sacrificing reliability.<br>Existing orchestration frameworks already provide important building blocks. LangGraph supports stateful agent orchestration, persistence, and human intervention, while Temporal emphasises on durable execution, troubleshooting, retries, and workflow stage. An agentic workflow compiler can be built on top of such infrastructure rather than replacing it.<br>The motivation becomes clearer when workflows must satisfy several requirements simultaneously:<br>•	<strong>Reliability</strong>: a failed API call or response should not destroy the entire pipeline.<br>•	<strong>Predictability</strong>: critical operations should follow explicit rules and fallback for the human supervisor.<br>•	<strong>Efficiency</strong>: independent tasks should run parallelly when safe.<br>•	<strong>Governance</strong>: sensitive actions should require appropriate authorization.<br>•	<strong>Adaptability</strong>: the system should respond to new samples without abandoning its original goal.<br>•	<strong>Observability</strong>: developers should be able to determine what to run, why to run, and which result influenced the next decision.<br>The central design principle is to let AI interpret ambiguity while using explicit software contracts to control execution.</p><h4>Core components of an Agentic Workflow Compiler</h4><p>4.1. <strong>Goal parser and planner</strong> <br>Converts a request into simpler and smaller tasks, identifies the goals, extracts constraints, and proposes the tools required. An LLM can perform this step, but its output should be treated as an untrusted proposal until results were validated. <br>4.2. <strong>Intermediate representation (IR)</strong> <br>Stores the required pipeline in a machine-readable form. Each node represents the tasks, edges represent dependencies for connected tasks; metadata specifies inputs, outputs, permissions, and constraints. This is the equivalent of an intermediate representation in a traditional compiler design. <br>4.3. <strong>Type checker and validator</strong> <br>These components used to Verifies the required inputs, output types validation of dependencies , and prohibited operations are excluded. It should reject cycles when the selected execution model requires a DAG. <br>4.4. <strong>Optimizer and scheduler</strong> <br>This component chooses execution order, selects the independent tasks, calculates the cost and latency, and respects concurrency limits. Optimization must maintain the dependencies and business rules. <br>4.5. <strong>Execution runtime and feedback controller</strong> <br>Key components that manage the execution for approved operations, records outputs, handles timeouts and troubleshoots, requests the human supervisor when required, and routes failures to recovery logic.</p><p>4.6. <strong>Tool registry and permission system</strong><br>An agent must not be able to run arbitrary functions simply because its language model guides them. A tool registry describes each allowed operation, controlling the restricted ones.<br>The registry performs the role of the controller interface between the AI’s reasoning and the outside environment v. Authorization must be enforced by the executer and downstream systems, not merely by a prompt.</p><p>4.7. <strong>Execution ledger and state store</strong><br>The compiler defines what should happen next , excution must also remember what actually happened. For each task, a production execution ledger might store:<br>•	Workflow and task identifiers.<br>•	Input and output samples.<br>•	Execution status with their timestamps.<br>•	Retry attempts along with failure reasons.<br>•	Model and tool versions.<br>•	Approval records and authorization context.<br>Execution ledger supports troubleshooting, auditing, and rollbacks. Durable orchestration systems such as Temporal use persisted event histories to undo workflow progress after problems.</p><h4>The mathematics behind workflow compilation</h4><p>The most useful mathematical concepts are graph theory, constrained optimization, probabilistic planning, type systems, and feedback control.<br>5.1. <strong>Representing a workflow as a directed graph</strong><br>Let a workflow be represented by a directed graph:<br>G = (V, E)<br>where V = {v₁, v₂, …, vₙ} is the V set of tasks and E is the set of directed dependency edges, where (vᵢ, vⱼ) means task vⱼ stage depends on task vᵢ.<br>continuing our insurance example workflow contains these tasks:<br>•	v₁: retrieve the policy.<br>•	v₂: inspect claimed things evidences.<br>•	v₃: validate coverage mentioned in the policy.<br>•	v₄: estimate the loss based on the inspection and coverage results.<br>•	v₅: prepare a recommendation for review.<br>Then dependency edges be looks like:<br>E = {(v₁, v₃), (v₂, v₄), (v₃, v₄), (v₄, v₅)}<br>Policy retrieval and damage inspection can begin independently. Coverages validation stage must wait for the policy. Loss estimation stage requires inputs from both damage inspection and coverage validation. Then the recommendation comes last.<br>A valid ordering of tasks in dependency edges is called a topological ordering. For a directed acyclic graph (DAG), at least one such ordering exists identified using topological sorting . If the planner accidentally introduces a cycle, the compiler should reject the workflow as it trapped in an iterative-loop.</p><p>5.2. <strong>Optimizing latency<br></strong>Assuming dᵢ represent the execution duration of task i. For a sequential workflow compiler , total execution time is approximately:<br>T_seq = Σ dᵢ<br>For a workflow compiler with parallel execution, the ideal completion time is determined by the critical path: the longest dependency-constrained path through the graph.<br>T_parallel ≥ T_critical</p><p>In our example sequential execution takes 13 seconds if all tasks run one after another. With ideal parallel execution, policy retrieval and image inspection start together. Coverage validation then follows policy retrieval, and loss estimation waits for both branches.<br>T_parallel = max(2 + 3, 5) + 2 + 1 = 8 seconds<br>Speedup = 13 / 8 = 1.625<br>Time saved = (13 - 8) / 13 × 100 ≈ 38.46%<br>These figures assume the given task durations, no scheduling overhead, and no resource contention. Real performance depends on API latency, model inference, queueing, and concurrency limits.</p><p>5.3. <strong>Optimizing cost and reliability</strong><br>A perfect compiler must optimize more than speed. A useful objective function is:<br>J(π) = αT(π) + βC(π) + γR(π) + δU(π)<br>π* = argmin J(π), for π in Π_valid<br>Here, π is the candidate execution plan; T is expected latency; C is expected execution cost; R is a risk measure; U is expected uncertainty or quality loss; and α, β, γ, δ represent business priorities. Π_valid contains plans that satisfy dependency, permission, data-type, and business constraints.<br>In our insurance example, an insurer might accept an extra two seconds of latency if it substantially reduces the chance of an incorrect recommendation. A customer-support chatbot may prioritize speed and cost more heavily. Hard safety rules must remain constraints but an unauthorized payment operation must stay prohibited regardless of its speed or cost.<br>5.4. <strong>Modeling task success probabilistically</strong><br>Suppose a workflow compiler contains five independent tasks, each with a success probability of 0.98. If all five must succeed and no recovery mechanism exists, makes even a reliable individual component can produce a less reliable end-to-end process.<br>P(success) = ∏ pᵢ = 0.98⁵ ≈ 0.9039<br>If probability p is for task succeeds per independent attempt and is attempted at most k times then:<br>P(success within k attempts) = 1 - (1 - p)ᵏ<br>For p = 0.8 and k = 3:<br>P = 1 - (0.2)³ = 0.992<br>That is a theoretical 99.2% success probability for the retried task, assuming independent failures and no adverse side effects. Retries do not automatically make the whole workflow 99.2% reliable, nor do they guarantee that a repeated external operation is safe. Idempotency, timeouts, and failure classification remain essential.<br>5.5. <strong>Type systems and contracts</strong><br>A compiler can verify whether task outputs are compatible with downstream inputs. For example:<br>inspect_damage: Image → DamageReport<br>estimate_loss: (DamageReport, Policy) → Currency<br>The compiler should reject a plan that passes raw image bytes directly to estimate_loss when a DamageReport is required. Types alone are insufficient for semantic correctness: a damage report may be structurally valid but refer to the wrong claim. Production contracts should also check identifiers, provenance, confidence thresholds, and domain-specific constraints.<br>5.6. <strong>Feedback and replanning</strong><br>Agentic workflows often operate in uncertain environments. Let sₜ represent the current workflow state and aₜ an action. The system transitions according to:<br>sₜ₊₁ = f(sₜ, aₜ, oₜ₊₁)<br>Here, oₜ₊₁ is the new observation returned by a tool or external system. A policy π(a | s) selects an action based on the current state. If document verification succeeds, the workflow proceeds; if the document is missing, it requests additional evidence.<br>This resembles a Markov decision process when the state, action, transition, and reward assumptions are appropriate. Real agentic systems may be partially observable, and an LLM’s proposed plan is not necessarily an optimal policy. Replan when new information changes the available actions or invalidates the existing plan, rather than regenerating the entire workflow after every tool response.</p><h4>6. How compilation works: From a goal to an executable plan</h4><p><strong>Stage 1 — Parse the goal</strong><br>Converting an incoming request into goals, constraints, expected output, and available capabilities.<br><strong>Stage 2 — Build the intermediate</strong> <strong>representation</strong><br>Represent the required tasks as nodes and dependencies as edges, in DAG format.<br><strong>Stage 3 — Perform semantic validation</strong><br>This stage checks whether every required task exists, whether dependencies, tools form an executable graph, and whether controls over actions are properly applied or not.<br><strong>Stage 4 — Optimize</strong><br>This stage helps in Identifying parallel branches, remove redundancies when safe, and select execution strategies under latency, cost, and resource constraints.<br><strong>Stage 5 — Generate and execute the plan</strong><br>This stage produces an executable graph or a plan for an orchestration engine.</p><h4>7. Python implementation: A mini Agentic Workflow Compiler</h4><blockquote><strong>Define the intermediate representation </strong><br>from dataclasses import dataclass, field <br>from collections import defaultdict <br>from concurrent.futures import ThreadPoolExecutor <br>from concurrent.futures import as_completed <br>import json <br>@dataclass <br>class Task: <br> name: str <br> tool: str <br> deps: list = field(default_factory=list) <br> args: dict = field(default_factory=dict) <br>@dataclass <br>class WorkflowIR: <br> name: str <br> tasks: list <br> goal: str = &quot;&quot;</blockquote><blockquote><strong>Create the available tool registry </strong><br>The compiler should permit only registered tools. These example functions simulate business operations using local data. <br>def fetch_policy(claim_id): <br> return { <br> &quot;claim_id&quot;: claim_id, <br> &quot;coverage&quot;: True, <br> &quot;coverage_limit&quot;: 50000 <br> } <br> <br> <br>def inspect_damage(image_id): <br> return { <br> &quot;image_id&quot;: image_id, <br> &quot;damage_detected&quot;: True, <br> &quot;estimated_repair&quot;: 12000 <br> } <br> <br> <br>def validate_coverage(policy): <br> return { <br> &quot;covered&quot;: policy[&quot;coverage&quot;], <br> &quot;limit&quot;: policy[&quot;coverage_limit&quot;] <br> } <br> <br> <br>def estimate_loss(damage, coverage): <br> if not coverage[&quot;covered&quot;]: <br> return {&quot;eligible&quot;: False, &quot;amount&quot;: 0} <br> <br> amount = min( <br> damage[&quot;estimated_repair&quot;], <br> coverage[&quot;limit&quot;] <br> ) <br> <br> return {&quot;eligible&quot;: True, &quot;amount&quot;: amount} <br> <br> <br>def prepare_recommendation(loss): <br> return { <br> &quot;status&quot;: &quot;READY_FOR_REVIEW&quot;, <br> &quot;recommended_amount&quot;: loss[&quot;amount&quot;] <br> } <br> <br> <br>TOOLS = { <br> &quot;fetch_policy&quot;: fetch_policy, <br> &quot;inspect_damage&quot;: inspect_damage, <br> &quot;validate_coverage&quot;: validate_coverage, <br> &quot;estimate_loss&quot;: estimate_loss, <br> &quot;prepare_recommendation&quot;: prepare_recommendation <br>}</blockquote><blockquote><strong>Compile and validate the graph </strong><br>We need to ensure that every dependency refers to an existing task and that the graph is acyclic. Kahn’s topological sorting algorithm repeatedly selects tasks with zero remaining dependencies. If tasks remain after this process, a cycle exists. <br>def compile_workflow(ir): <br> tasks = {task.name: task for task in ir.tasks} <br> <br> if len(tasks) != len(ir.tasks): <br> raise ValueError(&quot;Duplicate task identifiers&quot;) <br> <br> for task in ir.tasks: <br> if task.tool not in TOOLS: <br> raise ValueError(f&quot;Unknown tool: {task.tool}&quot;) <br> <br> for dep in task.deps: <br> if dep not in tasks: <br> raise ValueError( <br> f&quot;Unknown dependency {dep} in {task.name}&quot; <br> ) <br> <br> if dep == task.name: <br> raise ValueError(&quot;A task cannot depend on itself&quot;) <br> <br> remaining = { <br> name: set(task.deps) <br> for name, task in tasks.items() <br> } <br> <br> stages = [] <br> <br> while remaining: <br> ready = sorted( <br> name for name, deps in remaining.items() <br> if not deps <br> ) <br> <br> if not ready: <br> raise ValueError(&quot;Workflow contains a dependency cycle&quot;) <br> <br> stages.append(ready) <br> <br> for name in ready: <br> del remaining[name] <br> <br> for deps in remaining.values(): <br> deps.difference_update(ready) <br> <br> return {&quot;tasks&quot;: tasks, &quot;stages&quot;: stages}</blockquote><blockquote><strong>Execute the compiled workflow </strong><br>The runtime resolves references to earlier task outputs and executes independent tasks concurrently. <br>def resolve(value, results): <br> if isinstance(value, dict): <br> if set(value) == {&quot;$ref&quot;}: <br> return results[value[&quot;$ref&quot;]] <br> return { <br> key: resolve(item, results) <br> for key, item in value.items() <br> } <br> <br> if isinstance(value, list): <br> return [resolve(item, results) for item in value] <br> <br> return value <br> <br> <br>def execute_workflow(compiled, max_workers=4): <br> tasks = compiled[&quot;tasks&quot;] <br> results = {} <br> <br> for stage_number, stage in enumerate( <br> compiled[&quot;stages&quot;], start=1 <br> ): <br> print(f&quot;Stage {stage_number}: {stage}&quot;) <br> <br> def run_task(name): <br> task = tasks[name] <br> args = resolve(task.args, results) <br> return name, TOOLS[task.tool](**args) <br> <br> with ThreadPoolExecutor( <br> max_workers=max_workers <br> ) as executor: <br> futures = [ <br> executor.submit(run_task, name) <br> for name in stage <br> ] <br> <br> <strong>Collect every result before starting the next stage. </strong><br> for future in as_completed(futures): <br> name, output = future.result() <br> results[name] = output <br> <br> return results</blockquote><blockquote>Define the workflow and compile it <br>claim_workflow = WorkflowIR( <br> name=&quot;Motor Claim Investigation&quot;, <br> goal=&quot;Prepare a claim recommendation for human review&quot;, <br> tasks=[ <br> Task( <br> name=&quot;policy&quot;, <br> tool=&quot;fetch_policy&quot;, <br> args={&quot;claim_id&quot;: &quot;CLM-1001&quot;} <br> ), <br> Task( <br> name=&quot;damage&quot;, <br> tool=&quot;inspect_damage&quot;, <br> args={&quot;image_id&quot;: &quot;IMG-501&quot;} <br> ), <br> Task( <br> name=&quot;coverage&quot;, <br> tool=&quot;validate_coverage&quot;, <br> deps=[&quot;policy&quot;], <br> args={&quot;policy&quot;: {&quot;$ref&quot;: &quot;policy&quot;}} <br> ), <br> Task( <br> name=&quot;loss&quot;, <br> tool=&quot;estimate_loss&quot;, <br> deps=[&quot;damage&quot;, &quot;coverage&quot;], <br> args={ <br> &quot;damage&quot;: {&quot;$ref&quot;: &quot;damage&quot;}, <br> &quot;coverage&quot;: {&quot;$ref&quot;: &quot;coverage&quot;} <br> } <br> ), <br> Task( <br> name=&quot;recommendation&quot;, <br> tool=&quot;prepare_recommendation&quot;, <br> deps=[&quot;loss&quot;], <br> args={&quot;loss&quot;: {&quot;$ref&quot;: &quot;loss&quot;}} <br> ) <br> ] <br>) <br> <br>compiled = compile_workflow(claim_workflow) <br> <br>print(&quot;Compiled stages:&quot;) <br>print(json.dumps(compiled[&quot;stages&quot;], indent=2)) <br> <br>results = execute_workflow(compiled) <br> <br>print(&quot;\nFinal result:&quot;) <br>print(json.dumps(results[&quot;recommendation&quot;], indent=2))</blockquote><blockquote><strong>Sample input and output</strong><br>The workflow’s input consists of the claim identifier CLM-1001 and image reference IMG-501. The example tool functions return simulated records. <br>Compiled stages: <br>[ <br> [ <br> &quot;damage&quot;, <br> &quot;policy&quot; <br> ], <br> [ <br> &quot;coverage&quot; <br> ], <br> [ <br> &quot;loss&quot; <br> ], <br> [ <br> &quot;recommendation&quot; <br> ] <br>] <br> <br>Stage 1: [’damage’, &#39;policy’] <br>Stage 2: [’coverage’] <br>Stage 3: [’loss’] <br>Stage 4: [’recommendation’] <br> <br><strong>Final result: </strong><br>{ <br> &quot;status&quot;: &quot;READY_FOR_REVIEW&quot;, <br> &quot;recommended_amount&quot;: 12000 <br>}</blockquote><h4>What did we actually compile?</h4><p>The program converts a declarative workflow into an ordered collection of executable stages. It checks references, rejects unknown tools, detects cycles, resolves data dependencies, and executes independent operations concurrently. It does not yet use an LLM to interpret arbitrary natural-language requests. That would require an additional planning layer that produces a candidate WorkflowIR, followed by strict schema validation and policy checks. It also lacks durable persistence, authentication, real insurance integrations, distributed scheduling, and automatic recovery. Those are separate engineering concerns, not properties that automatically emerge from using a compiler pattern.</p><blockquote><strong>Connecting an LLM to the compiler </strong><br>The Python implementation uses a manually defined workflow. To make it agentic, we can add a planning model that converts a natural-language goal into a candidate workflow. <br>Example user input: <br>Investigate claim CLM-1001. <br>Verify policy coverage, inspect damage evidence, <br>estimate the loss, and prepare a recommendation. <br>Do not approve or pay the claim automatically. <br>An LLM could propose a structured representation such as: <br>{ <br> &quot;goal&quot;: &quot;Prepare claim recommendation&quot;, <br> &quot;tasks&quot;: [ <br> {&quot;name&quot;: &quot;policy&quot;, &quot;tool&quot;: &quot;fetch_policy&quot;, &quot;deps&quot;: []}, <br> {&quot;name&quot;: &quot;damage&quot;, &quot;tool&quot;: &quot;inspect_damage&quot;, &quot;deps&quot;: []}, <br> {&quot;name&quot;: &quot;coverage&quot;, &quot;tool&quot;: &quot;validate_coverage&quot;, <br> &quot;deps&quot;: [&quot;policy&quot;]}, <br> {&quot;name&quot;: &quot;loss&quot;, &quot;tool&quot;: &quot;estimate_loss&quot;, <br> &quot;deps&quot;: [&quot;damage&quot;, &quot;coverage&quot;]}, <br> {&quot;name&quot;: &quot;recommendation&quot;, &quot;tool&quot;: &quot;prepare_recommendation&quot;, <br> &quot;deps&quot;: [&quot;loss&quot;]} <br> ] <br>} <br>This JSON is a proposed plan, not executable authority.</blockquote><blockquote><strong>The compiler should validate it before execution.</strong> <br>1. Ask the LLM to generate a workflow using a predefined schema. <br>2. Parse and validate the schema. <br>3. Check every tool against an allowlist. <br>4. Validate dependencies, data types, permissions, and business rules. <br>5. Reject invalid plans or request a corrected proposal. <br>6. Execute only the validated graph. <br>7. Evaluate tool results and replan when necessary. <br>The LLM may decide which permitted task should happen next, but it must not be the sole authority deciding whether a sensitive operation is permitted. <br>For long-running workflows, established systems such as LangGraph and Temporal can provide state persistence, interruption, retries, and recovery mechanisms that would otherwise need to be built separately.</blockquote><h4>What happens when the plan fails?</h4><p><strong>Scenario A:</strong> All evidence is available <br>Then execute the required and planned tasks, estimate the loss, and create a recommendation for the authorized reviewer. <br><strong>Scenario B:</strong> A document is missing <br>System paused the affected branch, ask for the missing evidence, and resume or recompile the relevant portion after the document arrives from the human supervisor side or input pipeline. <br><strong>Scenario C:</strong> A restricted operation is proposed <br>Workflow block the operation unless the required authorization is obtained. Changes in plans must not bypass the restriction. A practical troubleshooting guideline should distinguish between transient failures, invalid input, permanent business-rule failures, and authorization failures. <br>A temporary network failure might justify a redo with exponential back off.</p><blockquote>A policy mismatch might require review from the human supervisor rather than repeated attempts. A missing document might require waiting for external input in the workflow. Reliable and durable orchestration is important because the system needs to preserve completed work, record the failure, and resume without accidentally repeating an irrecoverable operation.</blockquote><h4><strong>Real-world applications</strong></h4><p><strong>Banking and insurance</strong> — A workflow compiler can coordinate fraud screening, document extraction, policy verification, risk scoring, and report generation. <br><strong>Potential benefit:</strong> more consistent investigations, traceable decisions, and less manual coordination. <br> <strong>Industrial AI and manufacturing</strong> — An agent can analyze sensor anomalies, request diagnostic data, compare maintenance records, and prepare a repair plan. <br><strong>Potential benefit:</strong> coordinated diagnostics and faster maintenance planning. <br><strong>Software engineering</strong> — A workflow may interpret a bug report, retrieve relevant code, propose a patch, run tests, analyze failures, and prepare a pull request. <br><strong>Potential benefit:</strong> repeatable development workflows with clear checkpoints. <br><strong>Healthcare and medical research</strong> — A research workflow can retrieve authorized records, analyze pathology images, summarize relevant literature, compare model outputs, and prepare a draft report. <br> <strong>Potential benefit:</strong> reproducible research pipelines and reduced administrative work. <br><strong>Data analytics and business intelligence</strong> — A request such as “identify the causes of declining revenue” can be decomposed into data retrieval, data-quality checks, SQL execution, statistical analysis, visualization, and report generation. <br><strong>Potential benefit:</strong> automated analytical workflows with explicit dependencies and reproducible outputs. <br>These are application patterns, not claims that a compiler alone delivers a particular business improvement. Actual outcomes depend on data quality, model capability, integration design, validation, and operational controls.</p><h4><strong>When we should think an Agentic Workflow Compiler?</strong></h4><p>Use this architecture when …. :</p><blockquote>• A task requires multiple tools with complex dependencies. <br>• Different tasks can run in parallel. <br>• The workflow must be auditable or repeatable. <br>• Failures require controlled recovery. <br>• AI needs to adapt plans to new information. <br>• Sensitive operations require explicit approval or authorization.</blockquote><p>Avoid introducing a compiler when a simple deterministic function or short script already solves the problem. For a fixed three-step process, the additional architecture may cost more to build and maintain than it saves. <br>A practical adoption strategy is to begin with a small workflow, define its task contracts, measure latency and failure rates, and introduce dynamic planning only where it provides measurable value.</p><h4>Conclusion</h4><p>In this post we explored the Agentic AI Workflow Compilers able to provide a useful way to bridge the flexibility of AI agents and the discipline of conventional software engineering. They transform a natural language requirement into a structured workflow that can be validated, optimized, scheduled, and executed under defined constraints. This post gives us that mathematical foundation for these types of workflow compilers are not one single algorithm. It combines various mathematical concepts such as graph theory for dependencies, optimization for scheduling and cost, probability for reliability analysis, type systems for interface validation, and feedback-based planning for adaptation, in order to craft a reliable and durable agent ai workflows compilers. With this post reaches to end I hope it able to deliver the comprehensive understanding of Agent AI workflow compilers concept and readers can adopt these techniques for improve their business processes.</p><p><em>Note : Ultimately, the goal is not to make every part of an AI system autonomous. It is to make autonomy controllable, execution measurable, and failures recoverable.</em></p><p><strong><em>Thank you readers !!!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c553eead6b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Agent’s Harness Engineering: The Engineering Layer That Makes Autonomous AI Systems Reliable]]></title>
            <link>https://sumitkrsharma-ai.medium.com/ai-agents-harness-engineering-the-engineering-layer-that-makes-autonomous-ai-systems-reliable-54b9640c1bb8?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/54b9640c1bb8</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 04:23:16 GMT</pubDate>
            <atom:updated>2026-10-07T04:24:44.919Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine Company hired an extremely intelligent employee and commanded him to, “Handle customer complaints.” The person is smart enough to understand emails, search documents, write responses and even make decisions. But there is a problem: What systems can employees access? What information should the employee see? Are employees allowed to issue refunds? Who checks the employee’s decisions? What happens if an API fails? How does an employee know when the task is actually finished? hiring an intelligent person is not enough. Employers need an environment around the employees such as tools, permissions, instructions, monitoring, validation, memory and rules. That environment is essentially what an AI agent harness provides. The development of that kind of environment is also known as Agent Harness Engineering. The idea has become increasingly important as AI systems moving from AI Assistants to performing multi-step tasks autonomously. OpenAI’s recently launched product with Codex is a strong example that shifts the engineering effort from manually coding into designing the environment, tools, feedback loops, validation and constraints that allow agents to reliably produce software codes autonomously. This post is about granular understanding of Agent Harness Engineering that allows readers to understand their AI Agents environment before turning on the production mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PWXSYnjY0OGyI-bS8a_UoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*75zHwpCe2Snh4WAzJH6wtA.png" /></figure><h4><strong>What Is Agent Harness Engineering?</strong></h4><p>It is the discipline of designing, implementing and continuously improving the software environment around an AI model so that the model can operate as a reliable, controlled and observable agent. Simplified representation of Agent Harness Engineering is: <br><strong>Agent = AI Model + Harness</strong> <br>Where the model provides reasoning, thinking and generation capabilities. But harness provides the supporting environment under which Agent’s intelligence is used, once harness engineered it is able to control the following things :</p><p>Ask What context the model receives ?<br>• Which tools it can access <br>• How API’s are called <br>• Which memory it can use <br>• Where it can execute code <br>• How errors are handled <br>• How outputs are validated <br>• When the agent should stop <br>• How its actions are monitored <br>• How failures are rollbacked <br>• How and where human approval is introduced <br>• How expenses and token usage are controlled</p><p>That makes the harness engineering is different from simply writing a better system prompt. Prompt engineering majorly changes the instruction given to a model, whereas harness engineering considers the execution environment around the AI agent.</p><h3>Why Does the AI Agent’s Need Agent Harness Engineering?</h3><p>The Initial applications of GenAI majorly focused :</p><p><strong>User → Prompt → LLM → Answer </strong><br>But current new GenAI applications are increasingly looks like:</p><p><strong>User → Agent → Plan → Tools → Data → Actions → Validation → Final Result</strong></p><p>This is a completely different problem, consider an AI customer-service agent receives: “Refund my cancelled order and notify the customer.”</p><blockquote>Then agent needs to perform series of steps such as , the agent may need to: <br>1. Find the customer.</blockquote><blockquote>2. Find the order.</blockquote><blockquote>3. Verify cancellation status.</blockquote><blockquote>4. Check refund eligibility.</blockquote><blockquote>5. Calculate refund amount.</blockquote><blockquote>6. Call the payment system.</blockquote><blockquote>7. Update CRM. 8. Send an email.</blockquote><blockquote>9. Verify that everything succeeded.</blockquote><p>An LLM model can reason about these steps, but reasoning of LLM alone does not safely execute them. The harness is responsible for controlling this process. This is particularly important for sensitive data handling industries such as banking, healthcare, insurance, manufacturing, cybersecurity, government and enterprise systems, where an incorrect autonomous action can have financial or operational consequences.</p><h4>Mathematical Prospects of Agent Harness Engineering</h4><p>Model represented as follows in mathematical termnologies :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*x6WFVP6NIlJgChKa_3BRgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*M8sm0_BLXPeZXSfuQB-oCw.png" /></figure><p>And Harness decides that action can actually happen or not.</p><h4>Core Elements of Agent Harness Engineering</h4><blockquote>1. <strong>Model Layer</strong> — This layer contains a underlying LLM provides thinking, reasoning, planning and generation capabilities.</blockquote><blockquote>2. <strong>Context Layer</strong> — This is layer that determines which information enters the model’s working context, including user requests, instructions, retrieved documents, database information, previous decisions, tool’s outputs, memory and current state.</blockquote><blockquote>3.<strong> Tool Layer</strong> — This layer gives an ability to agents to interact with databases, REST APIs, Python interpreters, browsers, CRMs, ERPs, file systems, search engines, email systems, payment gateways and IoT devices.</blockquote><blockquote>4. <strong>Memory Layer</strong> —This layer responsible to manage the short-term memory, long-term memory and task state.</blockquote><blockquote>5. <strong>Guardrail Layer</strong> — This is safety layer that restricts dangerous or invalid behavior through transaction limits, allowed endpoints, restricted data access, PII filtering, prompt-injection detection and human approvals.</blockquote><blockquote>6. <strong>Execution Environment</strong> — It Provides the controlled environment where agents perform actions, such as VMs, sandboxes, containers, repositories, terminals, cloud APIs or enterprise applications.</blockquote><blockquote>7. <strong>Validation Layer </strong>— This layer Checks schema, business rules, security, data quality, policy compliance and numerical correctness.</blockquote><blockquote>8. <strong>Observability Layer</strong> — The monitoring layer that captures traces such as task ID, prompt, model call, selected tool, parameters, result, validation, retries and final outcome.</blockquote><h4>Sample Python Implementation</h4><p>Let’s see the example python implementation that demonstrates a simplified harness for an order-refund agent. Key point is that the LLM does not directly execute the refund, the developed harness checks the requested action first before making any decisions .</p><blockquote>class AgentHarness: <br> def __init__(self, refund_limit=10000): <br> self.refund_limit = refund_limit <br> <br> def validate_action(self, action): <br> if action[&quot;type&quot;] != &quot;refund&quot;: <br> return False, &quot;Unsupported action&quot; <br> <br> if action[&quot;amount&quot;] &lt;= 0: <br> return False, &quot;Invalid refund amount&quot; <br> <br> if action[&quot;amount&quot;] &gt; self.refund_limit: <br> return False, &quot;Human approval required&quot; <br> <br> return True, &quot;Action approved&quot; <br> <br> def execute(self, action): <br> valid, message = self.validate_action(action) <br> <br> if not valid: <br> return { <br> &quot;status&quot;: &quot;BLOCKED&quot;, <br> &quot;reason&quot;: message <br> } <br> <br> return { <br> &quot;status&quot;: &quot;SUCCESS&quot;, <br> &quot;message&quot;: f&quot;Refund of ₹{action[’amount’]} processed&quot; <br> } <br> <br>harness = AgentHarness(refund_limit=10000) <br> <br>request = { <br> &quot;type&quot;: &quot;refund&quot;, <br> &quot;amount&quot;: 8500 <br>} <br> <br>result = harness.execute(request) <br>print(result)</blockquote><blockquote><strong>Sample Input: </strong><br>{ <br> &quot;type&quot;: &quot;refund&quot;, <br> &quot;amount&quot;: 8500 <br>} <br><strong>Sample Output: </strong><br>{ <br> &#39;status’: &#39;SUCCESS’, <br> &#39;message’: &#39;Refund of ₹8500 processed&#39; <br>}</blockquote><p>In above example python implementation if a ₹25,000 refund, the harness returns a blocked result with the reason “Human approval required.” This sample shows that agent may recommend an action, but the harness controls whether that action is actually executed.</p><h4>Agent Harness Engineering vs Prompt Engineering</h4><p>Prompt Engineering <strong>focuses on instructions</strong> and asks: “What should I tell the model?” <br>Context Engineering <strong>keep eye on information</strong> and asks: “What information should the model see?” <br>However , Harness Engineering <strong>manages the end to end operating environment</strong> and asks: “What environment should the agent operate in, what can it do, how do we control it, and how do we know it succeeded?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1yCa0GjicsKjt6K7g94w9w.png" /></figure><h4>Advantages</h4><blockquote>1. Better Reliable Agents <br>2. Higher Security <br>3. Easy to Troubleshoot <br>4. Better Model Flexibility <br>5. Autonomous Improvements <br>6. Controlled access controls</blockquote><p><strong>Disadvantages and Challenges</strong></p><blockquote>1. Development Complexity. <br>2. Higher Initial Cost. <br>3. Maintenance and management efforts. <br>5. False Confidence due to bad business rules.</blockquote><h4>Real-World Applications</h4><blockquote>1. <strong>Banking and FinTech</strong> — Harnesses can manage the transaction limits, authentication, audit trails and approval workflows in the Agent.</blockquote><blockquote>2.<strong> Insurance</strong> — AI Agents can investigate the policies, claims, customer history, images and repair estimates while the harness controls high-value automatic approvals.</blockquote><blockquote>3. <strong>Manufacturing</strong> — AI Agents can monitor machine telemetry, production lines, quality reports and maintenance records while harness handles the safety conditions govern autonomous actions.</blockquote><blockquote>4. <strong>Customer Support </strong>— AI Agents can understand complaints, find customers, check orders and policies, propose resolutions, execute permitted actions, update CRM and close tickets under controlled environment.</blockquote><h4>What Business Owners Should Understand</h4><p>Setting up a powerful <strong>AI model does not automatically grantees you a reliable AI employee</strong>. The <strong>model itself only one part of complete system, If business owners want an AI agent to operate their CRM, analyze claims, manage inventory, write software or interact with customers, employer’s need to engineer the environment around their AI agent.</strong> <br>A useful business setup looks like : <br><strong>Business Agent = AI Model + Tools + Data + Access controls + Validation + Observability + Human Oversight </strong><br>Instead of searching only for, “Which is the best AI model?”, businesses must need to ask, “What environment will allow the AI model to perform this job safely and reliably?” That is where Agent Harness Engineering becomes strategically important before deployment of AI system.</p><h4><strong>Conclusion</strong></h4><p>In this post as reader we are able to understand that ‘Agent Harness Engineering’ is not simply different terminology for prompt engineering it represents a broader shift in how we build AI systems, where models provide the intelligence and harness provides the operational discipline. It helps the agent to decides what the need to see, what agent can access, what agent can execute, how agent’s actions are validated, what happens when something goes wrong and when a human involvement must require to take over the system. <br>In nutshell, <strong>Intelligence gives an agent capability</strong>; the <strong>harness gives it control</strong>. As businesses move from AI that generate the answers based on questions into the AI that takes actions, that distinction is likely to become one of the most key engineering ideas in the agentic AI era. And I hope this post delivers a quick hand notes for understating the Agentic Harness Engineering. <br> <br>Thank you, readers, !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54b9640c1bb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Continuous Deep Learning: Neural ODEs (Neural Ordinary Differential Equations) Explained in Brief]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sumitkrsharma-ai.medium.com/continuous-deep-learning-neural-odes-neural-ordinary-differential-equations-explained-in-brief-d8ba61c36330?source=rss-246685f156aa------2"><img src="https://cdn-images-1.medium.com/max/1280/1*bBSVP98txelSyCdzjHX-5Q.png" width="1280"></a></p><p class="medium-feed-snippet">Imagine you are trying to model the path of a human going down a curvy water slide. Models like ResNets, CNNs are Flipbook animation. You&#x2026;</p><p class="medium-feed-link"><a href="https://sumitkrsharma-ai.medium.com/continuous-deep-learning-neural-odes-neural-ordinary-differential-equations-explained-in-brief-d8ba61c36330?source=rss-246685f156aa------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sumitkrsharma-ai.medium.com/continuous-deep-learning-neural-odes-neural-ordinary-differential-equations-explained-in-brief-d8ba61c36330?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ba61c36330</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 10:21:44 GMT</pubDate>
            <atom:updated>2026-08-25T01:51:27.950Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Recurrent Vision Transformers: Let’s bring a Memory and Iterative Reasoning to Computer Vision]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sumitkrsharma-ai.medium.com/recurrent-vision-transformers-lets-bring-a-memory-and-iterative-reasoning-to-computer-vision-6f3f33f26a83?source=rss-246685f156aa------2"><img src="https://cdn-images-1.medium.com/max/1280/1*vbY2k-NduhCbqlufxnaatg.png" width="1280"></a></p><p class="medium-feed-snippet">Over decades, computer vision has undergone a remarkable transformation. Conventional image recognition tasks were dominated by&#x2026;</p><p class="medium-feed-link"><a href="https://sumitkrsharma-ai.medium.com/recurrent-vision-transformers-lets-bring-a-memory-and-iterative-reasoning-to-computer-vision-6f3f33f26a83?source=rss-246685f156aa------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sumitkrsharma-ai.medium.com/recurrent-vision-transformers-lets-bring-a-memory-and-iterative-reasoning-to-computer-vision-6f3f33f26a83?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f3f33f26a83</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 06:58:07 GMT</pubDate>
            <atom:updated>2026-07-04T06:58:07.397Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Recurrent Vision: Seeing Like Humans, Not Just Processing Like Machines]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sumitkrsharma-ai.medium.com/recurrent-vision-seeing-like-humans-not-just-processing-like-machines-efec0d1d9b15?source=rss-246685f156aa------2"><img src="https://cdn-images-1.medium.com/max/1280/1*TwI2QNDXwKtzKp23g-zglQ.png" width="1280"></a></p><p class="medium-feed-snippet">Picture yourself walking into a darkened room. You don&#x2019;t really get to see what&#x2019;s inside the first time you look.</p><p class="medium-feed-link"><a href="https://sumitkrsharma-ai.medium.com/recurrent-vision-seeing-like-humans-not-just-processing-like-machines-efec0d1d9b15?source=rss-246685f156aa------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sumitkrsharma-ai.medium.com/recurrent-vision-seeing-like-humans-not-just-processing-like-machines-efec0d1d9b15?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/efec0d1d9b15</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Fri, 15 May 2026 03:58:14 GMT</pubDate>
            <atom:updated>2026-05-15T03:58:14.570Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Detect the Failure for the User before they Complain about your GenAI Application!]]></title>
            <link>https://sumitkrsharma-ai.medium.com/detect-the-failure-for-the-user-of-your-genai-application-complaint-072b233e5b19?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/072b233e5b19</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 11:19:05 GMT</pubDate>
            <atom:updated>2026-03-30T11:21:18.404Z</atom:updated>
            <content:encoded><![CDATA[<p>“GenAI applications are probabilistic microservices those requires <strong>continuous health checkups !!</strong>”</p><p>Production-level observability in LLM application systems means <strong>having complete, real-time visibility into what deployed model is doing, how &amp; why it’s doing it, and even whether it’s doing it correctly or not before it affects the users or business outputs</strong>. It goes far beyond logging and error recordings; it is a <strong>multi-layered real-time tracing system combining all metrics, logs, evaluation, and alerting hits</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xTtPXSaU1nBJyfY_kpL8Nw.png" /></figure><h4>1. What “Production-Levle Observability” Really Means?</h4><p>In traditional ML application systems:<br> ➡Observability = log files + metrics scores</p><p>In LLM application systems:<br> ➡ Observability =<br> <strong>log files + traces + metric scores+ LLM evaluation + user behavior + cost tracking</strong></p><p>Because GenAI applications generate:</p><ul><li>Non-deterministic outputs.</li><li>Hallucinations.</li><li>Context dependency from retrieval-augmented generation (RAG).</li><li>High cost variability with each response in the form of tokens.</li></ul><h4>2. Key Pillars of Production Level Observaretry</h4><h4>2.1 Distributed Tracing of End-to-End System Pipeline Visibility</h4><p><strong>Purpose of distributed tracing — </strong>For tracking <strong>each and every request across the full GenAI application pipeline.</strong></p><p><strong>What we trace:</strong></p><ul><li>API request start/end</li><li>Retrieval latency (delays)</li><li>LLM call time (response time)</li><li>Token usage (per million tokens generally)</li><li>External API calls</li></ul><p><strong>Desired Output:</strong></p><p>A trace like:</p><blockquote>Query → Retriever (100ms) → LLM (2.3s) → Response generated!</blockquote><p><strong>Importance:</strong></p><ul><li>Detect bottlenecks in the flow of the GenAI application pipeline.</li><li>Detection of failures basically helps in identifying which component fails.</li><li>Debug complex pipelines such as agents and tools in GenAI systems.</li></ul><h4>2.2 Structured Logging for Context-Aware Logs</h4><h4>ML systems Traditional log files bad ):</h4><pre>Error: something failed</pre><h4>GenAI Production level log files good :)</h4><pre>{<br>  &quot;query&quot;: &quot;loan status&quot;,<br>  &quot;retrieved_docs&quot;: 2,<br>  &quot;context_length&quot;: 1200,<br>  &quot;response&quot;: &quot;...&quot;,<br>  &quot;latency&quot;: 2.1,<br>  &quot;model&quot;: &quot;gpt-4&quot;,<br>  &quot;user_id&quot;: &quot;123&quot;,<br>  &quot;trace_id&quot;: &quot;abc-xyz&quot;<br>}</pre><h4>Why so critical:</h4><ul><li>Allow system developers filtering, debugging, and analytics for faster troubleshooting.</li><li>Connects log files with traces.</li></ul><h4>2.3 Metrics for Quantitative Monitoring of GenAI Application Systems</h4><p><strong>Important metrics for GenAI application systems:</strong></p><ul><li>For latency using p50, p95, and p99</li><li>Token usage for each input/output</li><li>Cost per hit</li><li>Error rate in responses</li><li>Retrieval success rate</li></ul><p><strong>Other important metrics:</strong></p><ul><li>Hallucination rate</li><li>Answer relevance score</li><li>User satisfaction score</li></ul><h4>2.4 LLM Evaluation Layer for GenAI application systems is a UNIQUE PART!</h4><p>This section makes the GenAI observability different from other ML systems’ observability.</p><p>We need to measure:</p><ul><li><strong>Faithfulness</strong> → Is the answer grounded in context or not?</li><li><strong>Relevance</strong> → Does the generated response answer the question input?</li><li><strong>Toxicity / Safety of the generated response!</strong></li></ul><p>Some Popular Methods:</p><ul><li>Rule-based checks</li><li>LLM-as-a-judge</li><li>Benchmark datasets</li></ul><p>👉 Without these LLM evaluations, identification of <strong>silent failures is so hard. :(</strong></p><h4>2.5 Alerting generating elements of the GenAI system for proactively identifying silent failures in GenAI systems!</h4><p><strong>Setting the triggering conditions as the following:</strong></p><ul><li>Latency &gt; 4 sec</li><li>Faithfulness &lt; 0.8</li><li>Retrieval &lt; 1 document</li></ul><p><strong>After Trigger Actions:</strong></p><ul><li>Alert through various channels such as Slack / email / PagerDuty alerts</li></ul><p>This triggering system shifts the approach from reactive to proactive.</p><h4>2.6 User Behaviour Observatory</h4><p>GenAI system failures may not even produce errors; yes, that makes the tracking more crucial components for successful production-grade systems.</p><h4>So tracking it by:</h4><ul><li>Repeated queries and their responses</li><li>Session drop-offs for each session</li><li>Query reformulations for each new query into the system</li></ul><p>👉<strong><em> These show whether the system is having the following issues or not, such as:</em></strong></p><ul><li>Confusion while generating the responses</li><li>Poor responses even for simpler queries</li><li>Hidden failures</li></ul><h4>3. Observability layers for GenAI architecture have the following components:</h4><p>Think of it as a 5-layered<strong> architecture, each layer as mentioned</strong>:</p><p><strong>Layer 1: Infrastructure Layer</strong></p><ul><li>Includes system infrastructure details such as CPU, memory, and GPU usage</li><li>GenAI system API uptime</li></ul><p><strong>Layer 2: Application Layer</strong></p><ul><li>API latency for monitoring the delays</li><li>Error rates</li></ul><p><strong>Layer 3: LLM Pipeline Layer</strong></p><ul><li>Prompt for system and by the user</li><li>Context for reference for response generation</li><li>Response against the query by the user</li></ul><p><strong>Layer 4: Intelligence Layer</strong></p><ul><li>Evaluation scores computed</li><li>Hallucination detection performed</li></ul><p><strong>Layer 5: Business Layer</strong></p><ul><li>User satisfaction rate</li><li>Conversion rate</li><li>Task success rate</li><li>Costing rate as well 😃</li></ul><p>👉 Production-grade deployment = visibility across <strong>ALL layers of the architecture.</strong></p><h4>4. As a developer, you must detect the following failure types in a production-grade system 😳</h4><p><strong>A. Hard failures, these are easy to detect as well.</strong> 😄</p><ul><li>API crash</li><li>Timeout</li><li>No response</li></ul><p><strong>B. Soft failures are most dangerous for users. </strong>😬</p><ul><li>Hallucination</li><li>Irrelevant answer</li><li>Partial answer</li></ul><p><strong>C. Silent failures are the most dangerous critical.</strong> 😧</p><ul><li>User leaves without feedback</li><li>Wrong decision taken</li></ul><p>👉 Observability must be targeted over both <strong>soft and silent failures in the production system.</strong></p><h4>5. Key Design Principles of GenAI Application Systems</h4><p>5.1 Correlation is everything in GenAI systems.</p><p><strong>Use:</strong></p><ul><li>trace_id (unique id for each new hit)</li><li>request_id (unique id for each new hit)</li></ul><p><strong>To flow to connect the architecture:</strong></p><pre>Logs ↔ Traces ↔ Metrics ↔ Evaluation</pre><h4>5.2 Sampling Strategy</h4><p>You can’t log everything in the log file while the system is scaled.</p><p><strong>Use:</strong></p><ul><li>Ensure that all errors are fully logged into the log files.</li><li>Sampling for each normal request.</li></ul><h4>5.3 Data Privacy</h4><ul><li>Always mask the sensitive or even non-data-sensitive information in logs for security reasons.</li><li>Avoid storing raw PII in prompts into the system.</li></ul><h4>5.4 Real-time system flow be like 😅</h4><ul><li>Real-time system → alerts based on triggers</li><li>Batch → deeper analysis for each batch</li></ul><h4>6. Production-Grade Tooling Stack ⭐️</h4><p>Observability Popular Tools</p><ul><li>For Tracing → <strong>OpenTelemetry</strong></li><li>For logging of events →<strong> structlog / ELK stack</strong></li><li>Metrics for evaluation → <strong>Prometheus + Grafana</strong></li></ul><p>LLM Applications: Specific Platforms</p><ul><li><strong>LangSmith</strong><br>→ Used for prompt tracing, debugging, evaluation for queries, and responses both.</li><li><strong>Weights &amp; Biases</strong><br>→ Used for experimentation and metric tracking during the real-time working of the GenAI application.</li></ul><p><strong>Evaluation Tools Used —</strong></p><ul><li>RAGAS</li><li>DeepEval</li><li>Custom LLM judge pipelines</li></ul><h4>7. Production Observability architecture, it is just a sample that varies application to application. 🏗️</h4><pre>User Request<br>   ↓<br>API Layer (FastAPI)<br>   ↓<br>[Trace Start]<br>   ↓<br>Retriever (Vector DB)<br>   ↓<br>LLM Call<br>   ↓<br>Evaluation Layer<br>   ↓<br>Structured Logging<br>   ↓<br>Metrics Collection<br>   ↓<br>Dashboard (Grafana / LangSmith)<br>   ↓<br>Alerting System</pre><h4>8. Example Scenario: Detecting Failure BEFORE Complaint</h4><p><strong>Scenario:</strong></p><p>Consider a user asking, “What does my insurance policy cover?”</p><p><strong>What happens:</strong></p><ul><li>When the retriever returns the wrong doc</li><li>LLM generates a confident but wrong answer</li></ul><p>Defined Observability Detects The 😅:</p><ul><li>Faithfulness score = 0.3 ❌</li><li>Retrieval relevance low ❌</li></ul><p>Applied System Triggers:</p><ul><li>Generates the alert and sends it to the dev team through different channels such as Slack/Mail.</li><li>Logs recorded in log file flagged</li><li>Query added to evaluation dataset of the system</li></ul><p>👉 Issue identified and fixed before users complain</p><h4>End Note —</h4><p>Production-grade systems failure puts the liability on the development team. To reduce the damages and the cost borne by this liability, proactive approaches are required. These approaches can vary from application to application. A generic method discussed in the previous post helps alleviate customers’, partners’, and users’ fears of unwanted risks in business processes that arise from the inherent weaknesses of GenAI applications in generating responses to queries. I hope the post is able to deliver the proactive measures with their significance and sample integration methods.</p><p>Thank you readers !!! Stay connected !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=072b233e5b19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing Best Deployment Method for your GenAI Application]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sumitkrsharma-ai.medium.com/best-deployment-method-for-your-genai-application-067181ac8cfd?source=rss-246685f156aa------2"><img src="https://cdn-images-1.medium.com/max/1280/1*RObi3Aqa5qSdiSeeIwuiPQ.png" width="1280"></a></p><p class="medium-feed-snippet">Selecting the most productive deployment method for a GenAI or LLM application for any use case is equivalent to deciding how a business&#x2026;</p><p class="medium-feed-link"><a href="https://sumitkrsharma-ai.medium.com/best-deployment-method-for-your-genai-application-067181ac8cfd?source=rss-246685f156aa------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sumitkrsharma-ai.medium.com/best-deployment-method-for-your-genai-application-067181ac8cfd?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/067181ac8cfd</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Fri, 20 Mar 2026 10:31:36 GMT</pubDate>
            <atom:updated>2026-03-20T10:33:29.859Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Selecting Best Evalution metric for Evaluation of your LLM Application]]></title>
            <link>https://sumitkrsharma-ai.medium.com/selecting-best-evalution-metric-for-evalution-of-your-llm-application-d2df98588f42?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2df98588f42</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 08:50:41 GMT</pubDate>
            <atom:updated>2026-02-23T08:51:45.608Z</atom:updated>
            <content:encoded><![CDATA[<p>In continuation of our previous discussion on choosing the best evaluation metric for LLM models, we now dive deeper into a broader set of metrics and explore how they align with real-world applications. While selecting a single metric can provide some insight, understanding the context in which an LLM operates is crucial to measure its true effectiveness. In this post, we examine various evaluation strategies, discuss which metrics are most suitable for different types of LLM applications, and highlight how the right choice can reveal the actual working performance of these models in practical scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZtHkp_37HJo-CaO4AMng6w.png" /></figure><h4>Bias / Fairness Metrics</h4><p><strong>Real-World Use Case </strong>— Imagine a hiring assistant must treat all genders and groups equally. Many times in LLM-based hiring systems, during development, the model may score highly on overall accuracy because the training data is dominated by historically overrepresented groups. However, once the system was deployed in a real-world environment, it started unfairly disadvantaging candidates from certain genders, ethnicities, or educational backgrounds. Then bias/fairness metrics expose these issues; those who were having hidden performance drops are exposed by bias metrics, which do analysis over the evaluation outcomes across demographic groups rather than aggregate scores. Similarly, in the case of healthcare diagnostic models, high validation scores can mask poor performance for minority populations, which becomes evident only after deployment. By maintaining performance consistency across diverse real-world populations, fairness metrics directly address these gaps.</p><p><strong>Mathematical formula </strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*YTtnYCZ5Am_zaWEjsleTXA.png" /><figcaption><strong>Demographic Parity</strong>,</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*tLsOjUgLSXViG1rpvRNiWA.png" /></figure><p><strong>Applications of metrics — </strong>Compliance and regulation, like highly sensitive domains, extensively apply bias and fairness metrics, emphasizing the importance of ethical and legal accountability. In hiring portals, these metrics guarantee the equality in candidate’s profile selections. Whereas in financial services, they help maintain compliance with fair lending laws and prevent discriminatory credit decisions. Recommendation systems and LLM-based assistants increasingly employ fairness metrics to identify biased responses based on gender, culture, or ideology. For LLM applications, bias metrics are crucial for checking errors that mainly impact specific groups, making sure the model’s results are fair and reliable in real life. Overall, fairness metrics shift evaluation from “How accurate is the model?” to “Who does the model work for — and who does it fail?”</p><h4>Coherence</h4><p><strong>Real-World Use Case — </strong>Consider a scenario where customer support chatbots have higher evaluation scores in a development environment. In a production environment, that same chatbot starts generating fragmented or logically inconsistent responses across multi-turn conversations. Similar to that image, an enterprise report generation system had reports in production generated having contradictions with the actual input documents along with broken narrative flow and abrupt topic shifts in the responses generated in the report. Coherence metrics help in directly capturing the degradation by evaluating whether the generated content maintains logical flow and internal consistency over longer contexts, revealing issues that accuracy-based metrics fail to detect.</p><p><strong>Mathematical formula </strong>:</p><p>A commonly used method uses the s<strong>entence embedding similarity</strong>, in which coherence is computed as the average cosine similarity between embeddings of adjacent sentences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*NHbZKgyeei5tHLREtd7GvQ.png" /></figure><p>In the above equations, ‘base i’​ represents the embedding of sentence ‘i’ And further formulations use <strong>entity grid models</strong> that track the presence and transitions of entities across sentences to quantify discourse continuity. In some evaluation pipelines, coherence is also used in discriminative models that score how well a text follows a logical order. For coherence metrics, <strong>higher scores indicate better performance</strong>, as they reflect stronger logical flow and contextual consistency in generated outputs.</p><p><strong>Applications of metrics — </strong>Coherence metrics are generally used for long-form text generation, multi-turn conversational agents, and document-level reasoning systems. In chatbots, coherence evaluation ensures that responses remain aligned with earlier user inputs and avoid contradictions across turns. In content generation platforms, such as technical writing or news summarization tools, coherence metrics help maintain structured narratives and readable outputs. For LLM-based decision-support systems, coherence is critical for building user trust, as logically inconsistent responses — despite being factually correct — can reduce credibility in real-world deployment. Overall, coherence metrics play a key role in bridging the gap between high development scores and reliable real-world performance by focusing on discourse-level quality rather than isolated response accuracy.</p><h3>Consistency</h3><p><strong>Real-World Use Cases — </strong>Imagine a scenario where a multi-turn conversational assistant was deployed,<strong> </strong>and during the training/testing phase, the assistant is evaluated on single conversions or lightly contextual prompts, which gives higher scores during testing for evaluation. But, when deployed in production, users frequently rephrase questions, ask follow-ups, or reference earlier responses, causing the model to generate contradictory or changing answers. Enterprise knowledge assistants are another real-world example. When the same factual question is asked at different times or in different words, the answers may not be the same, which makes users less likely to trust the system even though it works well offline. Similarly, in <strong>policy, legal, or compliance-driven applications</strong>, inconsistency in answers, such as varying interpretations of the same rule, may lead to serious operational risks. However, consistency metrics help directly capture this gap by measuring whether the responses generated by the assistant are stable and aligned across paraphrases, sessions, or conversational turns, even in longer, sequential or non-sequential conversations.</p><p><strong>Mathematical formula </strong>:<br>In this metric consistency is commonaly measured by evaluating agreement between the model’s outputs for semantically equivalent inputs. A common formula used for is computes the average similarity between responses using embedding based cosine similarity and given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*g6KdQJ7X8xm9RECanLXDwQ.png" /></figure><p>where r′ ‘base i’​ and r′ ‘base i’ are responses to paraphrased or repeated versions of the same query. In the classification or structured outputs, consistency can also be measured as the proportion of identical predictions across equivalent inputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*gFTmMlFLnkTsUBFLeZlpoA.png" /></figure><p>For conversational systems, contradiction detection models are sometimes used to identify logical conflicts across turns. In all formulations, <strong>higher consistency scores are better</strong>, as they indicate stable, reliable, and repeatable model behavior.</p><p><strong>Applications of metrics — </strong>This measurement shows how well different parts of conversational systems agree with each other, making it popular in chatbots, virtual assistants, retrieval-augmented generation (RAG) systems, and decision-support tools. And specifically in the customer support and enterprise helpdesk systems, consistency ensures that users receive similar guidance regardless of the time or the way they ask a query to the conversational system. Even in cases where such as educational or advisory platforms devloped around the LLM this metric helps in preventing the confusion caused by conflicting explanations.</p><h4>Reasoning Accuracy</h4><p><strong>Real-World Use Case</strong> — Imagine that, as a developer, you develop a GenAI-based mathematical tool that helps students get answers with step-by-step solutions to any mathematical problem, and your AI-based math tutor gives correct answers in the end, even with the wrong steps taken during calculations. In this type of case, reasoning accuracy is used as an evaluator where GenAI deployment performs shallow pattern matching rather than true reasoning and leads to improper steps with correct answers. Another use case is <strong>legal conversational systems</strong>, where a system is developed that gives well-structured, single-step questions, but end users pose multi-step, conditional, or edge-case queries. In such cases, the model may generate fluent but logically incorrect answers. The reasoning accuracy metric helps in detecting these failures by validating the correctness of the next intermediate logical steps, not just the final output generated by the GenAI system.</p><p><strong>Mathematical formula </strong>:</p><p>Reasoning accuracy is defined by evaluating whether the model’s logical inference process is able to generate the correct conclusions. One generally used formulation is stepwise accuracy, where each reasoning step is compared against a ground-truth chain of reasoning and represented by the following equation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*Ez6UfWmC_ii26fHiZDPoOQ.png" /></figure><p><strong>Applications of metrics —</strong> — This metric checked an intermediate logic during the process, not just the final output that makes it widely used in evaluting the decision-support systems, analytical assistants, autonomous agents, and RAG-based reasoning pipelines.</p><h4>Step-wise F1 (Chain-of-Thought)</h4><p><strong>Real-World Use Case </strong>— Consider a GenAI system developed to solve complex physics equations and generate answers with proper derivations. The system correctly provides the final derived equations, but this is partially due to skipped steps or proofs that contradict the actual derivations in the response. To analyze and evaluate these types of GenAI systems, a stepwise F1 score is used, which prevents contradictions in the proofs of equations and also assesses the correctness at each intermediate step, rather than only evaluating the final generated response of the GenAI system.</p><p><strong>Mathematical formula </strong>:<br> Step-wise F1 fornulated by extending the normal F1-score formula for action steps. Where each intermeidcated generated step is compared with a reference set of correct steps, and allowing partial credit for correctly executed reasoning step. According to which belwo are methods by which precision and recall are computed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*242qqwh73FJnPXveG-17nw.png" /></figure><p>Then Step-wise F1 score is calculated given by :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*E39bx7ar2AbZAzpiY3eZ4w.png" /></figure><p>This evlaution method rewards the models which produce accurate and complete sequences of intermedicate steps. <strong>Higher Step-wise F1 values indicate better performance.</strong></p><p><strong>Applications of metrics —</strong> Step-wise F1 is iscommonaly used in use cases such as chain-of-thought evaluation, planning and reasoning agents, code generation workflows, and process-driven LLM systems. In autonomous agents that interact with tools or APIs, this metric ensures that each action aligns with the intended plan.</p><h4>Robustness Score</h4><p><strong>Real-World Problem Use Case</strong> — Imagine you are developing a customer-facing chatbot that you tested on clean, well-formatted queries, resulting in high scores. However, when end users introduce spelling mistakes, slang, incomplete sentences, or mixed languages in queries, that starts causing performance to decline in production. This very common use case is where the robustness score comes into the evaluation methodology list before the system goes to production, as the robustness score checks the stability of the system under controlled perturbations or permutations of common errors. That helps in revealing the weaknesses where traditional accuracy-based evaluation metrics fail to capture.</p><p><strong>Mathematical formula </strong>:</p><p>The robustness score metric helps in evaluating how consistently a model performs when queries are given as input without changing their semantic meaning or in natural language. Perturbations passed in the query may include paraphrasing, synonym substitution, typos, noise injection, or adversarial prompts. It is defined as the ratio of performance under perturbation to baseline performance and given by the below equation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*4uXVN9G8cSncp-4gpnQ2Ng.png" /></figure><p>where ​S ‘base k’ is the evaluation score under the k-th perturbation and S ‘original’ is the score on clear input query. <strong>Higher robustness scores are better</strong>,that indicates the stable performance on differnet input variations.</p><p><strong>Applications of metrics —</strong> It is commonly used in industry level chat agents and SaaS portals that validate whether the model is impacted or not and introduce brittleness under edge cases in conversations with chat agents.</p><h4>Adversarial Resistance</h4><p><strong>Real-World Use Case </strong>— Consider a <strong>content moderation systems</strong>, in hwich system may score highly on collected toxic or safe datasets however not perofmred well, when users enter intentionally a obfuscate harmful content by using a misspellings, coded language, or manipulation in prompts in production.</p><p><strong>Mathematical formula </strong>:</p><p>Robustness evaluated the stability of the model under paraphrasing or noise queries. Adversarial resistance is evaluated by comparing the model’s performance on adversarially perturbed inputs against its performance on original inputs. A common formulation defined as resistance as the complement of the adversarial error rate and given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*C4iobgoOSUatUUdpuQmvnA.png" /></figure><p>It also can be defined as the ratio of adversarial accuracy to clean accuracy,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*x2MLch14sfARv3MeIgOH9Q.png" /></figure><p>where S ‘adversarial’ represents performance under adversarial prompts.<strong>Higher adversarial resistance scores are better</strong>, that indicates the stronger resilience against suspiosus manipulation in the query.</p><p><strong>Applications of metrics — </strong>Adversarial resistance is useful for use cases where security and safety are as essential as generations of responses, such as content updates, financial advisor systems, self-driving agents for cars, and GenAI APIs that the public can access. In commeircal deplyments this evaluation helps in preventing data leakage, misuse, and policy violations after deployment. For consumer-facing AI systems, these metrics are key to maintaining trust and regulatory compliance by ensuring that real-world adversarial behaviour does not compromise system integrity. Overall, adversarial resistance bridges the gap between controlled development testing and unpredictable real-world usage by stress-testing models against hostile inputs rather than idealised scenarios.</p><h4>Diversity (Self-BLEU)</h4><p><strong>Real-World Use Case </strong>— Imagine a creative AI content generation platform where users can create a marketing campaign, product descriptions, or social media posts for better engagement. Models may use relevance and fluency metrics during testing; however, in production, they generate closely related responses even on different input prompts. That impacts the engagement and perceived quality of content posted. This evaluation metric directly handles this issue by analysing the responses and quantifying output diversity, which helps in revealing degeneration issues that traditional accuracy metrics fail to overtake.</p><p><strong>Mathematical formula </strong>:<br>Self-BLEU measures variety by calculating the BLEU scores between the responses generated by the model itself rather than against reference documents. Assume a set of generated responses represented as {G1, G2, …, GN}, where each output is treated as a hypothesis while the remaining outputs act as references:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*klY-UeY9aPclU9nNduQWDQ.png" /></figure><p><strong>The Lower Self-BLEU values indicate a better score, signifying</strong> greater diversity and reduced repetition.</p><p><strong>Applications of metrics — </strong>This evaluation metric is popularly used in creative content generation, script-generating systems, and generative recommendation engines for products. In deployments, metrics detect the model collapse, situations where models converge on a narrow set of safe but identical outputs.</p><h4>Novelty</h4><p><strong>Real-World Use Case </strong>— Consider idea generation and brainstorming tools, where the model needs to be innovative, but while it may generate well on test prompts, in deployment it generates identical ideas and offers only a few real innovative ideas. This evaluation metric explicitly handles these cases by evaluating how many new distinct responses are generated from the training data. It exposes the performance drops at the testing level, which stops further deployment of improper idea generation models, as if ideas were used by a marketing content generation firm; then they must avoid plagiarism that hampers their engagements, which they don’t bear in deployments.</p><p><strong>Mathematical formula </strong>:<br>Novelty is computed by quantifying the dissimilarity between generated responses and training content, such as training samples, prior responses, or reference context texts. It is formulated as it computes novelty using embedding-based distance given by the below equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*8ofInzdl7ktArscpAbH7xg.png" /></figure><p>In the above expression, ‘e base g​’ is the embedding of the generated response, and ‘e base r’ represents embeddings of the reference corpus. The higher the dissimilarity, the more novel the content. Another approach is the n-gram-based method, where novelty is computed as the ratio of n-grams not seen in the reference set. In both approaches, <strong>the higher the value of novelty scores, the better; that indicates more original and less identical responses.</strong></p><p><strong>Applications of metrics — </strong>This evaluation metric is commonly used in creative idea generation AI systems and generative or marketing content platforms. In commercial deployments, it helps to balance innovation with relevance, preventing models from collapsing into derivative responses over time. Novelty is very useful in tracking long-term performance drift, where model performance gradually becomes less innovative and starts generating the same responses despite stable development scores.</p><h4>Readability</h4><p><strong>Real-World Use Case </strong>— Imagine a healthcare and medical information system where models generate factually correct responses with high accuracy; however, the actual application of supporting patients, still struggles to help them understand complex medical terminology. This evaluation metric addresses this issue by evaluating how easily users can consume and take decisions based on model-generated text and unveiling issues that traditional performance metrics suffer from.</p><p><strong>Mathematical formula </strong>:<br>Readability is calculated using linguistic formulae that analyse the cognitive effort required to understand a text. One commonly used formulation is based on the Flesch Reading Ease Score, defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*8hulDoz5il4UYP-TU1JwpA.png" /></figure><p><strong>Higher the vlaue of FRE scores indicate easier-to-read text</strong>. Another generally used evaluation method is based on the Flesch–Kincaid Grade Level. However, <strong>if the Flesch–Kincaid grade level metric formulation is used, then lowering the value of scores may be better.</strong> Actual interpretation of direction is essential when applying readability evaluations and type formulation used.</p><p><strong>Applications of metrics — </strong>Readability metrics ensure that the generated response’s language is according to the audience level or not, so it is commonly used in customer-facing AI systems, educational platforms, and healthcare assistants. The readability evaluation method complements accuracy and relevance by focusing on the user-centred quality standards.</p><h4><strong>Latency</strong></h4><p><strong>Real-World Use Case </strong>— Consider a real-time customer support chatbot. A model may generate correct responses in the testing phase; however, the same chatbot starts user drop-off in production whenever the response generation time increases under peak hours. This evaluation metric addresses this issue by computing the responsiveness under realistic deployment environments and by capturing performance degradation where accuracy scores change based on the performance in the real world.</p><p><strong>Mathematical formula </strong>:<br>Latency is computed using the time taken for a model to generate a response after receiving an input query from the user. It is basically computed as the difference in wall clock time between request and response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*-M7gMFAxu5MfaPg_MESUAQ.png" /></figure><p>In deployment systems, latency is generally summarised using statistical analyses such as <strong>mean latency</strong>, <strong>median (P50)</strong>, and <strong>tail latency (P95 or P99)</strong> to record the worst-case delays in peak hours. <strong>Lower the value of latency metrics; the better the values are</strong>.</p><p><strong>Applications of metrics — </strong>The latency metric is generally used in mostly all types of applications where LLM is deployed as a backend, as the time it takes for the model to generate a response is always a key point for integrating the model with any other source system.</p><h4>Token Efficiency</h4><p><strong>Real-World Use Case </strong>— Token efficiency helps in commercial RAG-based chatbots integrated with SaaS services, as the costs of each token are high. Commonly, these types of implementation models achieve high functional scores during development but become impractical or costly after deployment, as these models may generate correct responses during testing but consume excessive tokens due to the processing of long prompts and verbose reasoning, which leads to high operational costs and latency in the deployments. This metric handles these real use case declines by levelling issues that are invisible in development or testing phase accuracy evaluations.</p><p><strong>Mathematical formula </strong>:<br>Token efficiency is defined as the ratio of a task-specific score, such as accuracy, relevance or another, to the total number of query and response tokens used, which is given by the below equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*rXJQ9z9TFp8eIKnR-33mbg.png" /></figure><p>where S ‘base task​’ shows the task-specific score and T ‘base used’​ is the total number of input and output tokens consumed. In token efficiency, the <strong>higher the values of token efficiency, the better the model is</strong>, as it indicates the model delivers strong performance with fewer tokens and lower computational cost.</p><p><strong>Applications of metrics — </strong>This metric is commonly used in enterprise RAG-based chatbots where LLM deployments are <strong>cost-sensitive,</strong> including SaaS platforms. In a deployed environment, continuous monitoring of tokens helps to optimise prompt design, context truncation, and reasoning verbosity.</p><h4>Cost per Query</h4><p><strong>Real-World Use Case </strong>— Similar to token efficiency, this metric also helps in controlling the cost associated with deployments of LLM-based applications. Consider an <strong>enterprise knowledge assistant</strong> model which may provide correct responses during the training/testing phase; however, in production, the combination of large context chunks and reasoning responses significantly increases per-query cost during scaling.</p><p><strong>Mathematical formula </strong>:</p><p>Cost per query is defined as the ratio of total tokens used in inference cost to the total number of tokens used in the queries; it is given by a simple expression as mentioned below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*FaCaq-lnx3F1rGVr51Dm2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*KpBUWy3nmPqXx2pUB13j-g.png" /></figure><p><strong>Applications of metrics — </strong>Similar to token efficiency, this metric is commonly used in enterprise-scale deployments of LLM-based solutions that help in tracking and monitoring of the cost associated with each query and help in decisions around prompt shortening, model selection, caching strategies, and fallback architectures. It allows developers to make the system scalable with a <strong>lower per-query cost per customer</strong>.Basically it helps in tracking of fincial efeciney of the deplyed mdoel.Basically it helps in tracking the financial efficiency of the deployed model.</p><h4>Human Preference Score</h4><p><strong>Real-World Use Case </strong>— Consider a scenario where <strong>a conversational chatbot</strong> generates responses that are engaging, helpful and factually correct during testing/training, but in production, the same deployed models start generating vague responses; those do not help the users or give instructions; those do not give the real-world results to the users. This can be monitored by using human preference scores and helps to make the deployed model as close as possible to the human expectations. It gives two models generating responses to the same query, but the user always prefers one.</p><p><strong>Mathematical formula :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*tL5fXb-H61QZo0wxnjOSCQ.png" /></figure><p>It is a type of rating given by users during the conversations, which is defined as the ratio of the number of preferred outputs to the total comparisons. Further, that score may be normalised using ranking algorithms such as Bradley–Terry or Elo. And <strong>the higher the value of the score, the better the model, as user alignment is visible through the score.</strong></p><p><strong>Applications of metrics — </strong>This metric helps in recording the users’ preferences and satisfaction levels after using the GenAI applications, so it is primarily used in reinforcement learning from human feedback (RLHF), product evaluation, and A/B testing. In deployed systems this metric helps in fine-tuning toward responses that users find more helpful, polite, concise, or engaging.</p><h4>Conclusion</h4><p>In the current era of developments of GenAI applications, the minds of technology users and developers shift from the thought of “Can we build it?” to “Can we trust it?” That motivates me to explore the evaluation metrics and prepare this post to explore multiple metrics, as not a single metric is able to tell the whole story. A robust evaluation pipeline is no longer optional; it must be there for the foundation of enterprise-ready GenAI solutions. This post explores various evaluation metrics in LLM in continuation of the previous post and allows readers to understand their models more effectively before putting them into the production pipeline that reduces the overall fallback or untrust on LLM-based systems.</p><p>Thank you readers !! Stay connected !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2df98588f42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Oh, does the selection of inappropriate evaluation metrics lead to complaints from users?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sumitkrsharma-ai.medium.com/oh-does-the-selection-of-inappropriate-evaluation-metrics-lead-to-complaints-from-users-02ba6d471521?source=rss-246685f156aa------2"><img src="https://cdn-images-1.medium.com/max/1280/1*ksijhwlnbm1qOtGG6OpdFQ.png" width="1280"></a></p><p class="medium-feed-snippet">Developers working on generative model applications around the LLM, once in a cycle of development, reach a point where the model seems to&#x2026;</p><p class="medium-feed-link"><a href="https://sumitkrsharma-ai.medium.com/oh-does-the-selection-of-inappropriate-evaluation-metrics-lead-to-complaints-from-users-02ba6d471521?source=rss-246685f156aa------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sumitkrsharma-ai.medium.com/oh-does-the-selection-of-inappropriate-evaluation-metrics-lead-to-complaints-from-users-02ba6d471521?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/02ba6d471521</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Mon, 19 Jan 2026 03:09:28 GMT</pubDate>
            <atom:updated>2026-01-19T03:09:28.487Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Large Language Model Pretraining & Alignment: A Practical, Intuitive, and Technical Beginner’s…]]></title>
            <link>https://sumitkrsharma-ai.medium.com/large-language-model-pretraining-alignment-a-practical-intuitive-and-technical-beginners-f56cf158fe11?source=rss-246685f156aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/f56cf158fe11</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sumitkrsharma]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 10:41:34 GMT</pubDate>
            <atom:updated>2025-12-29T10:41:34.733Z</atom:updated>
            <content:encoded><![CDATA[<h3>Large Language Model Pretraining &amp; Alignment: A Practical, Intuitive, and Technical Beginner’s Guide</h3><p>Have you wondered how these popular Large Language Models (LLMs), such as GPT by OpenAI, LLaMA by Meta, Claude by Anthropic, and Qwen by Alibaba, are created? These all model are not created in a one-shot process. Development of these required an evolution through two major phases:</p><ol><li><strong>Pretraining</strong> — At this stage the model starts learning language, facts, and patterns from massive data of images fed into wheat corp.</li><li><strong>Alignment &amp; Post-training</strong> — LLMs must need to learn how to act usefully, safely, and according to human intent based on prompts.</li></ol><p>This is about exploring the pretraining and alignment of LLMs concisely, from their definition, sample implementation, mathematical concepts and other help to get deeper insights into the proper working of these models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCBzqaRLcZJOwnK0VNx7kQ.png" /></figure><h4>PRETRAINING</h4><h4>1. Large-Scale Corpus Construction</h4><p><strong>Real-World Analogy — </strong>It is similar to how we humans learnt various patterns by accessing them. We assess training a human by giving them access to <strong>libraries, newspapers, forums, and textbooks</strong> from across the world.</p><p><strong>Technical Definition — </strong>LLMs are trained over large-sized textual corpuses; this step is called ‘large-scale corpus construction’, which involves collecting and normalising <strong>trillions of tokens</strong> from various data sources such as web crawls, books, Wikipedia, research papers, and source code.</p><p><strong>Mathematical Intuition</strong> — The primary goal of LLM models is to learn parameters (theta) by maximising the likelihood of observing text sequences in the input corpus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*dp7vnFW5I4Y4AZE9TI0z_w.png" /></figure><p><strong>Sample python code snippet :</strong></p><pre>data_sources = [&quot;web&quot;, &quot;books&quot;, &quot;wiki&quot;, &quot;code&quot;]<br>corpus = []<br>for src in data_sources:<br>    corpus.extend(load_clean_text(src))</pre><h4>Advantages</h4><ul><li>Wide knowledge base from the corpus trained on.</li><li>Better zero-shot and few-shot learning capabilities.</li><li>Having reasoning abilities in AI.</li></ul><h4>Limitations</h4><ul><li>Carrying bias in response generation.</li><li>Content generated without prior concern of the original author.</li><li>High computational cost.</li></ul><h4>Applications</h4><ul><li>Foundational models, such as GPT and Claude, are examples of AI-based multilingual assistants.</li><li>AI-based multilingual assistants.</li><li>AI-based software engineering or developer-coded LLMs.</li></ul><h4>2. Data Deduplication &amp; Filtering</h4><p><strong>Real-World Analogy</strong> — Assume you want to create a repository of documents in which there are no duplicate textbooks available and no spam notes; this can be done using data deduplication and filtering.</p><p><strong>Technical Definition</strong> — The deduplication process removes closely related samples, and filtering eliminates the low-quality, toxic, or any irrelevant content.</p><p><strong>Mathematical Intuition </strong>— Mathematically, it only reduces over-representation and memorisation by enhancing the entropy of the training distribution of datasets.</p><h4><strong>Sample python code snippet :</strong></h4><pre>from simhash import Simhash<br>seen = set()<br>def keep(text):<br>    h = Simhash(text).value<br>    if h in seen:<br>        return False<br>    seen.add(h)<br>    return True</pre><h4>Advantages</h4><ul><li>It offers better generalisation in the developed model system.</li><li>Duplication helps reduce the hallucinations.</li><li>Filtering and removal of duplicates help in reducing the privacy risks.</li></ul><h4>Limitations</h4><ul><li>Filtering and deprecation applied to the pipeline may suffer from more false positives.</li><li>During deduplications there is also a chance of losing rare data.</li></ul><h4>Applications</h4><ul><li>Offers a developed pipeline of safety-sensitive large language models.</li><li>As having a lower risk of privacy, which makes it useful in legal and healthcare AI pipelines.</li></ul><h4>3. Curriculum Learning</h4><p><strong>Real-World Analogy </strong>— It’s like a student learns addition by counting numbers on their hands; once they do, they move on to subtraction, multiplication, and other complex mathematical operations such as calculus and algebra.</p><p><strong>Technical Definition </strong>— The curriculum learning technique trains the model in a similar way by starting with simple, easy examples and then gradually moving to complex, harder tasks instead of a random approach or trial-and-error methods, which makes learning more stable and more accurate for real-world problems.</p><p><strong>Mathematical Intuition</strong> — Mathematically it optimises a simpler loss landscape initially by minimising easy losses.</p><h4>Sample python code snippet :</h4><pre>data = sorted(dataset, key=lambda x: x.difficulty)<br>for sample in data:<br>    train(sample)</pre><h4>Advantages</h4><ul><li>It makes it converge faster.</li><li>This learning method improves the reasoning ability of the trained model.</li></ul><h4>Limitations</h4><ul><li>In models trained with curriculum learning, having difficulty scoring is subjective.</li></ul><h4>Applications</h4><ul><li>This method is used in applications of LLMs where mathematical reasoning is a key component.</li><li>LLM-based code generation models are also an important application of curriculum learning.</li></ul><h4>4. Multi-Epoch vs Single-Epoch Training</h4><p><strong>Real-World Analogy</strong> — It is very similar to the scenario of reading your notes just before the exam once vs. reading and revising them multiple times before the exam.</p><p><strong>Technical Definition</strong> — Simply means in the single-epoch training method data is seen once by the model as compared to a model with multi-epoch training, which repeats multiple iterations over the dataset.</p><p><strong>Mathematical Intuition</strong> — In this approach the gradient is updated repeatedly to improve the fit, but this approach also has the risk of overfitting while performing multiple iterations.</p><h4>Sample python code snippet :</h4><pre>for epoch in range(E):<br>    for batch in dataloader:<br>        train(batch)</pre><p>Technical Definition</p><p>Simply means in the single-epoch training method data is seen once by the model as compared to a model with multi-epoch training, which repeats multiple iterations over the dataset.</p><p>Mathematical Intuition</p><p>In this approach the gradient is updated repeatedly to improve the fit, but this approach also has the risk of overfitting while performing multiple iterations.</p><h3>Advantages</h3><ul><li>Model trained with this apporach generally having higher accuracies scores.</li><li>Models having capability of better domain adaptation.</li></ul><h3>Limitations</h3><ul><li>Models have memorization risk</li><li>As mentioned above risk of overfitting that makes it training instability.</li></ul><h3>Applications</h3><ul><li>Used for development of domain-specific LLMs.</li><li>Good for LLM based instruction mannual tuning.</li></ul><h4>ALIGNMENT &amp; POST-TRAINING</h4><h4>5. Supervised Fine-Tuning (SFT)</h4><h4>Real-World Analogy</h4><p>It works simlar as kindergarden studnets answers corrected by teachers directly.</p><h4>Technical Definition</h4><p>In implemnetation stretergy fine-tuning of LLMs requried on human written queries and their answeres from the dataset.</p><h4>Mathematical Intuition</h4><p>It learns mathematically by using the cross-entropy loss between model output and target response.</p><h4>Sample python code snippet :</h4><pre>loss = cross_entropy(model(prompt), target_response)</pre><h4>Advantages</h4><ul><li>These types of models having strong instruction following capabilties And optimization is stable during training.</li></ul><h4>Limitations</h4><ul><li>Time comsuming manual labeling requring of training and limited new novalty in responses.</li></ul><h4>Applications</h4><ul><li>Chatbots and Question-answering systems common applications.</li></ul><h4>6. RLHF (PPO, DPO, RLAIF)</h4><h4>Real-World Analogy</h4><p>This type of training allow model to learn by thinking about <strong>reward as postive and punishment as negtaive</strong>.</p><h4>Technical Definition</h4><p>It works simialr to reinforcement learning, where it learns from human feedback optimizes outputs using preference based rewards, whether move positive or negative directions.</p><h4>Mathematical Intuition</h4><p>Maximize expected reward:</p><p>[\max_\theta \mathbb{E}[R(y|x)]]</p><h4>Sample python code snippet :</h4><pre>reward = reward_model(output)<br>loss = -reward * log_prob(output)</pre><h4>Advantages</h4><ul><li>This method of training likely aligns with human values and helps in reduces unsafe outputs.</li></ul><h4>Limitations</h4><ul><li>In few traning methods reward hacking makes it not proper trade of between bais and varience of LLM . Another issue is high cost of training as need regualr feedbacks for better responses.</li></ul><h3>Applications</h3><ul><li>Generel purpose applications such as ChatGPT and Claude uses this method.</li></ul><h4>7. Preference Modeling</h4><h4>Real-World Analogy</h4><p>As generelly we alsways select most perfect words or senetence based on any context when we have more than one words, snetecne or thoughst in partyicualr situtaion in this type of training also best out of one is selected from the twor choices.</p><h4>Technical Definition</h4><p>It is the process of learning based on the utility function from human (or AI) feedbacks for preference comparisons basically pairwise rankings over model responses created that guide policy optimization, and enable LLMs to align generation behavior with subjective quality evalution metrics such as helpfulness, harmlessness, and coherence, beyond likelihood-based objectives in LLMs.</p><h4>Mathematical Intuition</h4><p>Bradley–Terry model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*3wexjKya1r4Jx5Xj6rnAYA.png" /></figure><p>Details of equations symbols summaized in the below table :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3hJAuqZtgt55Y-dpLVBMg.png" /></figure><h4>Sample python code snippet :</h4><pre>loss = -torch.log(torch.sigmoid(score_a - score_b))</pre><h4>Advantages</h4><ul><li>Scalable and lower annotation cost make it preferable than other methods.</li></ul><h4>Limitations</h4><ul><li>These methods of training have subjective preferences</li></ul><h4>Applications</h4><ul><li>Response ranking and alignment pipelines are primary applications of this method.</li></ul><h4>8. Constitutional AI</h4><h4>Real-World Analogy</h4><p>This type of model training follows a method simialr to doctors as they follows <strong>medical code of ethics not w</strong>aiting for feedback after response of paitemt on medince prescriptions , they relies on a written set of principles that if medince is not hamring and ake imprvemnet in paitents health continue the drugs if harming then stop as ealirets as possible, as first rule is do not harm the paitent.</p><h4>Technical Definition</h4><p>Training of models depends upon self-critique and revised responses based on predefined ethical rules, not reiles over the human or AI feedbacks for every response.</p><h4>Sample python code snippet :</h4><pre>critique = model.check(output, rules)<br>final_answer = model.rewrite(output, critique)</pre><h4>Advantages</h4><ul><li>Less effrot reuqried as no human or AI feedback needed and consistent alignment with rules.</li></ul><h4>Limitations</h4><ul><li>Rule ambiguity hamper the training proceses.</li></ul><h4>Applications</h4><ul><li>Enterprise LLMs and Claude-style assistants are real world example of this type of AI models.</li></ul><h4>9. Safety Tuning</h4><h4>Real-World Analogy</h4><p>Simialr to <strong>training a kid with defined boundaries</strong> <em>what not to say or do</em> such as no abusive language, no dangerous instructions so that kid avoid harmful behavior even without constant supervision.</p><h4>Technical Definition</h4><p>It is a post training process in which an LLM is optimized using a supervised fine-tuning, preference learning, and defined rules constraints to <strong>suppress unsafe generations</strong>, that makes the model ensuring compliance with safety policies that minimally impacting performance of the model.</p><h4>Mathematical Intuition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*aY74rGRoaV6aDZZ7mAP0Jg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOakKop6STYKcAVX4McFhA.png" /></figure><h4>Sample python code snippet :</h4><pre>loss = task_loss + lambda_ * safety_penalty</pre><h4>Advantages</h4><ul><li>Trustworthy AI and regulatory compliance.</li></ul><h4>Limitations</h4><ul><li>Over-censorship risk in this post training method.</li></ul><h4>Applications</h4><ul><li>Healthcare AI and financial copilots are exmaples of it.</li></ul><h4>LLM Training Pipeline</h4><pre>Raw Data → Cleaning → Pretraining → SFT → RLHF / DPO → Safety Tuning → Deployed LLM</pre><h4>Conclusion</h4><p>Pretraining of LLMs that are equipped with general language understanding is allowing them to learn patterns extracted from large-scale datasets. This establishes a foundation for downstream tasks while also highlighting challenges related to efficiency and bias in scaling LLM models. This post provides summaries of pretraining methods. This post gives a real-world analogy, a technical definition, and brief applications of each type of pretraining method that make it a quick guide for pretraining in LLMs. I hope the post meets the readers’ expectations; with this, we conclude.</p><p>Thank you readers !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f56cf158fe11" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>