<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Drraghavendra on Medium]]></title>
        <description><![CDATA[Stories by Drraghavendra on Medium]]></description>
        <link>https://medium.com/@drraghavendra99?source=rss-9b994a01c790------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*vvLDfa2qNj7O5Y_o.jpg</url>
            <title>Stories by Drraghavendra on Medium</title>
            <link>https://medium.com/@drraghavendra99?source=rss-9b994a01c790------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:38:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@drraghavendra99/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Redefining the Future of AI: A Deep Dive into Traditional AI, Agentic Systems, RAG, and Agentic RAG.]]></title>
            <link>https://medium.com/codetodeploy/redefining-the-future-of-ai-a-deep-dive-into-traditional-ai-agentic-systems-rag-and-agentic-rag-d4763f5aa610?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4763f5aa610</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 10:39:02 GMT</pubDate>
            <atom:updated>2026-11-11T14:57:57.254Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>Artificial intelligence has evolved rapidly from narrowly scoped, data-driven models into systems that can reason, retrieve information, and act autonomously. Four terms now dominate conversations about modern AI architectures: <strong>Traditional AI</strong>, <strong>Agentic AI</strong>, <strong>Retrieval-Augmented Generation (RAG)</strong>, and <strong>Agentic RAG</strong>. Although they are frequently presented as a linear progression, they are better understood as combinations of two independent capabilities — whether a system can <em>fetch external knowledge</em> and whether it can <em>plan and act in loops</em>. Traditional AI does neither; RAG adds knowledge retrieval; Agentic AI adds autonomous reasoning and tool use; and Agentic RAG combines both. Understanding where these approaches overlap and where they diverge is essential for choosing the right architecture, because greater capability also brings greater cost, complexity, and unpredictability. This comparison examines each concept across key dimensions, highlights their shared foundations and critical differences, and illustrates how they behave when applied to the same real-world task.</p><blockquote><em>💥 </em><strong><em>Master AI &amp; Tech Skills </em></strong><br> 🔥 Up to 5<strong>0% OFF</strong> — Limited-Time Offer<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*UBJVGwkaIh7nVqHl6Vyh3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*DWAPODVe3ypgh0b9NJaRbg.png" /><figcaption>Depiction and Portraying of Comparision Table</figcaption></figure><h3>2. Key Similarities</h3><p><strong>All four:</strong></p><ul><li>Are built on machine learning foundations and ultimately produce outputs from inputs.</li><li>Require good data (training data, documents, or tool results) — quality in = quality out.</li><li>Need evaluation, monitoring, and governance in production.</li></ul><p><strong>Agentic AI &amp; Agentic RAG</strong> (the “agentic” pair):</p><ul><li>Use reasoning loops (e.g., ReAct, plan-and-execute).</li><li>Decide dynamically what to do next rather than following a fixed pipeline.</li><li>Can call tools/APIs and execute multi-step tasks.</li><li>Are non-deterministic and harder to test.</li></ul><p><strong>RAG &amp; Agentic RAG</strong> (the “retrieval” pair):</p><ul><li>Ground LLM outputs in external, updatable knowledge.</li><li>Separate <em>knowledge</em> (index) from <em>reasoning</em> (LLM), so facts can be refreshed without retraining.</li><li>Reduce hallucination and enable citations.</li></ul><p><strong>Traditional AI &amp; RAG</strong> (the “fixed-pipeline” pair):</p><ul><li>Follow a predetermined sequence of steps.</li><li>Produce one output per request.</li><li>Are relatively predictable, cheaper, and easier to debug.</li></ul><p><strong>Agentic AI &amp; RAG</strong> (the “LLM-extension” pair):</p><ul><li>Both extend a base LLM to overcome its limits — RAG fixes <em>knowledge gaps</em>, agents fix <em>action/reasoning gaps</em>.</li></ul><h3>3. Key Dissimilarities</h3><h4>Traditional AI vs. the other three</h4><ul><li><strong>Static vs. dynamic:</strong> Traditional AI is frozen at deployment; the others can change behavior or knowledge without retraining.</li><li><strong>Narrow vs. general:</strong> Traditional models do one task; the others use general-purpose LLMs that handle many tasks.</li><li><strong>No language reasoning:</strong> Traditional AI maps inputs to labels/values; it doesn’t “reason” in natural language.</li><li><strong>Predictable vs. probabilistic behavior:</strong> Traditional AI gives the same output for the same input; LLM-based systems may not.</li></ul><h4>Agentic AI vs. RAG</h4><ul><li><strong>Autonomy vs. grounding:</strong> Agentic AI is about <em>acting and deciding</em>; RAG is about <em>knowing</em>.</li><li><strong>Loop vs. pipeline:</strong> Agents iterate until a goal is met; RAG runs once.</li><li><strong>Breadth of tools:</strong> Agents can use any tool (code execution, APIs, browsers); RAG uses only a retriever.</li><li><strong>Risk profile:</strong> Agents can take real-world actions (send emails, make purchases) — higher risk; RAG only answers questions.</li></ul><h4>RAG vs. Agentic RAG</h4><ul><li><strong>Fixed retrieval vs. adaptive retrieval:</strong> RAG always retrieves once, up front. Agentic RAG decides <em>if</em> retrieval is needed, <em>which</em> source to query, and can retrieve <em>multiple times</em>.</li><li><strong>No self-correction vs. self-correction:</strong> Agentic RAG can judge retrieved results as insufficient and reformulate the query.</li><li><strong>Single source vs. multi-source routing:</strong> Agentic RAG can route a question to the right database, API, or web search.</li><li><strong>Cost:</strong> Agentic RAG can be several times more expensive and slower.</li></ul><h4>Agentic AI vs. Agentic RAG</h4><ul><li><strong>Agentic RAG is a subset/specialization</strong> of Agentic AI where retrieval is a first-class tool and knowledge grounding is a primary concern.</li><li>Not every agent needs retrieval (e.g., a code-refactoring agent); every Agentic RAG system is an agent.</li></ul><p>4. How Above <strong>Traditional AI</strong>, <strong>Agentic AI</strong>, RAG: <strong>Retrieval-Augmented Generation </strong>and <strong>Agentic RAG are related</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cflcR79fvgb8yjd5JaQkBw.png" /></figure><p>Think of it as two independent “upgrades”:</p><ul><li><strong>Retrieval axis:</strong> Does the system fetch external knowledge? (No → Yes)</li><li><strong>Agency axis:</strong> Does the system plan and act in loops? (No → Yes)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*ype7Nhfc8e-VsYn0GgWN6A.png" /></figure><h3>5. One Scenario, Four Approaches</h3><p>Task: “Which of our suppliers had the most late deliveries last quarter, and draft an email to them.”</p><ul><li>Traditional AI: A trained model could predict “likelihood of late delivery” per supplier — but cannot answer the question or write the email.</li><li>RAG: Retrieves delivery reports from the index and generates an answer — but may miss data split across multiple documents and cannot send anything.</li><li>Agentic AI: Plans: query the ERP API → aggregate → identify worst supplier → draft email via email tool. Works, but may hallucinate contract terms it lacks.</li><li>Agentic RAG: Plans the same steps <em>and</em> retrieves the supplier’s contract and past correspondence to make the email accurate and policy-compliant, verifying facts before drafting.</li></ul><h3>6. Important Caveats</h3><ol><li>“Continuous learning” is often overstated. In most Agentic RAG systems, the model weights do <em>not</em> change. “Learning” usually means updated indexes, memory stores, or feedback logs — not true model improvement.</li><li>More capability ≠ always better. For a fixed, well-defined problem (e.g., spam classification), Traditional AI is cheaper, faster, and more reliable than any agent.</li><li>Agentic systems carry new risks: runaway loops, unexpected tool actions, higher cost, and harder evaluation. Guardrails and human-in-the-loop checks are essential.</li><li>Hybrids are common. Real systems often use Traditional AI models <em>as tools</em> inside an agent (e.g., an agent calling a fraud-score model).</li></ol><p>7. Advanced Python Program</p><pre>from __future__ import annotations<br><br>import argparse<br>import hashlib<br>import json<br>import logging<br>import os<br>import re<br>import time<br>from collections import defaultdict<br>from dataclasses import dataclass, field<br>from datetime import date, datetime, timedelta<br>from decimal import Decimal<br>from functools import lru_cache<br>from pathlib import Path<br><br>import boto3<br>import numpy as np<br><br>logging.basicConfig(level=logging.INFO, format=&quot;%(asctime)s %(levelname)s %(name)s: %(message)s&quot;)<br>log = logging.getLogger(&quot;run&quot;)<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 0. CONFIGURATION<br># ═══════════════════════════════════════════════════════════════════════════<br>@dataclass(frozen=True)<br>class Settings:<br>    region: str = os.getenv(&quot;AWS_REGION&quot;, &quot;us-east-1&quot;)<br>    bucket: str = os.getenv(&quot;S3_BUCKET&quot;, &quot;&quot;)<br>    docs_prefix: str = &quot;knowledge/&quot;<br>    # Keras / SageMaker<br>    sm_role_arn: str = os.getenv(&quot;SM_ROLE_ARN&quot;, &quot;&quot;)<br>    sm_endpoint: str = os.getenv(&quot;SM_ENDPOINT&quot;, &quot;&quot;)<br>    local_model_dir: str = os.getenv(&quot;LOCAL_MODEL_DIR&quot;, &quot;./artifacts/late_delivery_model&quot;)<br>    # Bedrock<br>    llm_model_id: str = os.getenv(&quot;LLM_MODEL_ID&quot;, &quot;anthropic.claude-3-5-sonnet-20240620-v1:0&quot;)<br>    embed_model_id: str = os.getenv(&quot;EMBED_MODEL_ID&quot;, &quot;amazon.titan-embed-text-v2:0&quot;)<br>    embed_dim: int = 1024<br>    # OpenSearch Serverless (empty =&gt; local numpy store)<br>    aoss_host: str = os.getenv(&quot;AOSS_HOST&quot;, &quot;&quot;)<br>    aoss_index: str = &quot;supplier-knowledge&quot;<br>    local_store: str = &quot;./artifacts/vector_store.json&quot;<br>    # DynamoDB / SES<br>    ddb_deliveries: str = os.getenv(&quot;DDB_DELIVERIES&quot;, &quot;&quot;)<br>    ddb_memory: str = os.getenv(&quot;DDB_MEMORY&quot;, &quot;&quot;)<br>    ses_sender: str = os.getenv(&quot;SES_SENDER&quot;, &quot;procurement@example.com&quot;)<br>    # Agent guardrails<br>    max_steps: int = 12<br>    max_tokens_per_call: int = 2048<br>    token_budget: int = 60_000<br>    temperature: float = 0.1<br>    allowed_tools: tuple = (&quot;query_erp&quot;, &quot;predict_late_delivery&quot;, &quot;retrieve_documents&quot;, &quot;draft_email&quot;)<br><br><br>S = Settings()<br>Path(&quot;./artifacts&quot;).mkdir(exist_ok=True)<br><br><br>@lru_cache(maxsize=None)<br>def client(name: str):<br>    &quot;&quot;&quot;Lazily-created, cached boto3 clients (cheap to call repeatedly).&quot;&quot;&quot;<br>    return boto3.client(name, region_name=S.region)<br><br><br>@lru_cache(maxsize=None)<br>def resource(name: str):<br>    return boto3.resource(name, region_name=S.region)<br><br><br>def json_safe(o):<br>    if isinstance(o, Decimal):<br>        return float(o)<br>    if isinstance(o, dict):<br>        return {k: json_safe(v) for k, v in o.items()}<br>    if isinstance(o, (list, tuple)):<br>        return [json_safe(v) for v in o]<br>    if isinstance(o, (np.floating, np.integer)):<br>        return o.item()<br>    return o<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 1. TRADITIONAL AI — Keras late-delivery classifier<br># ═══════════════════════════════════════════════════════════════════════════<br>NUMERIC = [&quot;order_qty&quot;, &quot;lead_time_days&quot;, &quot;distance_km&quot;, &quot;historical_late_rate&quot;, &quot;month&quot;]<br>CATEGORICAL = [&quot;supplier_id&quot;, &quot;shipping_mode&quot;]<br>TARGET = &quot;is_late&quot;<br><br><br>def synthetic_deliveries(n: int = 20_000, seed: int = 42):<br>    &quot;&quot;&quot;Realistic synthetic data so the file runs end-to-end without a warehouse.&quot;&quot;&quot;<br>    import pandas as pd<br>    rng = np.random.default_rng(seed)<br>    suppliers = [f&quot;SUP-{i:03d}&quot; for i in range(1, 41)]<br>    sup_bias = dict(zip(suppliers, rng.beta(2, 8, len(suppliers))))<br>    df = pd.DataFrame({<br>        &quot;supplier_id&quot;: rng.choice(suppliers, n),<br>        &quot;shipping_mode&quot;: rng.choice([&quot;road&quot;, &quot;air&quot;, &quot;sea&quot;], n, p=[0.6, 0.2, 0.2]),<br>        &quot;order_qty&quot;: rng.lognormal(5.5, 0.8, n).round(),<br>        &quot;lead_time_days&quot;: rng.integers(2, 45, n),<br>        &quot;distance_km&quot;: rng.gamma(2.0, 400, n).round(),<br>        &quot;month&quot;: rng.integers(1, 13, n),<br>    })<br>    df[&quot;historical_late_rate&quot;] = df[&quot;supplier_id&quot;].map(sup_bias) + rng.normal(0, 0.03, n)<br>    logit = (-2.2 + 4.0 * df[&quot;historical_late_rate&quot;] + 0.0006 * df[&quot;distance_km&quot;]<br>             - 0.03 * df[&quot;lead_time_days&quot;] + (df[&quot;shipping_mode&quot;] == &quot;sea&quot;) * 0.6<br>             + df[&quot;month&quot;].isin([11, 12]) * 0.5)<br>    df[TARGET] = (rng.random(n) &lt; 1 / (1 + np.exp(-logit))).astype(int)<br>    return df<br><br><br>def _tf_dataset(df, batch: int, train: bool):<br>    import tensorflow as tf<br>    feats = {c: df[c].values.astype(&quot;float32&quot;) for c in NUMERIC}<br>    feats.update({c: df[c].astype(str).values for c in CATEGORICAL})<br>    ds = tf.data.Dataset.from_tensor_slices((feats, df[TARGET].values.astype(&quot;float32&quot;)))<br>    if train:<br>        ds = ds.shuffle(min(len(df), 10_000), seed=42)<br>    return ds.batch(batch).cache().prefetch(tf.data.AUTOTUNE)<br><br><br>def build_keras_model(train_df, hidden=(128, 64, 32), dropout=0.3, lr=1e-3):<br>    &quot;&quot;&quot;Preprocessing lives INSIDE the model → zero train/serve skew.&quot;&quot;&quot;<br>    from tensorflow import keras<br>    from tensorflow.keras import layers<br><br>    inputs, encoded = {}, []<br>    for col in NUMERIC:<br>        inp = keras.Input(shape=(1,), name=col, dtype=&quot;float32&quot;)<br>        norm = layers.Normalization(name=f&quot;norm_{col}&quot;)<br>        norm.adapt(train_df[col].values.astype(&quot;float32&quot;).reshape(-1, 1))<br>        inputs[col], _ = inp, encoded.append(norm(inp))<br><br>    for col in CATEGORICAL:<br>        inp = keras.Input(shape=(1,), name=col, dtype=&quot;string&quot;)<br>        lookup = layers.StringLookup(num_oov_indices=1, name=f&quot;lookup_{col}&quot;)<br>        lookup.adapt(train_df[col].astype(str).values)<br>        vocab = lookup.vocabulary_size()<br>        emb = layers.Embedding(vocab, int(min(50, round(1.6 * vocab ** 0.56))))(lookup(inp))<br>        inputs[col], _ = inp, encoded.append(layers.Flatten()(emb))<br><br>    x = layers.Concatenate()(encoded)<br>    for i, u in enumerate(hidden):<br>        x = layers.Dense(u, name=f&quot;dense_{i}&quot;)(x)<br>        x = layers.BatchNormalization()(x)<br>        x = layers.Activation(&quot;swish&quot;)(x)<br>        x = layers.Dropout(dropout)(x)<br>    out = layers.Dense(1, activation=&quot;sigmoid&quot;, dtype=&quot;float32&quot;, name=&quot;late_probability&quot;)(x)<br><br>    model = keras.Model(inputs, out, name=&quot;late_delivery_classifier&quot;)<br>    model.compile(<br>        optimizer=keras.optimizers.AdamW(learning_rate=lr, weight_decay=1e-4),<br>        loss=keras.losses.BinaryFocalCrossentropy(gamma=2.0),<br>        metrics=[keras.metrics.AUC(name=&quot;auc&quot;), keras.metrics.AUC(name=&quot;pr_auc&quot;, curve=&quot;PR&quot;),<br>                 keras.metrics.Precision(name=&quot;precision&quot;), keras.metrics.Recall(name=&quot;recall&quot;)],<br>    )<br>    return model<br><br><br>def train_keras(df=None, epochs: int = 40, batch: int = 256) -&gt; dict:<br>    import tensorflow as tf<br>    from tensorflow import keras<br>    from sklearn.model_selection import train_test_split<br><br>    if tf.config.list_physical_devices(&quot;GPU&quot;):<br>        keras.mixed_precision.set_global_policy(&quot;mixed_float16&quot;)<br><br>    df = df if df is not None else synthetic_deliveries()<br>    tr, va = train_test_split(df, test_size=0.2, stratify=df[TARGET], random_state=42)<br>    model = build_keras_model(tr)<br><br>    pos = tr[TARGET].sum()<br>    class_weight = {0: 1.0, 1: float((len(tr) - pos) / max(pos, 1))}<br>    callbacks = [<br>        keras.callbacks.EarlyStopping(monitor=&quot;val_pr_auc&quot;, mode=&quot;max&quot;, patience=6,<br>                                      restore_best_weights=True),<br>        keras.callbacks.ReduceLROnPlateau(monitor=&quot;val_loss&quot;, factor=0.5, patience=3),<br>        keras.callbacks.ModelCheckpoint(&quot;./artifacts/best.keras&quot;, monitor=&quot;val_pr_auc&quot;,<br>                                        mode=&quot;max&quot;, save_best_only=True),<br>    ]<br>    model.fit(_tf_dataset(tr, batch, True), validation_data=_tf_dataset(va, batch, False),<br>              epochs=epochs, class_weight=class_weight, callbacks=callbacks, verbose=2)<br><br>    metrics = model.evaluate(_tf_dataset(va, batch, False), return_dict=True)<br>    log.info(&quot;FINAL_METRICS %s&quot;, json.dumps(json_safe(metrics)))<br><br>    export_dir = os.path.join(S.local_model_dir, &quot;1&quot;)           # TF-Serving layout<br>    model.export(export_dir)<br>    Path(S.local_model_dir, &quot;metrics.json&quot;).write_text(json.dumps(json_safe(metrics)))<br><br>    if S.bucket:                                                # push artifact to S3<br>        import tarfile<br>        tar_path = &quot;./artifacts/model.tar.gz&quot;<br>        with tarfile.open(tar_path, &quot;w:gz&quot;) as tar:<br>            tar.add(S.local_model_dir, arcname=&quot;.&quot;)<br>        client(&quot;s3&quot;).upload_file(tar_path, S.bucket, &quot;models/late_delivery/model.tar.gz&quot;)<br>        log.info(&quot;uploaded s3://%s/models/late_delivery/model.tar.gz&quot;, S.bucket)<br>    return json_safe(metrics)<br><br><br>def deploy_sagemaker():<br>    &quot;&quot;&quot;Train on SageMaker (spot) with Bayesian HPO and deploy the best model as an endpoint.&quot;&quot;&quot;<br>    import sagemaker<br>    from sagemaker.tensorflow import TensorFlow<br>    from sagemaker.tuner import HyperparameterTuner, ContinuousParameter<br><br>    if not (S.bucket and S.sm_role_arn):<br>        raise SystemExit(&quot;S3_BUCKET and SM_ROLE_ARN are required for deploy&quot;)<br><br>    df = synthetic_deliveries()<br>    df.to_csv(&quot;./artifacts/deliveries.csv&quot;, index=False)<br>    client(&quot;s3&quot;).upload_file(&quot;./artifacts/deliveries.csv&quot;, S.bucket, &quot;training/deliveries.csv&quot;)<br><br>    # SageMaker runs THIS file as the entry point with --sagemaker-train<br>    est = TensorFlow(entry_point=&quot;run.py&quot;, role=S.sm_role_arn, instance_count=1,<br>                     instance_type=&quot;ml.m5.xlarge&quot;, framework_version=&quot;2.16&quot;, py_version=&quot;py310&quot;,<br>                     hyperparameters={&quot;sagemaker-train&quot;: 1, &quot;epochs&quot;: 40},<br>                     metric_definitions=[{&quot;Name&quot;: &quot;val_pr_auc&quot;, &quot;Regex&quot;: &#39;&quot;val_pr_auc&quot;: ([0-9\\.]+)&#39;}],<br>                     use_spot_instances=True, max_run=3600, max_wait=7200,<br>                     output_path=f&quot;s3://{S.bucket}/models/&quot;)<br>    tuner = HyperparameterTuner(est, &quot;val_pr_auc&quot;, {&quot;lr&quot;: ContinuousParameter(1e-4, 5e-3, &quot;Logarithmic&quot;),<br>                                                    &quot;dropout&quot;: ContinuousParameter(0.1, 0.5)},<br>                                est.metric_definitions, objective_type=&quot;Maximize&quot;,<br>                                max_jobs=6, max_parallel_jobs=2, strategy=&quot;Bayesian&quot;)<br>    tuner.fit({&quot;train&quot;: f&quot;s3://{S.bucket}/training&quot;})<br>    pred = tuner.best_estimator().deploy(initial_instance_count=1, instance_type=&quot;ml.m5.large&quot;,<br>                                         endpoint_name=S.sm_endpoint or &quot;late-delivery-keras&quot;)<br>    log.info(&quot;Endpoint live: %s&quot;, pred.endpoint_name)<br><br><br>@lru_cache(maxsize=1)<br>def _local_model():<br>    import tensorflow as tf<br>    path = os.path.join(S.local_model_dir, &quot;1&quot;)<br>    if not os.path.exists(path):<br>        log.warning(&quot;No local model found — training a quick one (10 epochs)&quot;)<br>        train_keras(epochs=10)<br>    return tf.saved_model.load(path).signatures[&quot;serving_default&quot;]<br><br><br>def predict_late_delivery(supplier_id: str, order_qty: float, lead_time_days: float,<br>                          distance_km: float, historical_late_rate: float, month: int,<br>                          shipping_mode: str = &quot;road&quot;) -&gt; dict:<br>    &quot;&quot;&quot;Tool: Keras model via SageMaker endpoint (preferred) or local SavedModel fallback.&quot;&quot;&quot;<br>    feats = {&quot;supplier_id&quot;: [supplier_id], &quot;shipping_mode&quot;: [shipping_mode],<br>             &quot;order_qty&quot;: [float(order_qty)], &quot;lead_time_days&quot;: [float(lead_time_days)],<br>             &quot;distance_km&quot;: [float(distance_km)], &quot;historical_late_rate&quot;: [float(historical_late_rate)],<br>             &quot;month&quot;: [float(month)]}<br>    if S.sm_endpoint:<br>        resp = client(&quot;sagemaker-runtime&quot;).invoke_endpoint(<br>            EndpointName=S.sm_endpoint, ContentType=&quot;application/json&quot;,<br>            Body=json.dumps({&quot;instances&quot;: [feats]}))<br>        prob = float(json.loads(resp[&quot;Body&quot;].read())[&quot;predictions&quot;][0][0])<br>    else:<br>        import tensorflow as tf<br>        fn = _local_model()<br>        tensors = {k: tf.constant([[v[0]]], dtype=tf.string if k in CATEGORICAL else tf.float32)<br>                   for k, v in feats.items()}<br>        prob = float(list(fn(**tensors).values())[0].numpy()[0][0])<br>    return {&quot;supplier_id&quot;: supplier_id, &quot;late_probability&quot;: round(prob, 4),<br>            &quot;risk_band&quot;: &quot;high&quot; if prob &gt; 0.6 else &quot;medium&quot; if prob &gt; 0.3 else &quot;low&quot;}<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 2. RAG — embeddings, ingestion, hybrid retrieval, reranking<br># ═══════════════════════════════════════════════════════════════════════════<br>@lru_cache(maxsize=2048)<br>def embed(text: str) -&gt; tuple:<br>    body = json.dumps({&quot;inputText&quot;: text[:8000], &quot;dimensions&quot;: S.embed_dim, &quot;normalize&quot;: True})<br>    resp = client(&quot;bedrock-runtime&quot;).invoke_model(modelId=S.embed_model_id, body=body)<br>    return tuple(json.loads(resp[&quot;body&quot;].read())[&quot;embedding&quot;])<br><br><br>def chunk_text(text: str, max_chars=1200, overlap=150) -&gt; list[str]:<br>    paras = [p.strip() for p in re.split(r&quot;\n\s*\n&quot;, text) if p.strip()]<br>    chunks, buf = [], &quot;&quot;<br>    for p in paras:<br>        if len(buf) + len(p) &gt; max_chars and buf:<br>            chunks.append(buf)<br>            buf = buf[-overlap:] + &quot;\n&quot; + p<br>        else:<br>            buf = f&quot;{buf}\n{p}&quot; if buf else p<br>    return chunks + ([buf] if buf else [])<br><br><br>class VectorStore:<br>    &quot;&quot;&quot;OpenSearch Serverless k-NN when AOSS_HOST is set; otherwise a local JSON/numpy store.&quot;&quot;&quot;<br><br>    def __init__(self):<br>        self.remote = bool(S.aoss_host)<br>        if self.remote:<br>            from opensearchpy import OpenSearch, RequestsHttpConnection<br>            from requests_aws4auth import AWS4Auth<br>            c = boto3.Session().get_credentials()<br>            auth = AWS4Auth(c.access_key, c.secret_key, S.region, &quot;aoss&quot;, session_token=c.token)<br>            self.os = OpenSearch(hosts=[{&quot;host&quot;: S.aoss_host, &quot;port&quot;: 443}], http_auth=auth,<br>                                 use_ssl=True, verify_certs=True, connection_class=RequestsHttpConnection)<br>            if not self.os.indices.exists(S.aoss_index):<br>                self.os.indices.create(S.aoss_index, body={<br>                    &quot;settings&quot;: {&quot;index&quot;: {&quot;knn&quot;: True}},<br>                    &quot;mappings&quot;: {&quot;properties&quot;: {<br>                        &quot;embedding&quot;: {&quot;type&quot;: &quot;knn_vector&quot;, &quot;dimension&quot;: S.embed_dim,<br>                                      &quot;method&quot;: {&quot;name&quot;: &quot;hnsw&quot;, &quot;engine&quot;: &quot;faiss&quot;, &quot;space_type&quot;: &quot;cosinesimil&quot;,<br>                                                 &quot;parameters&quot;: {&quot;ef_construction&quot;: 256, &quot;m&quot;: 32}}},<br>                        &quot;text&quot;: {&quot;type&quot;: &quot;text&quot;}, &quot;source&quot;: {&quot;type&quot;: &quot;keyword&quot;},<br>                        &quot;supplier_id&quot;: {&quot;type&quot;: &quot;keyword&quot;}, &quot;doc_type&quot;: {&quot;type&quot;: &quot;keyword&quot;}}}})<br>        else:<br>            self.docs = json.loads(Path(S.local_store).read_text()) if Path(S.local_store).exists() else []<br><br>    def add(self, docs: list[dict]):<br>        if self.remote:<br>            from opensearchpy import helpers<br>            helpers.bulk(self.os, [{&quot;_index&quot;: S.aoss_index, &quot;_id&quot;: d[&quot;id&quot;], &quot;_source&quot;: d} for d in docs])<br>        else:<br>            self.docs.extend(docs)<br>            Path(S.local_store).write_text(json.dumps(self.docs))<br><br>    def _filters(self, supplier_id, doc_type):<br>        f = []<br>        if supplier_id:<br>            f.append({&quot;terms&quot;: {&quot;supplier_id&quot;: [supplier_id, &quot;ALL&quot;]}})<br>        if doc_type:<br>            f.append({&quot;term&quot;: {&quot;doc_type&quot;: doc_type}})<br>        return f<br><br>    def knn(self, vec, k, supplier_id=None, doc_type=None):<br>        if self.remote:<br>            q = {&quot;knn&quot;: {&quot;embedding&quot;: {&quot;vector&quot;: list(vec), &quot;k&quot;: k}}}<br>            body = {&quot;size&quot;: k, &quot;query&quot;: {&quot;bool&quot;: {&quot;must&quot;: [q], &quot;filter&quot;: self._filters(supplier_id, doc_type)}}}<br>            return [h[&quot;_source&quot;] for h in self.os.search(index=S.aoss_index, body=body)[&quot;hits&quot;][&quot;hits&quot;]]<br>        cands = [d for d in self.docs if (not supplier_id or d[&quot;supplier_id&quot;] in (supplier_id, &quot;ALL&quot;))<br>                 and (not doc_type or d[&quot;doc_type&quot;] == doc_type)]<br>        if not cands:<br>            return []<br>        M = np.array([d[&quot;embedding&quot;] for d in cands])<br>        sims = M @ np.array(vec)<br>        return [cands[i] for i in np.argsort(-sims)[:k]]<br><br>    def bm25(self, query, k, supplier_id=None, doc_type=None):<br>        if self.remote:<br>            body = {&quot;size&quot;: k, &quot;query&quot;: {&quot;bool&quot;: {&quot;must&quot;: [{&quot;match&quot;: {&quot;text&quot;: query}}],<br>                                                  &quot;filter&quot;: self._filters(supplier_id, doc_type)}}}<br>            return [h[&quot;_source&quot;] for h in self.os.search(index=S.aoss_index, body=body)[&quot;hits&quot;][&quot;hits&quot;]]<br>        terms = set(re.findall(r&quot;\w+&quot;, query.lower()))<br>        scored = [(sum(t in d[&quot;text&quot;].lower() for t in terms), d) for d in self.docs<br>                  if (not supplier_id or d[&quot;supplier_id&quot;] in (supplier_id, &quot;ALL&quot;))<br>                  and (not doc_type or d[&quot;doc_type&quot;] == doc_type)]<br>        return [d for s, d in sorted(scored, key=lambda x: -x[0])[:k] if s &gt; 0]<br><br><br>@lru_cache(maxsize=1)<br>def store() -&gt; VectorStore:<br>    return VectorStore()<br><br><br>SAMPLE_DOCS = {  # used when S3_BUCKET is not set — lets `ingest` + `agentic-rag` run locally<br>    &quot;knowledge/contract/SUP-007/msa.txt&quot;: &quot;MASTER SUPPLY AGREEMENT — SUP-007 (Northwind Components)\n\n&quot;<br>        &quot;Clause 9.2 On-Time Delivery: Supplier shall achieve ≥ 95% on-time delivery per calendar quarter.\n\n&quot;<br>        &quot;Clause 9.4 Service Credits: For each percentage point below 95%, Supplier shall credit 1.5% of &quot;<br>        &quot;the quarter&#39;s invoiced value, capped at 10%.\n\nClause 14 Escalation: Buyer must notify Supplier &quot;<br>        &quot;in writing within 15 business days of quarter end to claim credits.&quot;,<br>    &quot;knowledge/correspondence/SUP-007/2024-q2.txt&quot;: &quot;Email thread with SUP-007, June 2024.\n\nSUP-007 &quot;<br>        &quot;acknowledged port congestion delays and committed to a recovery plan including air-freight &quot;<br>        &quot;for priority SKUs at their cost through end of Q3.&quot;,<br>    &quot;knowledge/sla/ALL/procurement-policy.txt&quot;: &quot;PROCUREMENT POLICY\n\nAll supplier performance emails &quot;<br>        &quot;must be factual, reference the governing contract clause, state the measured metric, and request &quot;<br>        &quot;a corrective action plan within 10 business days. Tone: professional, non-adversarial.&quot;,<br>}<br><br><br>def ingest():<br>    &quot;&quot;&quot;S3 (or sample docs) → chunk → embed → vector store.&quot;&quot;&quot;<br>    from concurrent.futures import ThreadPoolExecutor<br>    docs = {}<br>    if S.bucket:<br>        s3 = client(&quot;s3&quot;)<br>        for page in s3.get_paginator(&quot;list_objects_v2&quot;).paginate(Bucket=S.bucket, Prefix=S.docs_prefix):<br>            for o in page.get(&quot;Contents&quot;, []):<br>                if o[&quot;Key&quot;].endswith(&quot;.txt&quot;):<br>                    docs[o[&quot;Key&quot;]] = s3.get_object(Bucket=S.bucket, Key=o[&quot;Key&quot;])[&quot;Body&quot;].read().decode()<br>    else:<br>        docs = SAMPLE_DOCS<br><br>    def process(item):<br>        key, text = item<br>        parts = key.split(&quot;/&quot;)<br>        meta = {&quot;doc_type&quot;: parts[1] if len(parts) &gt; 2 else &quot;general&quot;,<br>                &quot;supplier_id&quot;: parts[2] if len(parts) &gt; 3 else &quot;ALL&quot;, &quot;source&quot;: key}<br>        return [{&quot;id&quot;: hashlib.sha1(f&quot;{key}#{i}&quot;.encode()).hexdigest(), &quot;text&quot;: c,<br>                 &quot;embedding&quot;: list(embed(c)), **meta} for i, c in enumerate(chunk_text(text))]<br><br>    with ThreadPoolExecutor(max_workers=8) as ex:<br>        for batch in ex.map(process, docs.items()):<br>            store().add(batch)<br>            log.info(&quot;indexed %d chunks from %s&quot;, len(batch), batch[0][&quot;source&quot;])<br><br><br>def _rrf(lists, k=60):<br>    scores, docs = defaultdict(float), {}<br>    for lst in lists:<br>        for rank, d in enumerate(lst):<br>            scores[d[&quot;id&quot;]] += 1 / (k + rank + 1)<br>            docs[d[&quot;id&quot;]] = d<br>    return [docs[i] for i, _ in sorted(scores.items(), key=lambda x: -x[1])]<br><br><br>def llm(prompt: str, system: str | None = None, max_tokens=600, temperature=0.0, trace=None) -&gt; str:<br>    kw = {&quot;modelId&quot;: S.llm_model_id, &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: [{&quot;text&quot;: prompt}]}],<br>          &quot;inferenceConfig&quot;: {&quot;maxTokens&quot;: max_tokens, &quot;temperature&quot;: temperature}}<br>    if system:<br>        kw[&quot;system&quot;] = [{&quot;text&quot;: system}]<br>    resp = client(&quot;bedrock-runtime&quot;).converse(**kw)<br>    if trace is not None:<br>        trace.add_usage(resp.get(&quot;usage&quot;, {}))<br>    return resp[&quot;output&quot;][&quot;message&quot;][&quot;content&quot;][0][&quot;text&quot;]<br><br><br>def _rerank(query, docs, top_n):<br>    if len(docs) &lt;= top_n:<br>        return docs<br>    numbered = &quot;\n\n&quot;.join(f&quot;[{i}] {d[&#39;text&#39;][:800]}&quot; for i, d in enumerate(docs))<br>    raw = llm(f&quot;Query: {query}\n\nRate each passage&#39;s relevance 0-10. Return ONLY JSON &quot;<br>              f&quot;{{index: score}}.\n\n{numbered}&quot;, max_tokens=300)<br>    try:<br>        scores = json.loads(raw[raw.find(&quot;{&quot;): raw.rfind(&quot;}&quot;) + 1])<br>        order = sorted(range(len(docs)), key=lambda i: -float(scores.get(str(i), 0)))<br>        return [docs[i] for i in order[:top_n]]<br>    except Exception:<br>        return docs[:top_n]<br><br><br>def retrieve(query: str, supplier_id: str | None = None, doc_type: str | None = None,<br>             k=12, top_n=5) -&gt; list[dict]:<br>    st = store()<br>    fused = _rrf([st.knn(embed(query), k, supplier_id, doc_type), st.bm25(query, k, supplier_id, doc_type)])<br>    return [{&quot;text&quot;: d[&quot;text&quot;], &quot;source&quot;: d[&quot;source&quot;], &quot;doc_type&quot;: d[&quot;doc_type&quot;],<br>             &quot;supplier_id&quot;: d[&quot;supplier_id&quot;]} for d in _rerank(query, fused[:k], top_n)]<br><br><br>def plain_rag(question: str) -&gt; dict:<br>    &quot;&quot;&quot;Classic single-shot RAG.&quot;&quot;&quot;<br>    ctx = retrieve(question)<br>    block = &quot;\n\n&quot;.join(f&quot;&lt;doc source=&#39;{d[&#39;source&#39;]}&#39;&gt;{d[&#39;text&#39;]}&lt;/doc&gt;&quot; for d in ctx)<br>    answer = llm(f&quot;{block}\n\nQuestion: {question}&quot;,<br>                 system=&quot;Answer ONLY from the documents. Cite sources. If not present, say so.&quot;,<br>                 max_tokens=S.max_tokens_per_call)<br>    return {&quot;mode&quot;: &quot;rag&quot;, &quot;answer&quot;: answer, &quot;sources&quot;: sorted({d[&quot;source&quot;] for d in ctx})}<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 3. TOOLS (shared by Agentic AI and Agentic RAG)<br># ═══════════════════════════════════════════════════════════════════════════<br>def query_erp(quarter: str = &quot;last&quot;) -&gt; dict:<br>    &quot;&quot;&quot;Aggregate late deliveries per supplier (DynamoDB, or synthetic data locally).&quot;&quot;&quot;<br>    today = date.today()<br>    qs = date(today.year, ((today.month - 1) // 3) * 3 + 1, 1)<br>    if quarter == &quot;last&quot;:<br>        end = qs - timedelta(days=1)<br>        start = date(end.year, ((end.month - 1) // 3) * 3 + 1, 1)<br>    else:<br>        start, end = qs, today<br><br>    if S.ddb_deliveries:<br>        from boto3.dynamodb.conditions import Key<br>        items = resource(&quot;dynamodb&quot;).Table(S.ddb_deliveries).query(<br>            IndexName=&quot;period-index&quot;,<br>            KeyConditionExpression=Key(&quot;period&quot;).eq(&quot;ALL&quot;) &amp;<br>            Key(&quot;delivery_date&quot;).between(start.isoformat(), end.isoformat()))[&quot;Items&quot;]<br>    else:<br>        df = synthetic_deliveries(3000, seed=7)<br>        items = df[[&quot;supplier_id&quot;, TARGET]].rename(columns={TARGET: &quot;is_late&quot;}).to_dict(&quot;records&quot;)<br><br>    agg = defaultdict(lambda: {&quot;total&quot;: 0, &quot;late&quot;: 0})<br>    for it in items:<br>        agg[it[&quot;supplier_id&quot;]][&quot;total&quot;] += 1<br>        agg[it[&quot;supplier_id&quot;]][&quot;late&quot;] += int(it.get(&quot;is_late&quot;, 0))<br>    ranked = sorted(({&quot;supplier_id&quot;: k, **v, &quot;late_rate&quot;: round(v[&quot;late&quot;] / v[&quot;total&quot;], 3)}<br>                     for k, v in agg.items()), key=lambda x: -x[&quot;late&quot;])<br>    return {&quot;period&quot;: f&quot;{start} to {end}&quot;, &quot;suppliers&quot;: ranked[:10]}<br><br><br>def retrieve_documents(query: str, supplier_id: str | None = None, doc_type: str | None = None) -&gt; dict:<br>    docs = retrieve(query, supplier_id, doc_type)<br>    return {&quot;query&quot;: query, &quot;count&quot;: len(docs), &quot;documents&quot;: docs}<br><br><br>def draft_email(to: str, subject: str, body: str, supplier_id: str) -&gt; dict:<br>    &quot;&quot;&quot;Creates a draft only. Sending happens after human approval (see send_approved_email).&quot;&quot;&quot;<br>    draft = {&quot;to&quot;: to, &quot;subject&quot;: subject, &quot;body&quot;: body, &quot;supplier_id&quot;: supplier_id,<br>             &quot;status&quot;: &quot;DRAFT_PENDING_APPROVAL&quot;, &quot;created&quot;: datetime.utcnow().isoformat()}<br>    if S.ddb_memory:<br>        resource(&quot;dynamodb&quot;).Table(S.ddb_memory).put_item(Item={&quot;pk&quot;: f&quot;DRAFT#{supplier_id}&quot;,<br>                                                                &quot;sk&quot;: draft[&quot;created&quot;], **draft})<br>    else:<br>        Path(&quot;./artifacts/drafts.jsonl&quot;).open(&quot;a&quot;).write(json.dumps(draft) + &quot;\n&quot;)<br>    return draft<br><br><br>def send_approved_email(draft: dict) -&gt; dict:<br>    resp = client(&quot;ses&quot;).send_email(Source=S.ses_sender, Destination={&quot;ToAddresses&quot;: [draft[&quot;to&quot;]]},<br>                                    Message={&quot;Subject&quot;: {&quot;Data&quot;: draft[&quot;subject&quot;]},<br>                                             &quot;Body&quot;: {&quot;Text&quot;: {&quot;Data&quot;: draft[&quot;body&quot;]}}})<br>    return {&quot;message_id&quot;: resp[&quot;MessageId&quot;], &quot;status&quot;: &quot;SENT&quot;}<br><br><br>REGISTRY = {&quot;query_erp&quot;: query_erp, &quot;predict_late_delivery&quot;: predict_late_delivery,<br>            &quot;retrieve_documents&quot;: retrieve_documents, &quot;draft_email&quot;: draft_email}<br><br>TOOL_SPECS = {<br>    &quot;query_erp&quot;: {&quot;toolSpec&quot;: {&quot;name&quot;: &quot;query_erp&quot;,<br>        &quot;description&quot;: &quot;Aggregate delivery performance per supplier from the ERP for a quarter.&quot;,<br>        &quot;inputSchema&quot;: {&quot;json&quot;: {&quot;type&quot;: &quot;object&quot;, &quot;properties&quot;: {<br>            &quot;quarter&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;enum&quot;: [&quot;last&quot;, &quot;current&quot;]}}}}}},<br>    &quot;predict_late_delivery&quot;: {&quot;toolSpec&quot;: {&quot;name&quot;: &quot;predict_late_delivery&quot;,<br>        &quot;description&quot;: &quot;Predict late-delivery probability for a future shipment using the Keras model.&quot;,<br>        &quot;inputSchema&quot;: {&quot;json&quot;: {&quot;type&quot;: &quot;object&quot;,<br>            &quot;required&quot;: [&quot;supplier_id&quot;, &quot;order_qty&quot;, &quot;lead_time_days&quot;, &quot;distance_km&quot;, &quot;historical_late_rate&quot;, &quot;month&quot;],<br>            &quot;properties&quot;: {&quot;supplier_id&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;order_qty&quot;: {&quot;type&quot;: &quot;number&quot;},<br>                           &quot;lead_time_days&quot;: {&quot;type&quot;: &quot;number&quot;}, &quot;distance_km&quot;: {&quot;type&quot;: &quot;number&quot;},<br>                           &quot;historical_late_rate&quot;: {&quot;type&quot;: &quot;number&quot;}, &quot;month&quot;: {&quot;type&quot;: &quot;integer&quot;},<br>                           &quot;shipping_mode&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;enum&quot;: [&quot;road&quot;, &quot;air&quot;, &quot;sea&quot;]}}}}}},<br>    &quot;retrieve_documents&quot;: {&quot;toolSpec&quot;: {&quot;name&quot;: &quot;retrieve_documents&quot;,<br>        &quot;description&quot;: &quot;Search contracts, SLAs, policies, reports and correspondence. &quot;<br>                       &quot;Narrow with supplier_id and doc_type (contract|sla|correspondence|report).&quot;,<br>        &quot;inputSchema&quot;: {&quot;json&quot;: {&quot;type&quot;: &quot;object&quot;, &quot;required&quot;: [&quot;query&quot;], &quot;properties&quot;: {<br>            &quot;query&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;supplier_id&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;doc_type&quot;: {&quot;type&quot;: &quot;string&quot;}}}}}},<br>    &quot;draft_email&quot;: {&quot;toolSpec&quot;: {&quot;name&quot;: &quot;draft_email&quot;,<br>        &quot;description&quot;: &quot;Create an email draft for human approval. Does NOT send.&quot;,<br>        &quot;inputSchema&quot;: {&quot;json&quot;: {&quot;type&quot;: &quot;object&quot;, &quot;required&quot;: [&quot;to&quot;, &quot;subject&quot;, &quot;body&quot;, &quot;supplier_id&quot;],<br>            &quot;properties&quot;: {&quot;to&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;subject&quot;: {&quot;type&quot;: &quot;string&quot;},<br>                           &quot;body&quot;: {&quot;type&quot;: &quot;string&quot;}, &quot;supplier_id&quot;: {&quot;type&quot;: &quot;string&quot;}}}}}},<br>}<br><br><br>def execute_tool(name: str, args: dict) -&gt; dict:<br>    if name not in S.allowed_tools or name not in REGISTRY:<br>        return {&quot;error&quot;: f&quot;tool &#39;{name}&#39; not permitted&quot;}<br>    try:<br>        log.info(&quot;TOOL %s %s&quot;, name, json.dumps(args)[:300])<br>        return json_safe(REGISTRY[name](**args))<br>    except Exception as exc:<br>        log.exception(&quot;tool failure&quot;)<br>        return {&quot;error&quot;: f&quot;{type(exc).__name__}: {exc}&quot;}<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 4. AGENT CORE — ReAct loop with guardrails<br># ═══════════════════════════════════════════════════════════════════════════<br>@dataclass<br>class Trace:<br>    steps: list = field(default_factory=list)<br>    input_tokens: int = 0<br>    output_tokens: int = 0<br>    started: float = field(default_factory=time.time)<br><br>    def add_usage(self, u: dict):<br>        self.input_tokens += u.get(&quot;inputTokens&quot;, 0)<br>        self.output_tokens += u.get(&quot;outputTokens&quot;, 0)<br><br>    @property<br>    def total_tokens(self):<br>        return self.input_tokens + self.output_tokens<br><br>    def summary(self):<br>        return {&quot;tool_calls&quot;: len(self.steps), &quot;tokens&quot;: self.total_tokens,<br>                &quot;seconds&quot;: round(time.time() - self.started, 2)}<br><br><br>class BedrockAgent:<br>    def __init__(self, system_prompt: str, tool_names: list[str], max_steps=S.max_steps):<br>        self.system = system_prompt<br>        self.tool_config = {&quot;tools&quot;: [TOOL_SPECS[n] for n in tool_names]}<br>        self.max_steps = max_steps<br><br>    def run(self, task: str, trace: Trace | None = None, history: list | None = None) -&gt; dict:<br>        trace = trace or Trace()<br>        messages = (history or []) + [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: [{&quot;text&quot;: task}]}]<br>        for step in range(self.max_steps):<br>            if trace.total_tokens &gt; S.token_budget:<br>                return {&quot;answer&quot;: None, &quot;error&quot;: &quot;token budget exceeded&quot;, &quot;trace&quot;: trace, &quot;messages&quot;: messages}<br><br>            resp = client(&quot;bedrock-runtime&quot;).converse(<br>                modelId=S.llm_model_id, system=[{&quot;text&quot;: self.system}], messages=messages,<br>                toolConfig=self.tool_config,<br>                inferenceConfig={&quot;maxTokens&quot;: S.max_tokens_per_call, &quot;temperature&quot;: S.temperature})<br>            trace.add_usage(resp.get(&quot;usage&quot;, {}))<br>            msg = resp[&quot;output&quot;][&quot;message&quot;]<br>            messages.append(msg)<br><br>            if resp[&quot;stopReason&quot;] != &quot;tool_use&quot;:<br>                return {&quot;answer&quot;: &quot; &quot;.join(c[&quot;text&quot;] for c in msg[&quot;content&quot;] if &quot;text&quot; in c),<br>                        &quot;trace&quot;: trace, &quot;messages&quot;: messages}<br><br>            results = []<br>            for block in msg[&quot;content&quot;]:<br>                if &quot;toolUse&quot; in block:<br>                    tu = block[&quot;toolUse&quot;]<br>                    out = execute_tool(tu[&quot;name&quot;], tu[&quot;input&quot;])<br>                    trace.steps.append({&quot;step&quot;: step, &quot;tool&quot;: tu[&quot;name&quot;], &quot;args&quot;: tu[&quot;input&quot;],<br>                                        &quot;ok&quot;: &quot;error&quot; not in out})<br>                    results.append({&quot;toolResult&quot;: {&quot;toolUseId&quot;: tu[&quot;toolUseId&quot;],<br>                                                   &quot;content&quot;: [{&quot;json&quot;: out}],<br>                                                   &quot;status&quot;: &quot;error&quot; if &quot;error&quot; in out else &quot;success&quot;}})<br>            messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: results})<br>        return {&quot;answer&quot;: None, &quot;error&quot;: &quot;max steps reached&quot;, &quot;trace&quot;: trace, &quot;messages&quot;: messages}<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 5. AGENTIC AI  vs  AGENTIC RAG<br># ═══════════════════════════════════════════════════════════════════════════<br>AGENTIC_AI_SYSTEM = &quot;&quot;&quot;You are a procurement operations agent. Think step by step and use tools<br>for real data — never invent numbers. Workflow: query_erp → (optional) predict_late_delivery<br>→ draft_email. Do NOT quote contract terms you have not obtained from a tool.<br>Finish with a concise summary of findings and actions.&quot;&quot;&quot;<br><br>AGENTIC_RAG_SYSTEM = &quot;&quot;&quot;You are a procurement operations agent with a knowledge base.<br>Rules:<br> • Ground every factual claim (SLA thresholds, penalties, past commitments, policy) in a<br>   retrieve_documents result. If retrieval is weak, REWRITE the query (add supplier_id /<br>   doc_type) and retry — up to 3 times.<br> • Numbers come from query_erp; forward risk from predict_late_delivery; terms from documents.<br> • Before draft_email, retrieve the supplier&#39;s contract/SLA, recent correspondence, and the<br>   procurement email policy, and comply with all of them.<br> • Cite sources inline like [source: knowledge/contract/SUP-007/msa.txt].<br>Finish with sections: FINDINGS, EVIDENCE (sources), ACTIONS TAKEN.&quot;&quot;&quot;<br><br>VERIFIER = &quot;&quot;&quot;You are a strict fact-checker. List every factual claim in the REPORT that is NOT<br>supported by the TOOL OUTPUTS. Respond ONLY with JSON:<br>{&quot;unsupported_claims&quot;: [...], &quot;verdict&quot;: &quot;PASS&quot; | &quot;FAIL&quot;}&quot;&quot;&quot;<br><br><br>def run_agentic_ai(task: str) -&gt; dict:<br>    r = BedrockAgent(AGENTIC_AI_SYSTEM, [&quot;query_erp&quot;, &quot;predict_late_delivery&quot;, &quot;draft_email&quot;]).run(task)<br>    return {&quot;mode&quot;: &quot;agentic_ai&quot;, &quot;answer&quot;: r.get(&quot;answer&quot;), &quot;error&quot;: r.get(&quot;error&quot;),<br>            &quot;trace&quot;: r[&quot;trace&quot;].summary(), &quot;steps&quot;: r[&quot;trace&quot;].steps}<br><br><br>class Memory:<br>    &quot;&quot;&quot;Episodic memory: DynamoDB in prod, JSONL locally.&quot;&quot;&quot;<br><br>    def load(self, key: str, limit=3) -&gt; str:<br>        if S.ddb_memory:<br>            items = resource(&quot;dynamodb&quot;).Table(S.ddb_memory).query(<br>                KeyConditionExpression=&quot;pk = :p&quot;, ExpressionAttributeValues={&quot;:p&quot;: f&quot;TASK#{key}&quot;},<br>                ScanIndexForward=False, Limit=limit)[&quot;Items&quot;]<br>        else:<br>            p = Path(&quot;./artifacts/memory.jsonl&quot;)<br>            items = [json.loads(l) for l in p.read_text().splitlines()][-limit:] if p.exists() else []<br>        if not items:<br>            return &quot;&quot;<br>        return &quot;Past outcomes for similar tasks:\n&quot; + &quot;\n&quot;.join(<br>            f&quot;- {i[&#39;sk&#39;]} [{i.get(&#39;verdict&#39;)}]: {i.get(&#39;outcome&#39;, &#39;&#39;)[:300]}&quot; for i in items)<br><br>    def save(self, key: str, outcome: str, verdict: str):<br>        item = {&quot;pk&quot;: f&quot;TASK#{key}&quot;, &quot;sk&quot;: datetime.utcnow().isoformat(),<br>                &quot;outcome&quot;: outcome[:2000], &quot;verdict&quot;: verdict}<br>        if S.ddb_memory:<br>            resource(&quot;dynamodb&quot;).Table(S.ddb_memory).put_item(Item=item)<br>        else:<br>            Path(&quot;./artifacts/memory.jsonl&quot;).open(&quot;a&quot;).write(json.dumps(item) + &quot;\n&quot;)<br><br><br>def _verify(answer: str, messages: list, trace: Trace) -&gt; dict:<br>    outputs = [json.dumps(c[&quot;toolResult&quot;][&quot;content&quot;][0].get(&quot;json&quot;, {}))[:3000]<br>               for m in messages for c in m.get(&quot;content&quot;, []) if &quot;toolResult&quot; in c]<br>    raw = llm(f&quot;{VERIFIER}\n\n=== REPORT ===\n{answer}\n\n=== TOOL OUTPUTS ===\n&quot; + &quot;\n---\n&quot;.join(outputs),<br>              max_tokens=600, trace=trace)<br>    try:<br>        return json.loads(raw[raw.find(&quot;{&quot;): raw.rfind(&quot;}&quot;) + 1])<br>    except Exception:<br>        return {&quot;unsupported_claims&quot;: [], &quot;verdict&quot;: &quot;PASS&quot;}<br><br><br>def run_agentic_rag(task: str, task_key: str = &quot;supplier_review&quot;) -&gt; dict:<br>    trace, mem = Trace(), Memory()<br>    prior = mem.load(task_key)<br>    agent = BedrockAgent(AGENTIC_RAG_SYSTEM, list(S.allowed_tools))<br><br>    r = agent.run(f&quot;{prior}\n\nTASK: {task}&quot; if prior else task, trace=trace)<br>    if r.get(&quot;error&quot;):<br>        return {&quot;mode&quot;: &quot;agentic_rag&quot;, &quot;answer&quot;: None, &quot;error&quot;: r[&quot;error&quot;], &quot;trace&quot;: trace.summary()}<br><br>    verdict = _verify(r[&quot;answer&quot;], r[&quot;messages&quot;], trace)          # reflection pass<br>    if verdict.get(&quot;verdict&quot;) == &quot;FAIL&quot; and verdict.get(&quot;unsupported_claims&quot;):<br>        fix = (&quot;Your report contained unsupported claims:\n&quot;<br>               + &quot;\n&quot;.join(f&quot;- {c}&quot; for c in verdict[&quot;unsupported_claims&quot;])<br>               + &quot;\nRetrieve evidence for them or remove them, then re-issue the full report.&quot;)<br>        r2 = agent.run(fix, trace=trace, history=r[&quot;messages&quot;])<br>        if not r2.get(&quot;error&quot;):<br>            r, verdict = r2, _verify(r2[&quot;answer&quot;], r2[&quot;messages&quot;], trace)<br><br>    mem.save(task_key, r.get(&quot;answer&quot;) or &quot;&quot;, verdict.get(&quot;verdict&quot;, &quot;UNKNOWN&quot;))<br>    return {&quot;mode&quot;: &quot;agentic_rag&quot;, &quot;answer&quot;: r.get(&quot;answer&quot;), &quot;verification&quot;: verdict,<br>            &quot;trace&quot;: trace.summary(), &quot;steps&quot;: trace.steps}<br><br><br># ═══════════════════════════════════════════════════════════════════════════<br># 6. ENTRY POINTS — CLI, SageMaker training, AWS Lambda<br># ═══════════════════════════════════════════════════════════════════════════<br>def lambda_handler(event, context=None):<br>    body = json.loads(event[&quot;body&quot;]) if isinstance(event.get(&quot;body&quot;), str) else event<br>    mode, task = body.get(&quot;mode&quot;, &quot;agentic_rag&quot;), body.get(&quot;task&quot;, &quot;&quot;)<br>    dispatch = {<br>        &quot;traditional&quot;: lambda: predict_late_delivery(**body[&quot;features&quot;]),<br>        &quot;rag&quot;: lambda: plain_rag(task),<br>        &quot;agentic_ai&quot;: lambda: run_agentic_ai(task),<br>        &quot;agentic_rag&quot;: lambda: run_agentic_rag(task),<br>        &quot;send_approved&quot;: lambda: send_approved_email(body[&quot;draft&quot;]),   # called by Step Functions<br>    }<br>    if mode not in dispatch:<br>        return {&quot;statusCode&quot;: 400, &quot;body&quot;: json.dumps({&quot;error&quot;: f&quot;unknown mode {mode}&quot;})}<br>    return {&quot;statusCode&quot;: 200, &quot;body&quot;: json.dumps(dispatch[mode](), default=str)}<br><br><br>def main():<br>    p = argparse.ArgumentParser(description=&quot;Traditional AI · RAG · Agentic AI · Agentic RAG&quot;)<br>    sub = p.add_subparsers(dest=&quot;cmd&quot;, required=True)<br>    t = sub.add_parser(&quot;train&quot;); t.add_argument(&quot;--epochs&quot;, type=int, default=40)<br>    sub.add_parser(&quot;deploy&quot;); sub.add_parser(&quot;ingest&quot;)<br>    pr = sub.add_parser(&quot;predict&quot;)<br>    for a, ty in [(&quot;--supplier&quot;, str), (&quot;--qty&quot;, float), (&quot;--lead&quot;, float), (&quot;--dist&quot;, float),<br>                  (&quot;--rate&quot;, float), (&quot;--month&quot;, int)]:<br>        pr.add_argument(a, type=ty, required=True)<br>    pr.add_argument(&quot;--mode&quot;, default=&quot;road&quot;)<br>    for name in (&quot;rag&quot;, &quot;agent&quot;, &quot;agentic-rag&quot;):<br>        sub.add_parser(name).add_argument(&quot;--task&quot;, required=True)<br>    # SageMaker passes hyperparameters as flags to this same file<br>    p.add_argument(&quot;--sagemaker-train&quot;, type=int, default=0)<br>    p.add_argument(&quot;--epochs&quot;, type=int, default=40)<br>    p.add_argument(&quot;--lr&quot;, type=float, default=1e-3)<br>    p.add_argument(&quot;--dropout&quot;, type=float, default=0.3)<br>    args, _ = p.parse_known_args()<br><br>    if args.cmd == &quot;train&quot;:<br>        print(json.dumps(train_keras(epochs=args.epochs), indent=2))<br>    elif args.cmd == &quot;deploy&quot;:<br>        deploy_sagemaker()<br>    elif args.cmd == &quot;ingest&quot;:<br>        ingest()<br>    elif args.cmd == &quot;predict&quot;:<br>        print(json.dumps(predict_late_delivery(args.supplier, args.qty, args.lead, args.dist,<br>                                               args.rate, args.month, args.mode), indent=2))<br>    elif args.cmd == &quot;rag&quot;:<br>        print(json.dumps(plain_rag(args.task), indent=2))<br>    elif args.cmd == &quot;agent&quot;:<br>        print(json.dumps(run_agentic_ai(args.task), indent=2, default=str))<br>    elif args.cmd == &quot;agentic-rag&quot;:<br>        print(json.dumps(run_agentic_rag(args.task), indent=2, default=str))<br><br><br>if __name__ == &quot;__main__&quot;:<br>    if os.getenv(&quot;SM_TRAINING_ENV&quot;) or &quot;--sagemaker-train&quot; in &quot; &quot;.join(os.sys.argv):<br>        # Inside a SageMaker training container: read channel, train, export to SM_MODEL_DIR<br>        import pandas as pd<br>        _a, _ = argparse.ArgumentParser().parse_known_args()<br>        df = pd.read_csv(os.path.join(os.getenv(&quot;SM_CHANNEL_TRAIN&quot;, &quot;.&quot;), &quot;deliveries.csv&quot;))<br>        S = Settings(local_model_dir=os.getenv(&quot;SM_MODEL_DIR&quot;, &quot;./model&quot;))  # noqa: F811<br>        train_keras(df, epochs=int(os.getenv(&quot;SM_HP_EPOCHS&quot;, 40)))<br>    else:<br>        main()</pre><h3>8. Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*sIbz0-1BvANJ1g7s2xiVJA.png" /></figure><p>The progression is not “old → new, replace everything” but rather <strong>choose the simplest approach that solves the problem</strong> — and move up the ladder only when the task demands adaptability, grounding, or both.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4763f5aa610" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/redefining-the-future-of-ai-a-deep-dive-into-traditional-ai-agentic-systems-rag-and-agentic-rag-d4763f5aa610">Redefining the Future of AI: A Deep Dive into Traditional AI, Agentic Systems, RAG, and Agentic RAG.</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Architecture of Privacy: Unlocking Practical FHE for Finance and Blockchains]]></title>
            <link>https://blog.blockmagnates.com/the-architecture-of-privacy-unlocking-practical-fhe-for-finance-and-blockchains-b942272ef01e?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/b942272ef01e</guid>
            <category><![CDATA[fhe]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[polygon-network]]></category>
            <category><![CDATA[solidity]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 07:34:40 GMT</pubDate>
            <atom:updated>2026-11-09T07:34:40.352Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vTtPfC5mFwPbNgUCeGGI5g.png" /></figure><h4>Introduction:</h4><p>Fully homomorphic encryption (FHE) promises a simple, almost magical idea: compute <em>any</em> function directly on encrypted data without ever decrypting it. In theory, this would allow privacy‑preserving machine learning, auditable finance computations, and encrypted smart contracts — all while keeping raw values sealed. Yet, in practice, there is no single “best” FHE scheme that wins in every setting; instead, the landscape is a trade‑off between precision, throughput, latency, and circuit structure. Recent work and industry deployments show that schemes built for arithmetic circuits — such as BGV and BFV — have surprisingly similar performance on basic operations, while emerging variants like decomposable BFV (DBFV) are making exact FHE viable for complex, high‑volume use cases such as blockchain‑based financial logic.</p><h4>The Dream and the Reality of FHE</h4><p>Fully Homomorphic Encryption (FHE) promises a simple, almost magical idea: <strong>compute any function directly on encrypted data without ever decrypting it</strong>. In theory, this unlocks privacy-preserving machine learning, auditable finance computations, and encrypted smart contracts — all while keeping raw values completely sealed.</p><p>Yet, in practice, there is no single “best” FHE scheme that wins in every setting. Instead, the landscape is defined by a delicate balance between <strong>precision, throughput, latency, and circuit structure</strong>. Recent work and industry deployments demonstrate that schemes built for arithmetic circuits — such as <strong>BGV</strong> and <strong>BFV</strong> — achieve surprisingly competitive performance on basic operations. Meanwhile, emerging variants like <strong>decomposable BFV (DBFV)</strong> are making exact FHE viable for complex, high-volume use cases like blockchain-based financial logic.</p><h4>The Core Promise of FHE</h4><p>At its core, FHE is the cryptographic ideal of <strong>privacy-preserving computation</strong>:</p><ol><li>You encrypt your data locally.</li><li>You ship the ciphertext to an untrusted server.</li><li>The server evaluates a function (e.g., a neural-net layer, a risk-score model, or a voting tally) homomorphically.</li><li>Upon receiving the result back, decryption reveals the exact same answer as cleartext computation — while the server never sees the underlying data.</li></ol><p>However, this capability is not free. Current FHE schemes differ fundamentally in how they encode data (integers vs. approximations), how they handle noise growth during multiplications, and whether they optimize for latency or throughput. Consequently, no universal scheme rules all scenarios: some excel at low-latency comparisons, others at high-throughput aggregation, and others at exact integer arithmetic required for finance and governance.</p><h4>Arithmetic-Friendly Schemes: BGV, BFV, and Their Lik</h4><p>Among the most widely used schemes are <strong>BGV</strong> and <strong>BFV</strong>, both engineered for arithmetic circuits over integers. Rooted in the <strong>Ring-Learning-With-Errors (RLWE)</strong> framework, they treat encrypted data as polynomials, enabling efficient homomorphic addition and multiplication.</p><p>A striking insight from recent research is that, for basic comparison tasks like <em>“less-than,” “maximum,”</em> and <em>“minima,”</em> BGV and BFV exhibit performance comparable to <strong>TFHE</strong>, a scheme traditionally optimized for non-arithmetic (bit-level) circuits. This reveals that arithmetic-oriented FHE is not inherently hindered by branching logic; with advanced algorithmic engineering — such as packed encodings and optimized comparison routines — arithmetic circuits can hold their own.</p><h4>Why Arithmetic Circuits Matter for Finance and Blockchains</h4><p>Arithmetic circuits are the natural language of structured numeric data: loan-risk models, identity-agnostic KYC checks, on-chain auctions, and privacy-preserving financial aggregation. In these domains, precision is non-negotiable. You cannot tolerate rounding or approximation errors when <strong>1 unit of currency or 1 vote must be counted precisely</strong>.</p><p>Exact FHE schemes like BFV and BGV encrypt integers without floating-point approximations, guaranteeing that decryption recovers the exact plaintext. However, the price of this precision is steep: plaintext magnitude and noise growth scale non-trivially with circuit depth and data size, meaning naive implementations quickly bog down under heavy workloads.</p><h4>Decomposable BFV: Tackling the Scaling Bottleneck</h4><p>Fhenix’s <strong>decomposable BFV (DBFV)</strong> is a cryptographic breakthrough designed to confront this scaling barrier head-on. Instead of treating a large integer as a single, monolithic BFV ciphertext, DBFV <strong>decomposes the plaintext into smaller “limbs”</strong> (components) and encrypts them independently. Homomorphic arithmetic is performed across these limbs, merging the final results at the end.</p><p>This approach delivers two massive advantages:</p><ul><li><strong>Predictable Noise Management:</strong> Smaller limb sizes mean lower noise growth per operation. This delays the need for costly rescaling or modulus switching, enabling deeper circuits before requiring bootstrapping.</li><li><strong>Maximized Throughput:</strong> By parallelizing operations over multiple limbs and distributing computational load, DBFV achieves significantly higher throughput than standard single-modulus BFV deployments — especially for large-scale financial and stateful computations.</li></ul><p>For blockchain environments — where transaction volume and gas-cost constraints dictate viability — DBFV transforms exact FHE from a theoretical curiosity into a deployable building block for encrypted smart contracts.</p><h4>Throughput vs. Latency: Why DBFV Shines for Financial Logic</h4><p>In cryptographic discussions, the trade-off between <strong>throughput</strong> (total operations per second) and <strong>latency</strong> (time to complete a single operation) is decisive. While generic comparison-oriented schemes may optimize for single-operation latency, financial logic operates differently. Whether you are aggregating thousands of positions, running risk-engine evaluations, or settling batches of trades, <strong>aggregate throughput matters far more than microsecond-level comparison speeds</strong>.</p><p>DBFV is explicitly throughput-oriented. Its limb decomposition and parallel processing pipeline make it uniquely suited for:</p><ul><li>Batch-settlement engines</li><li>Privacy-preserving analytics over encrypted transaction histories</li><li>Stateful smart contracts that evolve balances and positions across multiple events</li></ul><p>In short, DBFV is superior for complex financial logic not because it wins every micro-benchmark, but because it aligns the performance curve with real-world fintech and blockchain workflows.</p><h3>Where FHE is Headed</h3><p>The discovery that arithmetic FHE schemes (BGV, BFV) can match non-arithmetic schemes on basic comparisons is profoundly encouraging. It proves that the traditional divide between “arithmetic” and “boolean” FHE is a continuous spectrum rather than a hard boundary.</p><p>DBFV and similar innovations point toward a larger truth: <strong>specialization is the key to practicality</strong>. Rather than a single universal scheme, we are moving toward a rich ecosystem of specialized, interoperable FHE primitives — tuned for low-latency interactive proofs, high-throughput batch processing, and exact integer arithmetic.</p><p>For builders across DeFi, enterprise privacy, and cryptography, FHE has officially transitioned from a pre-alpha toy into a production contender. The holy grail may not be a one-size-fits-all protocol, but a modular toolkit that finally lets us compute securely on any function — without pretending every function is equally easy to evaluate.</p><h4>Decomposable BFV: tackling the scaling bottleneck</h4><p>Fhenix’s decomposable BFV (DBFV) is a recent cryptographic innovation that attempts to squarely address this scaling problem. DBFV no longer treats a large integer as a single, monolithic BFV ciphertext. Instead, it <em>decomposes</em> the plaintext into smaller “limbs” (components) and encrypts them independently, then performs homomorphic arithmetic on the limbs, merging the result at the end.chainwire+2</p><p>This degree of decomposition has two main effects:</p><ul><li><strong>Noise management becomes more predictable</strong>: smaller limb sizes mean lower noise growth per operation, which defers the need for costly rescaling or modulus switching and allows deeper circuits before bootstrapping.markets.businessinsider+1</li><li><strong>Throughput improves for high‑volume workloads</strong>: by parallelizing operations over multiple limbs and distributing the computational load, DBFV can achieve a higher throughput than a naive single‑modulus BFV deployment, especially for large‑scale financial and stateful computations.</li></ul><p>For blockchain applications, where transaction volume and gas‑cost constraints are paramount, DBFV effectively turns “exact” FHE from a theoretical curiosity into a deployable building block for encrypted smart contracts and privacy‑preserving financial primitives.ainvest+2</p><h4>Throughput vs. latency: why DBFV shines for financial logic</h4><p>In many discussions of FHE, the trade‑off between <em>throughput</em> (total operations per second) and <em>latency</em> (time to complete a single operation) is subtle but decisive. Generic comparison‑oriented schemes may optimize for single‑operation latency, but for financial logic — where you are aggregating thousands of positions, running risk‑engine evaluations, or settling batches of trades — aggregate throughput matters more than the speed of any one</p><p>The DBFV approach, by design, is throughput‑oriented. Limb decomposition and parallel processing allow more operations to be queued and pipelined, which is especially beneficial for:</p><ul><li>batch‑settlement engines,</li><li>privacy‑preserving analytics over encrypted transaction histories, and</li><li>stateful smart contracts that evolve balances and positions over many events.</li></ul><p>In this light, DBFV becomes a “superior choice for complex financial logic” not because it is faster at every micro‑benchmark, but because it shifts the performance curve toward the regime where real‑world fintech and blockchain workloads operate.</p><p>The Below Solidity program OF CONFIDENTIAL POLYGON VAULT V2</p><p>* Architecture: * Polygon EVM + Fhenix CoFHE<br> Features:<br> 1. Encrypted deposits<br> 2. Encrypted withdrawals (ledger debit)<br> 3. Confidential peer-to-peer transfers<br> 4. Encrypted balance comparisons<br> 5. Encrypted operation results<br> 6. User-specific balance permissions<br> 7. Delegated encrypted balance access<br> 8. Confidential transaction accounting<br> 9. Administrative pause controls<br> 10. Ownership transfer</p><pre><br><br>// SPDX-License-Identifier: MIT<br>pragma solidity ^0.8.25;<br><br><br>import {<br>    FHE,<br>    euint64,<br>    ebool,<br>    externalEuint64<br>} from &quot;@fhenixprotocol/cofhe-contracts/FHE.sol&quot;;<br><br>import {<br>    Ownable<br>} from &quot;@openzeppelin/contracts/access/Ownable.sol&quot;;<br><br>import {<br>    Ownable2Step<br>} from &quot;@openzeppelin/contracts/access/Ownable2Step.sol&quot;;<br><br>import {<br>    Pausable<br>} from &quot;@openzeppelin/contracts/utils/Pausable.sol&quot;;<br><br><br>contract ConfidentialPolygonVaultV2<br>    is Ownable2Step,<br>       Pausable<br>{<br><br>    // ========================================================<br>    // CONSTANTS<br>    // ========================================================<br><br>    uint256 public constant MAX_USERS_PER_BATCH = 50;<br><br>    uint256 public constant MAX_OPERATION_DELAY = 30 days;<br><br>    // ========================================================<br>    // ENCRYPTED STORAGE<br>    // ========================================================<br><br>    mapping(address =&gt; euint64) private encryptedBalances;<br><br>    mapping(address =&gt; ebool) private lastOperationSuccess;<br><br>    // Confidential aggregate accounting.<br>    euint64 private encryptedTotalDeposits;<br><br>    euint64 private encryptedTotalDebits;<br><br>    // ========================================================<br>    // USER ACCESS CONTROL<br>    // ========================================================<br><br>    mapping(address =&gt; address) public balanceDelegate;<br><br>    mapping(address =&gt; bool) public isAuthorizedUser;<br><br>    // ========================================================<br>    // PUBLIC ACCOUNTING METADATA<br>    // ========================================================<br><br>    uint256 public totalDepositOperations;<br><br>    uint256 public totalTransferOperations;<br><br>    uint256 public totalDebitOperations;<br><br>    uint256 public registeredUsers;<br><br>    // ========================================================<br>    // ENUMS<br>    // ========================================================<br><br>    enum OperationType {<br>        Deposit,<br>        Transfer,<br>        Debit<br>    }<br><br>    // ========================================================<br>    // EVENTS<br>    // ========================================================<br><br>    event UserRegistered(<br>        address indexed user<br>    );<br><br>    event EncryptedDeposit(<br>        address indexed user,<br>        uint256 indexed operationId<br>    );<br><br>    event EncryptedTransfer(<br>        address indexed sender,<br>        address indexed receiver,<br>        uint256 indexed operationId<br>    );<br><br>    event EncryptedDebit(<br>        address indexed user,<br>        uint256 indexed operationId<br>    );<br><br>    event DelegateUpdated(<br>        address indexed user,<br>        address indexed delegate<br>    );<br><br>    event EncryptedOperationResult(<br>        address indexed user,<br>        OperationType operationType<br>    );<br><br>    event ConfidentialBalanceUpdated(<br>        address indexed user<br>    );<br><br>    // ========================================================<br>    // CUSTOM ERRORS<br>    // ========================================================<br><br>    error InvalidAddress();<br><br>    error UserNotRegistered();<br><br>    error AlreadyRegistered();<br><br>    error InvalidOperation();<br><br>    error SelfTransferNotAllowed();<br><br>    error DelegateCannotBeSelf();<br><br>    // ========================================================<br>    // CONSTRUCTOR<br>    // ========================================================<br><br>    constructor()<br>        Ownable(msg.sender)<br>    {}<br><br>    // ========================================================<br>    // MODIFIERS<br>    // ========================================================<br><br>    modifier onlyRegistered() {<br><br>        if (!isAuthorizedUser[msg.sender]) {<br>            revert UserNotRegistered();<br>        }<br><br>        _;<br>    }<br><br>    // ========================================================<br>    // USER REGISTRATION<br>    // ========================================================<br><br>    function registerUser()<br>        external<br>        whenNotPaused<br>    {<br><br>        if (isAuthorizedUser[msg.sender]) {<br>            revert AlreadyRegistered();<br>        }<br><br>        isAuthorizedUser[msg.sender] = true;<br><br>        registeredUsers++;<br><br>        // Initialize encrypted state.<br>        euint64 initialBalance = FHE.asEuint64(0);<br><br>        _storeBalance(msg.sender, initialBalance);<br><br>        emit UserRegistered(msg.sender);<br>    }<br><br>    // ========================================================<br>    // ENCRYPTED DEPOSIT<br>    // ========================================================<br><br>    function deposit(<br>        externalEuint64 encryptedAmount,<br>        bytes calldata inputProof<br>    )<br>        external<br>        whenNotPaused<br>        onlyRegistered<br>    {<br><br>        // Verify client-side encrypted input.<br>        euint64 amount = FHE.asEuint64(<br>            encryptedAmount,<br>            inputProof<br>        );<br><br>        // Read encrypted balance.<br>        euint64 currentBalance =<br>            encryptedBalances[msg.sender];<br><br>        // Perform encrypted addition.<br>        euint64 newBalance = FHE.add(<br>            currentBalance,<br>            amount<br>        );<br><br>        // Store and authorize encrypted state.<br>        _storeBalance(<br>            msg.sender,<br>            newBalance<br>        );<br><br>        // Update confidential aggregate.<br>        encryptedTotalDeposits = FHE.add(<br>            encryptedTotalDeposits,<br>            amount<br>        );<br><br>        FHE.allowThis(encryptedTotalDeposits);<br><br>        totalDepositOperations++;<br><br>        _recordResult(<br>            msg.sender,<br>            FHE.asEbool(true),<br>            OperationType.Deposit<br>        );<br><br>        emit EncryptedDeposit(<br>            msg.sender,<br>            totalDepositOperations<br>        );<br><br>        emit ConfidentialBalanceUpdated(msg.sender);<br>    }<br><br>    // ========================================================<br>    // ENCRYPTED TRANSFER<br>    // ========================================================<br><br>    function transferEncrypted(<br>        address recipient,<br>        externalEuint64 encryptedAmount,<br>        bytes calldata inputProof<br>    )<br>        external<br>        whenNotPaused<br>        onlyRegistered<br>    {<br><br>        if (<br>            recipient == address(0) ||<br>            !isAuthorizedUser[recipient]<br>        ) revert InvalidAddress();<br><br>        if (recipient == msg.sender) {<br>            revert SelfTransferNotAllowed();<br>        }<br><br>        // Validate encrypted transfer input.<br>        euint64 amount = FHE.asEuint64(<br>            encryptedAmount,<br>            inputProof<br>        );<br><br>        euint64 senderBalance =<br>            encryptedBalances[msg.sender];<br><br>        euint64 receiverBalance =<br>            encryptedBalances[recipient];<br><br>        // Confidential balance validation.<br>        ebool sufficientBalance = FHE.lte(<br>            amount,<br>            senderBalance<br>        );<br><br>        // Compute candidate balances.<br>        euint64 candidateSenderBalance = FHE.sub(<br>            senderBalance,<br>            amount<br>        );<br><br>        euint64 candidateReceiverBalance = FHE.add(<br>            receiverBalance,<br>            amount<br>        );<br><br>        // Select valid or unchanged balances without<br>        // exposing the comparison result.<br>        euint64 newSenderBalance = FHE.select(<br>            sufficientBalance,<br>            candidateSenderBalance,<br>            senderBalance<br>        );<br><br>        euint64 newReceiverBalance = FHE.select(<br>            sufficientBalance,<br>            candidateReceiverBalance,<br>            receiverBalance<br>        );<br><br>        // Commit encrypted state.<br>        _storeBalance(<br>            msg.sender,<br>            newSenderBalance<br>        );<br><br>        _storeBalance(<br>            recipient,<br>            newReceiverBalance<br>        );<br><br>        // Make operation status available only to sender.<br>        _recordResult(<br>            msg.sender,<br>            sufficientBalance,<br>            OperationType.Transfer<br>        );<br><br>        totalTransferOperations++;<br><br>        emit EncryptedTransfer(<br>            msg.sender,<br>            recipient,<br>            totalTransferOperations<br>        );<br>    }<br><br>    // ========================================================<br>    // ENCRYPTED WITHDRAWAL / LEDGER DEBIT<br>    // ========================================================<br><br>    function debitEncrypted(<br>        externalEuint64 encryptedAmount,<br>        bytes calldata inputProof<br>    )<br>        external<br>        whenNotPaused<br>        onlyRegistered<br>    {<br><br>        euint64 amount = FHE.asEuint64(<br>            encryptedAmount,<br>            inputProof<br>        );<br><br>        euint64 currentBalance =<br>            encryptedBalances[msg.sender];<br><br>        // Compare encrypted values.<br>        ebool sufficientBalance = FHE.lte(<br>            amount,<br>            currentBalance<br>        );<br><br>        euint64 candidateBalance = FHE.sub(<br>            currentBalance,<br>            amount<br>        );<br><br>        // No balance change if funds are insufficient.<br>        euint64 newBalance = FHE.select(<br>            sufficientBalance,<br>            candidateBalance,<br>            currentBalance<br>        );<br><br>        _storeBalance(<br>            msg.sender,<br>            newBalance<br>        );<br><br>        // Record encrypted debit amount only when valid.<br>        euint64 effectiveDebit = FHE.select(<br>            sufficientBalance,<br>            amount,<br>            FHE.asEuint64(0)<br>        );<br><br>        encryptedTotalDebits = FHE.add(<br>            encryptedTotalDebits,<br>            effectiveDebit<br>        );<br><br>        FHE.allowThis(encryptedTotalDebits);<br><br>        _recordResult(<br>            msg.sender,<br>            sufficientBalance,<br>            OperationType.Debit<br>        );<br><br>        totalDebitOperations++;<br><br>        emit EncryptedDebit(<br>            msg.sender,<br>            totalDebitOperations<br>        );<br>    }<br><br>    // ========================================================<br>    // ENCRYPTED BALANCE ACCESS<br>    // ========================================================<br><br>    function getEncryptedBalance(<br>        address account<br>    )<br>        external<br>        view<br>        returns (bytes32)<br>    {<br><br>        if (!isAuthorizedUser[account]) {<br>            revert UserNotRegistered();<br>        }<br><br>        return euint64.unwrap(<br>            encryptedBalances[account]<br>        );<br>    }<br><br>    // ========================================================<br>    // ENCRYPTED OPERATION STATUS<br>    // ========================================================<br><br>    function getEncryptedOperationStatus()<br>        external<br>        view<br>        onlyRegistered<br>        returns (bytes32)<br>    {<br><br>        return ebool.unwrap(<br>            lastOperationSuccess[msg.sender]<br>        );<br>    }<br><br>    // ========================================================<br>    // USER DELEGATION<br>    // ========================================================<br><br>    function setBalanceDelegate(<br>        address delegate<br>    )<br>        external<br>        whenNotPaused<br>        onlyRegistered<br>    {<br><br>        if (delegate == address(0)) {<br>            revert InvalidAddress();<br>        }<br><br>        if (delegate == msg.sender) {<br>            revert DelegateCannotBeSelf();<br>        }<br><br>        balanceDelegate[msg.sender] = delegate;<br><br>        // Authorize delegate to access current ciphertext.<br>        FHE.allow(<br>            encryptedBalances[msg.sender],<br>            delegate<br>        );<br><br>        emit DelegateUpdated(<br>            msg.sender,<br>            delegate<br>        );<br>    }<br><br>    // ========================================================<br>    // REVOKE DELEGATE<br>    // ========================================================<br><br>    function revokeBalanceDelegate()<br>        external<br>        onlyRegistered<br>    {<br><br>        balanceDelegate[msg.sender] = address(0);<br><br>        // CoFHE ACL permissions already granted to previous<br>        // ciphertexts may not be revocable by this assignment.<br>        // Use a fresh ciphertext and protocol-supported ACL<br>        // revocation when available.<br>        emit DelegateUpdated(<br>            msg.sender,<br>            address(0)<br>        );<br>    }<br><br>    // ========================================================<br>    // ENCRYPTED BALANCE AGGREGATION<br>    // ========================================================<br><br>    function getEncryptedTotalDeposits()<br>        external<br>        view<br>        onlyOwner<br>        returns (bytes32)<br>    {<br><br>        return euint64.unwrap(<br>            encryptedTotalDeposits<br>        );<br>    }<br><br>    function getEncryptedTotalDebits()<br>        external<br>        view<br>        onlyOwner<br>        returns (bytes32)<br>    {<br><br>        return euint64.unwrap(<br>            encryptedTotalDebits<br>        );<br>    }<br><br>    // ========================================================<br>    // INTERNAL BALANCE MANAGEMENT<br>    // ========================================================<br><br>    function _storeBalance(<br>        address account,<br>        euint64 newBalance<br>    )<br>        internal<br>    {<br><br>        encryptedBalances[account] = newBalance;<br><br>        // Persistent contract-level permission.<br>        FHE.allowThis(newBalance);<br><br>        // Persistent account-level permission.<br>        FHE.allow(<br>            newBalance,<br>            account<br>        );<br><br>        // Reapply configured delegate to new ciphertext.<br>        address delegate = balanceDelegate[account];<br><br>        if (delegate != address(0)) {<br>            FHE.allow(<br>                newBalance,<br>                delegate<br>            );<br>        }<br><br>        emit ConfidentialBalanceUpdated(account);<br>    }<br><br>    // ========================================================<br>    // INTERNAL OPERATION RESULT<br>    // ========================================================<br><br>    function _recordResult(<br>        address account,<br>        ebool result,<br>        OperationType operationType<br>    )<br>        internal<br>    {<br><br>        lastOperationSuccess[account] = result;<br><br>        FHE.allowThis(result);<br><br>        FHE.allow(<br>            result,<br>            account<br>        );<br><br>        emit EncryptedOperationResult(<br>            account,<br>            operationType<br>        );<br>    }<br><br>    // ========================================================<br>    // ADMINISTRATIVE FUNCTIONS<br>    // ========================================================<br><br>    function pause()<br>        external<br>        onlyOwner<br>    {<br><br>        _pause();<br>    }<br><br>    function unpause()<br>        external<br>        onlyOwner<br>    {<br><br>        _unpause();<br>    }<br><br>    // ========================================================<br>    // PUBLIC PROTOCOL INFORMATION<br>    // ========================================================<br><br>    function getProtocolStatistics()<br>        external<br>        view<br>        returns (<br>            uint256 users,<br>            uint256 deposits,<br>            uint256 transfers,<br>            uint256 debits,<br>            bool pausedStatus<br>        )<br>    {<br><br>        return (<br>            registeredUsers,<br>            totalDepositOperations,<br>            totalTransferOperations,<br>            totalDebitOperations,<br>            paused()<br>        );<br>    }<br>}</pre><h4>Contemplations: where FHE is headed</h4><p>The observation that FHE schemes for arithmetic circuits (BGV, BFV) can match or approach the performance of non‑arithmetic schemes on basic comparisons is profoundly encouraging. It suggests that the old dichotomy — “arithmetic vs. boolean” FHE — is not a hard boundary, but a spectrum along which algorithmic innovation can blur the gap.</p><p>DBFV and similar advances push a further thesis: that <em>specialization</em> is the key to making FHE practical. There will not be a single universal scheme, but rather a family of schemes tuned to different regimes — low‑latency interactive proofs, high‑throughput batch processing, and exact integer arithmetic for finance and governance. For blockchain and DeFi builders, this means an emerging toolkit where FHE is no longer a pre‑alpha toy, but a contender for real‑world, privacy‑preserving application logic — if carefully calibrated to the right use case.</p><p>In time, the “holy grail” of FHE may not be a one‑size‑fits‑all scheme, but a rich ecosystem of specialized, interoperable, and composable FHE primitives that finally let us compute <em>any</em> function on encrypted values — without pretending that every function is equally easy to evaluate.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b942272ef01e" width="1" height="1" alt=""><hr><p><a href="https://blog.blockmagnates.com/the-architecture-of-privacy-unlocking-practical-fhe-for-finance-and-blockchains-b942272ef01e">The Architecture of Privacy: Unlocking Practical FHE for Finance and Blockchains</a> was originally published in <a href="https://blog.blockmagnates.com/">Block Magnates</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How QR Code Payments Work: A UPI Walkthrough with Paytm and PhonePe, and an AI Risk Engine in…]]></title>
            <link>https://blog.stackademic.com/how-qr-code-payments-work-a-upi-walkthrough-with-paytm-and-phonepe-and-an-ai-risk-engine-in-23ab96055c0a?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/23ab96055c0a</guid>
            <category><![CDATA[microservice-architecture]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[qr-code-payments]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Thu, 22 Oct 2026 02:43:28 GMT</pubDate>
            <atom:updated>2026-10-22T02:53:25.438Z</atom:updated>
            <content:encoded><![CDATA[<h3>How QR Code Payments Work: A UPI Walkthrough with Paytm and PhonePe, and an AI Risk Engine in Keras/TensorFlow</h3><h3>Introduction:</h3><p>It is 8:10 a.m. at a street-side tea stall. The chaiwala pours a cup and points to a laminated square taped to his counter. You raise your phone, there is a soft <em>beep</em>, you type a PIN, and two phones buzz at once: <strong>“Payment Successful.”</strong> The whole thing took about four seconds.</p><p>Behind those four seconds are a QR standard, a national payment switch, two banks, and a risk engine making a decision. This article explains each piece and then builds the risk engine in Python with <strong>Keras and TensorFlow</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YoXJxpay9q2x0sD1sirNVA.png" /><figcaption>The Entire Workflow of QR Code Payments Presented as follows</figcaption></figure><h4>1. What a UPI QR Code Actually Contains</h4><p>A UPI QR is not an image of money. It is a <strong>text string</strong> drawn as a picture, usually a deep link like this:</p><p>upi://pay?pa=chaistall@okaxis&amp;pn=Sharma%20Tea%20Stall&amp;am=15.00&amp;cu=INR&amp;tn=Order%201042&amp;tr=ORD1042&amp;mc=5812</p><p><strong>Static QR:</strong> one permanent code per shop. It usually has no amount, so the payer types it in. Some shops print a fixed-price code, like the ₹50 chai in this example. <strong>Dynamic QR:</strong> generated per bill by a POS or app. It carries the amount and order reference, and it expires. This reduces typing errors and replay fraud.</p><blockquote><strong><em>Author’s note:</em></strong><em> The QR does not move money. It only pre-fills the payment request, so you never type the merchant’s details by hand.</em></blockquote><h4>2. The Journey of ₹50, Step by Step</h4><ol><li><strong>Scan.</strong> You open Paytm, PhonePe, or GPay and tap <em>Scan</em>. The app decodes the string and shows the payee name and amount.</li><li><strong>Confirm and authenticate.</strong> You enter your <strong>UPI PIN</strong>. The PIN is encrypted on the device and never visible to the app or the merchant.</li><li><strong>PSP to NPCI.</strong> Your app, acting as a TPAP (Third Party App Provider) backed by a PSP bank, sends the request to <strong>NPCI’s UPI switch</strong>.</li><li><strong>Debit.</strong> NPCI routes the request to your <strong>remitter (issuer) bank</strong>, which checks the PIN and balance and debits your account.</li><li><strong>Credit.</strong> NPCI forwards the credit to the <strong>beneficiary (acquirer) bank</strong> of the merchant.</li><li><strong>Confirmation.</strong> Both sides receive a push notification or SMS with a transaction ID (UTR). Interbank settlement runs on NPCI’s cycles. Merchants may see funds the same day or T+1, depending on their bank and settlement arrangement, but the confirmation is immediate.</li></ol><blockquote><strong><em>Author’s note:</em></strong><em> The merchant’s app never sees your bank balance or PIN. Only the minimum data travels between parties.</em></blockquote><p><strong>Two clarifications on the usual retelling:</strong></p><ul><li>The flow is payer app → NPCI → remitter bank (debit) → NPCI → beneficiary bank (credit). It is not “issuer, then acquirer” as two separate hops.</li><li>After RBI’s 2024 restrictions on Paytm Payments Bank, Paytm’s UPI runs as a TPAP over multiple partner banks.</li></ul><h4>3. Security Layers</h4><ul><li><strong>Two factors:</strong> the registered device (SIM binding) and the UPI PIN.</li><li><strong>Encrypted PIN capture</strong> through NPCI’s common library.</li><li><strong>Dynamic QR</strong> adds expiry and a unique reference, which blocks replay.</li><li><strong>Payee name display</strong> lets you catch a swapped QR sticker.</li><li><strong>Risk monitoring</strong> by PSPs, banks, and NPCI scores each transaction for anomalies.</li></ul><p>The last layer is where machine learning fits. <strong>Keras and TensorFlow do not process the payment.</strong> They help decide whether a given payment looks normal.</p><h4>4. The Program: UPI QR Toolkit and Fraud-Risk Engine</h4><p>The program has four parts:</p><ol><li><strong>QR layer:</strong> build, sign, parse, and validate UPI URIs (static and dynamic) with expiry and an HMAC tamper check. The signature scheme is a demo, not NPCI’s real signing.</li><li><strong>Data layer:</strong> a synthetic transaction generator. Real data is private, so it is simulated.</li><li><strong>Model layer:</strong> a Keras residual MLP with embeddings, focal loss, AdamW with cosine decay, early stopping, and a tf.data pipeline.</li><li><strong>Decision layer:</strong> hard rules first, then ML, producing <strong>APPROVE / STEP-UP / DECLINE</strong>, with threshold tuning, TFLite export, and a latency benchmark.</li></ol><h4>5. Python Program Approach to explain the concept of QR code Payment</h4><pre>pip install tensorflow scikit-learn num.py qrcode[pil]</pre><pre>import hmac, hashlib, re, time, os<br>from dataclasses import dataclass<br>from decimal import Decimal, InvalidOperation<br>from typing import Optional, Tuple<br>from urllib.parse import urlencode, urlparse, parse_qs, quote<br>from datetime import datetime<br><br>import numpy as np<br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras import layers, regularizers<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import (roc_auc_score, average_precision_score,<br>                             precision_recall_curve, confusion_matrix,<br>                             classification_report)<br><br>SEED = 42<br>np.random.seed(SEED)<br>tf.random.set_seed(SEED)<br>keras.utils.set_random_seed(SEED)<br><br>SECRET = b&quot;demo-secret-rotate-me&quot;          # In reality: HSM / KMS-managed keys<br>VPA_RE = re.compile(r&quot;^[a-zA-Z0-9.\-_]{2,256}@[a-zA-Z][a-zA-Z0-9]{1,63}$&quot;)<br>MAX_UPI_AMOUNT = Decimal(&quot;100000&quot;)         # typical ₹1 lakh P2M cap (varies by category)<br><br>@dataclass<br>class UpiQR:<br>    pa: str                                 # payee VPA<br>    pn: str                                 # payee name<br>    am: Optional[str] = None                # amount<br>    cu: str = &quot;INR&quot;<br>    tn: str = &quot;&quot;                            # note<br>    tr: str = &quot;&quot;                            # transaction reference / order id<br>    mc: str = &quot;0000&quot;                        # merchant category code<br>    exp: Optional[int] = None               # DEMO: expiry epoch (dynamic only)<br>    sig: Optional[str] = None               # DEMO: HMAC signature (dynamic only)<br><br>def _canon(q: UpiQR) -&gt; bytes:<br>    return &quot;|&quot;.join([q.pa, q.pn, q.am or &quot;&quot;, q.tr, str(q.exp or &quot;&quot;)]).encode()<br><br>def _sign(q: UpiQR) -&gt; str:<br>    return hmac.new(SECRET, _canon(q), hashlib.sha256).hexdigest()[:32]<br><br>def _valid_amount(a: str) -&gt; bool:<br>    try:<br>        d = Decimal(a)<br>    except InvalidOperation:<br>        return False<br>    return Decimal(&quot;0.01&quot;) &lt;= d &lt;= MAX_UPI_AMOUNT and d == d.quantize(Decimal(&quot;0.01&quot;))<br><br>def create_static_qr(vpa, name, mcc=&quot;5812&quot;, amount: Optional[str] = None) -&gt; UpiQR:<br>    return UpiQR(pa=vpa, pn=name, am=amount, mc=mcc)<br><br>def create_dynamic_qr(vpa, name, amount, order_id, note=&quot;&quot;, mcc=&quot;5812&quot;, ttl=300) -&gt; UpiQR:<br>    q = UpiQR(pa=vpa, pn=name, am=amount, tn=note, tr=order_id, mc=mcc,<br>              exp=int(time.time()) + ttl)<br>    q.sig = _sign(q)<br>    return q<br><br>def to_uri(q: UpiQR) -&gt; str:<br>    params = {k: v for k, v in vars(q).items() if v not in (None, &quot;&quot;)}<br>    return &quot;upi://pay?&quot; + urlencode(params, quote_via=quote)<br><br>def parse_uri(uri: str) -&gt; UpiQR:<br>    u = urlparse(uri)<br>    if u.scheme != &quot;upi&quot; or u.netloc != &quot;pay&quot;:<br>        raise ValueError(&quot;Not a UPI pay URI&quot;)<br>    qs = {k: v[0] for k, v in parse_qs(u.query).items()}<br>    if &quot;pa&quot; not in qs or &quot;pn&quot; not in qs:<br>        raise ValueError(&quot;Missing mandatory fields (pa, pn)&quot;)<br>    return UpiQR(pa=qs[&quot;pa&quot;], pn=qs[&quot;pn&quot;], am=qs.get(&quot;am&quot;), cu=qs.get(&quot;cu&quot;, &quot;INR&quot;),<br>                 tn=qs.get(&quot;tn&quot;, &quot;&quot;), tr=qs.get(&quot;tr&quot;, &quot;&quot;), mc=qs.get(&quot;mc&quot;, &quot;0000&quot;),<br>                 exp=int(qs[&quot;exp&quot;]) if &quot;exp&quot; in qs else None, sig=qs.get(&quot;sig&quot;))<br><br>def verify_qr(q: UpiQR, now: Optional[float] = None) -&gt; Tuple[bool, str]:<br>    now = now or time.time()<br>    if not VPA_RE.match(q.pa):                     return False, &quot;Invalid VPA format&quot;<br>    if q.cu != &quot;INR&quot;:                              return False, &quot;Unsupported currency&quot;<br>    if q.am is not None and not _valid_amount(q.am): return False, &quot;Invalid amount&quot;<br>    if q.sig is not None or q.exp is not None:     # dynamic QR path<br>        if q.sig is None or q.exp is None:         return False, &quot;Incomplete dynamic QR&quot;<br>        if not hmac.compare_digest(q.sig, _sign(q)): return False, &quot;Signature mismatch (tampered QR)&quot;<br>        if now &gt; q.exp:                            return False, &quot;Dynamic QR expired&quot;<br>    return True, &quot;OK&quot;<br><br>def save_qr_image(uri: str, path: str):<br>    try:<br>        import qrcode<br>        qrcode.make(uri).save(path)<br>        print(f&quot;  QR image saved → {path}&quot;)<br>    except ImportError:<br>        print(&quot;  (pip install qrcode[pil] to render images)&quot;)<br><br><br><br>NUM_FEATURES = [&quot;amount&quot;, &quot;log_amount&quot;, &quot;hour_sin&quot;, &quot;hour_cos&quot;, &quot;is_dynamic&quot;,<br>                &quot;qr_age_s&quot;, &quot;log_merchant_age&quot;, &quot;payer_txn_1h&quot;, &quot;log_amt_ratio&quot;,<br>                &quot;new_payee&quot;, &quot;new_device&quot;, &quot;pin_failures&quot;, &quot;geo_km&quot;,<br>                &quot;sig_valid&quot;, &quot;amount_edited&quot;]<br>N_MERCHANT_CAT, N_APPS = 12, 4                      # PhonePe, Paytm, GPay, Other<br>APP_ID = {&quot;phonepe&quot;: 0, &quot;paytm&quot;: 1, &quot;gpay&quot;: 2, &quot;other&quot;: 3}<br><br>def assemble_numeric(amount, hour, is_dynamic, qr_age, merchant_age, velocity,<br>                     amt_ratio, new_payee, new_device, pin_fail, geo_km,<br>                     sig_valid, amount_edited) -&gt; np.ndarray:<br>    f = lambda x: np.atleast_1d(np.asarray(x, dtype=&quot;float32&quot;))<br>    amount, hour = f(amount), f(hour)<br>    return np.column_stack([<br>        amount, np.log1p(amount),<br>        np.sin(2*np.pi*hour/24), np.cos(2*np.pi*hour/24),<br>        f(is_dynamic), f(qr_age), np.log1p(f(merchant_age)), f(velocity),<br>        np.log(np.clip(f(amt_ratio), 1e-3, None)),<br>        f(new_payee), f(new_device), f(pin_fail), f(geo_km),<br>        f(sig_valid), f(amount_edited)]).astype(&quot;float32&quot;)<br><br>def synth(n: int, seed: int = 7):<br>    r = np.random.default_rng(seed)<br>    mcat = r.integers(0, N_MERCHANT_CAT, n)<br>    app = r.integers(0, N_APPS, n)<br>    dyn = r.random(n) &lt; 0.45<br>    amount = np.clip(r.lognormal(5.0, 1.2, n), 1, 100000)<br>    hour = r.integers(0, 24, n)<br>    qr_age = np.where(dyn, r.exponential(60, n), 0)<br>    m_age = r.exponential(400, n) + 1<br>    vel = r.poisson(0.6, n)<br>    ratio = amount / np.clip(r.lognormal(5.0, 0.6, n), 10, None)<br>    new_payee = r.random(n) &lt; 0.35<br>    new_dev = r.random(n) &lt; 0.04<br>    pin_fail = r.choice([0, 1, 2, 3], n, p=[.93, .05, .015, .005])<br>    geo = r.exponential(8, n)<br>    sig_ok = ~(dyn &amp; (r.random(n) &lt; 0.01))<br>    edited = dyn &amp; (r.random(n) &lt; 0.01)<br><br>    z = (-4.0 + 0.7*np.clip(np.log(ratio), 0, 4) + 0.6*np.clip(vel-1, 0, 5)<br>         + 1.2*new_dev + 0.9*pin_fail + 0.015*np.minimum(geo, 300)<br>         + 0.8*(new_payee &amp; (amount &gt; 5000)) + 0.6*(hour &lt; 5)<br>         + 2.5*(~sig_ok) + 2.0*edited - 0.25*np.log1p(m_age/30)<br>         + 0.5*(qr_age &gt; 600) + r.normal(0, 0.5, n))<br>    y = (r.random(n) &lt; 1/(1+np.exp(-z))).astype(&quot;float32&quot;)<br><br>    X = assemble_numeric(amount, hour, dyn, qr_age, m_age, vel, ratio,<br>                         new_payee, new_dev, pin_fail, geo, sig_ok, edited)<br>    return X, mcat.astype(&quot;int32&quot;), app.astype(&quot;int32&quot;), y<br><br><br><br>def pack(X, mcat, app):<br>    return {&quot;num&quot;: X, &quot;merchant_cat&quot;: mcat.reshape(-1, 1), &quot;psp_app&quot;: app.reshape(-1, 1)}<br><br>def make_ds(X, mcat, app, y, shuffle=False, batch=1024):<br>    ds = tf.data.Dataset.from_tensor_slices((pack(X, mcat, app), y))<br>    if shuffle:<br>        ds = ds.shuffle(len(y), seed=SEED, reshuffle_each_iteration=True)<br>    return ds.batch(batch).cache().prefetch(tf.data.AUTOTUNE)<br><br>def build_model(norm: layers.Normalization, total_steps: int) -&gt; keras.Model:<br>    l2 = regularizers.l2(1e-5)<br>    num_in = keras.Input((len(NUM_FEATURES),), name=&quot;num&quot;)<br>    m_in = keras.Input((1,), dtype=&quot;int32&quot;, name=&quot;merchant_cat&quot;)<br>    a_in = keras.Input((1,), dtype=&quot;int32&quot;, name=&quot;psp_app&quot;)<br><br>    x = layers.Concatenate()([<br>        norm(num_in),<br>        layers.Flatten()(layers.Embedding(N_MERCHANT_CAT, 4)(m_in)),<br>        layers.Flatten()(layers.Embedding(N_APPS, 2)(a_in))])<br>    x = layers.Dense(128, activation=&quot;swish&quot;, kernel_regularizer=l2)(x)<br><br>    for units in (128, 64):                         # residual blocks<br>        h = layers.Dense(units, use_bias=False, kernel_regularizer=l2)(x)<br>        h = layers.BatchNormalization()(h)<br>        h = layers.Activation(&quot;swish&quot;)(h)<br>        h = layers.Dropout(0.2)(h)<br>        skip = x if x.shape[-1] == units else layers.Dense(units, use_bias=False)(x)<br>        x = layers.Add()([h, skip])<br><br>    out = layers.Dense(1, activation=&quot;sigmoid&quot;, dtype=&quot;float32&quot;, name=&quot;risk&quot;)(x)<br>    model = keras.Model([num_in, m_in, a_in], out, name=&quot;upi_risk_net&quot;)<br><br>    lr = keras.optimizers.schedules.CosineDecay(2e-3, decay_steps=total_steps, alpha=0.02)<br>    model.compile(<br>        optimizer=keras.optimizers.AdamW(lr, weight_decay=1e-4, clipnorm=1.0),<br>        loss=keras.losses.BinaryFocalCrossentropy(apply_class_balancing=True,<br>                                                  alpha=0.75, gamma=2.0),<br>        metrics=[keras.metrics.AUC(name=&quot;roc_auc&quot;),<br>                 keras.metrics.AUC(name=&quot;pr_auc&quot;, curve=&quot;PR&quot;),<br>                 keras.metrics.Precision(name=&quot;precision&quot;),<br>                 keras.metrics.Recall(name=&quot;recall&quot;)])<br>    return model<br><br>def pick_thresholds(y, s, min_recall=0.85, min_precision=0.80):<br>    &quot;&quot;&quot;t_low → STEP-UP above it; t_high → DECLINE above it.&quot;&quot;&quot;<br>    p, r, t = precision_recall_curve(y, s)<br>    p, r = p[:-1], r[:-1]<br>    i_low = np.where(r &gt;= min_recall)[0]<br>    i_high = np.where(p &gt;= min_precision)[0]<br>    t_low = t[i_low[-1]] if len(i_low) else t[0]<br>    t_high = t[i_high[0]] if len(i_high) else t[-1]<br>    return float(t_low), float(max(t_high, t_low))<br><br>def export_and_benchmark(model, sample):<br>    model.save(&quot;upi_risk.keras&quot;)<br>    try:<br>        conv = tf.lite.TFLiteConverter.from_keras_model(model)<br>        conv.optimizations = [tf.lite.Optimize.DEFAULT]<br>        open(&quot;upi_risk.tflite&quot;, &quot;wb&quot;).write(conv.convert())<br>        print(f&quot;  TFLite size: {os.path.getsize(&#39;upi_risk.tflite&#39;)/1024:.1f} KB&quot;)<br>    except Exception as e:<br>        print(&quot;  TFLite export skipped:&quot;, e)<br>    infer = tf.function(lambda d: model(d, training=False), reduce_retracing=True)<br>    infer(sample)                                   # warm-up / trace<br>    t0 = time.perf_counter()<br>    for _ in range(200):<br>        infer(sample)<br>    print(f&quot;  Single-txn latency: {(time.perf_counter()-t0)/200*1000:.2f} ms&quot;)<br><br><br><br>def decide(uri, ctx, model, t_low, t_high):<br>    try:<br>        q = parse_uri(uri)<br>    except ValueError as e:<br>        return {&quot;action&quot;: &quot;DECLINE&quot;, &quot;why&quot;: f&quot;Rule: {e}&quot;}<br>    ok, why = verify_qr(q)<br>    if not ok:<br>        return {&quot;action&quot;: &quot;DECLINE&quot;, &quot;why&quot;: f&quot;Rule: {why}&quot;}<br><br>    is_dyn = q.sig is not None<br>    amount = float(ctx.get(&quot;amount&quot;) or q.am)<br>    edited = bool(is_dyn and q.am and abs(float(q.am) - amount) &gt; 1e-9)<br>    X = assemble_numeric(<br>        amount, ctx.get(&quot;hour&quot;, datetime.now().hour), is_dyn, ctx.get(&quot;qr_age&quot;, 0),<br>        ctx.get(&quot;merchant_age&quot;, 365), ctx.get(&quot;velocity&quot;, 0),<br>        amount / ctx.get(&quot;payer_avg&quot;, 150.0), ctx.get(&quot;new_payee&quot;, False),<br>        ctx.get(&quot;new_device&quot;, False), ctx.get(&quot;pin_fail&quot;, 0), ctx.get(&quot;geo_km&quot;, 2.0),<br>        True, edited)<br>    inp = pack(X, np.array([int(q.mc[:2]) % N_MERCHANT_CAT], &quot;int32&quot;),<br>               np.array([APP_ID.get(ctx.get(&quot;app&quot;, &quot;other&quot;), 3)], &quot;int32&quot;))<br>    score = float(model(inp, training=False).numpy()[0, 0])<br>    action = &quot;DECLINE&quot; if score &gt;= t_high else &quot;STEP-UP&quot; if score &gt;= t_low else &quot;APPROVE&quot;<br>    return {&quot;action&quot;: action, &quot;risk&quot;: round(score, 4), &quot;why&quot;: &quot;ML risk score&quot;}<br><br><br>def main():<br>    print(&quot;▶ Building dataset …&quot;)<br>    X, mc, ap, y = synth(120_000)<br>    print(f&quot;  Fraud prevalence: {y.mean():.2%}&quot;)<br>    idx = np.arange(len(y))<br>    tr, tmp = train_test_split(idx, test_size=0.30, stratify=y, random_state=SEED)<br>    va, te = train_test_split(tmp, test_size=0.50, stratify=y[tmp], random_state=SEED)<br><br>    norm = layers.Normalization(axis=-1)<br>    norm.adapt(X[tr])                               # fit on train only (no leakage)<br><br>    EPOCHS, BATCH = 40, 1024<br>    steps = int(np.ceil(len(tr)/BATCH)) * EPOCHS<br>    model = build_model(norm, steps)<br><br>    ds_tr = make_ds(X[tr], mc[tr], ap[tr], y[tr], shuffle=True, batch=BATCH)<br>    ds_va = make_ds(X[va], mc[va], ap[va], y[va], batch=BATCH)<br><br>    print(&quot;▶ Training …&quot;)<br>    model.fit(ds_tr, validation_data=ds_va, epochs=EPOCHS, verbose=2,<br>              callbacks=[keras.callbacks.EarlyStopping(<br>                  monitor=&quot;val_pr_auc&quot;, mode=&quot;max&quot;, patience=6,<br>                  restore_best_weights=True)])<br><br>    print(&quot;▶ Threshold tuning on validation …&quot;)<br>    s_va = model.predict(pack(X[va], mc[va], ap[va]), batch_size=4096, verbose=0).ravel()<br>    t_low, t_high = pick_thresholds(y[va], s_va)<br>    print(f&quot;  STEP-UP ≥ {t_low:.3f} | DECLINE ≥ {t_high:.3f}&quot;)<br><br>    print(&quot;▶ Held-out test evaluation …&quot;)<br>    s_te = model.predict(pack(X[te], mc[te], ap[te]), batch_size=4096, verbose=0).ravel()<br>    print(f&quot;  ROC-AUC: {roc_auc_score(y[te], s_te):.4f} | &quot;<br>          f&quot;PR-AUC: {average_precision_score(y[te], s_te):.4f}&quot;)<br>    pred = (s_te &gt;= t_high).astype(int)<br>    print(confusion_matrix(y[te], pred))<br>    print(classification_report(y[te], pred, digits=3, zero_division=0))<br><br>    print(&quot;▶ Export &amp; latency …&quot;)<br>    export_and_benchmark(model, pack(X[:1], mc[:1], ap[:1]))<br><br>    print(&quot;\n▶ Live scenarios&quot;)<br>    # 1) Static chai QR (₹50), PhonePe<br>    chai = to_uri(create_static_qr(&quot;chaistall@okaxis&quot;, &quot;Sharma Tea Stall&quot;, amount=&quot;50.00&quot;))<br>    print(&quot; Static ₹50 / PhonePe :&quot;, decide(chai, {&quot;app&quot;: &quot;phonepe&quot;}, model, t_low, t_high))<br>    save_qr_image(chai, &quot;chai_static.png&quot;)<br><br>    # 2) Dynamic ₹15 QR with order note, Paytm<br>    dyn = to_uri(create_dynamic_qr(&quot;chaistall@okaxis&quot;, &quot;Sharma Tea Stall&quot;,<br>                                   &quot;15.00&quot;, &quot;ORD1042&quot;, &quot;Masala chai&quot;))<br>    print(&quot; Dynamic ₹15 / Paytm  :&quot;, decide(dyn, {&quot;app&quot;: &quot;paytm&quot;, &quot;qr_age&quot;: 20}, model, t_low, t_high))<br><br>    # 3) Tampered dynamic QR (amount changed after signing)<br>    print(&quot; Tampered amount      :&quot;, decide(dyn.replace(&quot;am=15.00&quot;, &quot;am=1.00&quot;),<br>                                            {&quot;app&quot;: &quot;paytm&quot;}, model, t_low, t_high))<br><br>    # 4) Suspicious pattern: new device, new payee, 3 AM, large amount, PIN retries<br>    risky = to_uri(create_static_qr(&quot;newmerch@okicici&quot;, &quot;Unknown Traders&quot;, mcc=&quot;7995&quot;))<br>    print(&quot; Suspicious pattern   :&quot;, decide(risky, {<br>        &quot;app&quot;: &quot;gpay&quot;, &quot;amount&quot;: 48000, &quot;hour&quot;: 3, &quot;new_payee&quot;: True, &quot;new_device&quot;: True,<br>        &quot;pin_fail&quot;: 2, &quot;velocity&quot;: 5, &quot;geo_km&quot;: 400, &quot;merchant_age&quot;: 3, &quot;payer_avg&quot;: 200},<br>        model, t_low, t_high))<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4>6. Engineering concepts at play include:</h4><p><strong>1.</strong> <strong>Stateless Transactions: Each order is a new event, independent &amp; traceable.</strong></p><p><strong>2.</strong> <strong>Idempotent APIs: Prevents duplicate charges during retries or failed attempts.</strong></p><p><strong>3.</strong> <strong>Tokenized URLs: Secure QR payloads with minimal risk of tampering.</strong></p><p><strong>4.</strong> <strong>Real-time Event Flow: Backend → PSP → Wallet → Gateway in under a second.</strong></p><p><strong>5.</strong> <strong>Microservice Architecture:Loose coupling between merchant app, PSP, and wallets.</strong></p><p><strong>6.</strong> <strong>QR as an Interface: A visual bridge between offline experience &amp; online processing.</strong></p><h4>7. Conclusion :</h4><p>QR code Payments where backend engineering meets real-world UX. Consumers just tap, scan, and go — oblivious to the magic happening behind the scenes. Kudos to all the engineers designing these flows, securing APIs, handling retries, and building seamless payment experiences for millions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23ab96055c0a" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/how-qr-code-payments-work-a-upi-walkthrough-with-paytm-and-phonepe-and-an-ai-risk-engine-in-23ab96055c0a">How QR Code Payments Work: A UPI Walkthrough with Paytm and PhonePe, and an AI Risk Engine in…</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Privacy vs. Transparency: The DPDP Act, the RTI Act, and India’s Constitutional Balance]]></title>
            <link>https://medium.com/codetodeploy/privacy-vs-transparency-the-dpdp-act-the-rti-act-and-indias-constitutional-balance-bc034d77abdf?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc034d77abdf</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[rti-act]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[dpdp-act]]></category>
            <category><![CDATA[transparency]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 07:42:32 GMT</pubDate>
            <atom:updated>2026-10-10T07:47:13.717Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*50HuKmg4_bvMAHIcnvjR4w.png" /></figure><h3>Introduction: Two Rights, One Statute</h3><p>India’s democratic architecture rests on two ideas that are usually allies but are now being pitted against each other. The first is that citizens have a right to know how they are governed. The second is that individuals have a right to be left alone, to control what is known about them. The first found its most powerful statutory expression in the Right to Information (RTI) Act, 2005. The second was elevated to a fundamental right by the Supreme Court in <em>Justice K.S. Puttaswamy v. Union of India</em> (2017), and was given a legislative framework through the Digital Personal Data Protection (DPDP) Act, 2023.</p><blockquote><em>💥 </em><strong><em>Master AI &amp; Tech Skills</em></strong><br> 🎓 Get Up to 50% OFF Premium Courses<br> ⏰ Limited-Time Offer<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITogvtC6mF2BUF35Qsd9pg.png" /></figure><p>The collision came through a single, quiet provision. The DPDP Act amended Section 8(1)(j) of the RTI Act, the clause that governs when personal information may be withheld from an applicant. Supporters call this a necessary harmonisation of India’s transparency law with its privacy jurisprudence. Critics call it a structural tilt towards secrecy, one that weakens a core accountability tool used by citizens, journalists and civil society. The dispute is now before the Supreme Court, where three writ petitions challenge the amendment and the larger Data Protection framework around it.</p><p>This article walks through what changed, why it matters, what the petitioners argue, how the government’s side frames the matter, and what is at stake for accountability in public life.</p><h3>Part One: What Changed in Section 8(1)(j)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*zCOdu91RtDVJgcT1e64ovA.png" /></figure><h3>The old test</h3><p>Before the DPDP Act, Section 8(1)(j) of the RTI Act protected personal information, but only conditionally. Information could be withheld if:</p><ul><li>it had <strong>no relationship to any public activity or public interest</strong>, or</li><li>its disclosure would cause an <strong>unwarranted invasion of privacy</strong>.</li></ul><p>Even then, the exemption was not absolute. A Public Information Officer (PIO or CPIO) could still order disclosure if satisfied that a <strong>“larger public interest”</strong> justified it. In other words, the clause carried its own built-in balancing mechanism. The officer was expected to weigh the privacy interest on one side and the public interest on the other, and decide case by case.</p><h3>The new formulation</h3><p>Through <strong>Section 44(3)</strong>, the DPDP Act replaced this layered test with a much simpler formulation. Under the amended clause, “information which relates to personal information” is exempt.</p><p>The difference is stark. The earlier text asked several questions: Is there a public activity? Is there a public interest? Would the invasion of privacy be unwarranted? Does a larger public interest nonetheless justify disclosure? The new text asks essentially one: does the information relate to a person?</p><h3>The missing override</h3><p>Crucially, the <strong>internal public-interest override</strong> inside clause (j) was removed. The general public-interest provision in <strong>Section 8(2)</strong> of the RTI Act, which allows disclosure where the public interest outweighs the harm to a protected interest, remains on the statute book. But petitioners argue that, in practice, the removal of the override within clause (j) will push officials towards <strong>risk-averse denials</strong>. An officer who sees the words “personal information” in a record now has a simple, defensible reason to refuse, and little institutional incentive to engage in the harder task of balancing.</p><h3>Part Two: Why “Personal Information” Now Matters More</h3><p>The significance of the amendment is magnified by the breadth of the concept it relies on. The DPDP Act defines <strong>personal data</strong> widely, as any data about an individual who is identifiable by or in relation to such data (Section 2(t)).</p><p>The RTI Act itself never defined “personal data.” This gap means the DPDP Act’s expansive definition can be borrowed to decide what counts as “personal information” under the RTI regime. The consequence is that records containing <strong>names, designations, contact details or other identifiers</strong> can be brought within the exemption, even when those records relate entirely to public functions.</p><p>Consider how many routine government documents contain a name. A file noting who approved a contract, a list naming officials who attended an inspection, a notesheet recording who recommended a posting: each is, in a technical sense, “about an identifiable individual.” If the mere presence of an identifier is enough to invite the exemption, the amendment could reach far beyond intimate private details and into the ordinary machinery of public administration.</p><h3>Part Three: The Supreme Court Challenge</h3><h3>Three petitions</h3><p>Three writ petitions are pending before the Supreme Court, each challenging the effect of the DPDP Act on the RTI regime:</p><ol><li><strong><em>Venkatesh Nayak v. Union of India</em></strong> (W.P.© №177/2026)</li><li><strong><em>The Reporters Collective Trust &amp; Anr. v. Union of India</em></strong> (W.P.© №211/2026)</li><li><strong><em>National Campaign for People’s Right to Information v. Union of India</em></strong> (W.P.© №212/2026)</li></ol><p>The petitioners include a long-standing transparency researcher, a journalistic organisation and a national campaign body that was central to the movement that produced the RTI Act. Their range reflects how widely the concern is felt across those who use the law.</p><h3>What the Court has done so far</h3><p>On <strong>16 February 2026</strong>, a Bench issued notice to the Union of India and referred the matter to a larger Bench, describing the issues as “complex and constitutionally sensitive.” The referral signals that the Court sees the questions as weighty enough to require a broader set of judges.</p><p>Importantly, the Court has <strong>not stayed</strong> the DPDP Act or its rules while the case proceeds. The amended Section 8(1)(j) is therefore in force today, and information officers across the country are applying it while the constitutional challenge is pending.</p><h3>Part Four: The Petitioners’ Case</h3><p>The challenge is built around four interlinked concerns.</p><h3>1. A blanket exemption and the erosion of public-interest balancing</h3><p>The first argument is that Section 44(3) of the DPDP Act converts a conditional exemption into a broad one, and that this defeats the <strong>right to know</strong>, which flows from the freedom of speech and expression under <strong>Article 19(1)(a)</strong>.</p><p>Petitioners also contend that the amendment nullifies the <strong>proportionality-based, case-by-case test</strong> articulated by the Supreme Court in <em>CPIO v. Subhash Chandra Agarwal</em> (2019). That judgment recognised that privacy and transparency must be balanced, and that disclosure is permissible where a larger public interest exists. By stripping out the internal public-interest override, the amendment is said to remove the very mechanism through which that balance was struck.</p><h3>2. State exemptions and surveillance risks</h3><p>The challenge extends beyond the RTI amendment. Petitioners also target <strong>Sections 17(1)© and 17(2)</strong> of the DPDP Act, which grant the State wide exemptions from data-protection obligations for processing related to <strong>law enforcement, national security and public order</strong>.</p><p>The complaint is that these exemptions come without sufficient safeguards or independent oversight. Critics warn that this could facilitate a <strong>surveillance regime</strong> and enable arbitrary state access to personal data. The argument has a pointed irony: a law justified in the name of privacy is accused of leaving citizens’ privacy exposed to the State while restricting citizens’ access to information about the State.</p><h3>3. Violations of Articles 14, 19(1)(a) and 21</h3><p>The third strand is a direct constitutional attack. Petitioners say the amendments are <strong>arbitrary</strong>, violating the equality guarantee of Article 14; that they deny the <strong>right to know</strong> under Article 19(1)(a); and that they <strong>weaken privacy guarantees</strong> under Article 21 by enabling unchecked state processing while limiting what citizens can learn. The claim is that the law fails on both of the values it purports to serve.</p><h3>4. A chilling effect on journalism and accountability</h3><p>Investigative journalism often relies on records that carry personal identifiers: beneficiary lists, asset declarations, procurement files and appointment records. With broader exemptions in place, and with <strong>heavy penalties of up to ₹250 crore</strong> for non-compliance under the data-protection framework, media organisations and journalists fear a <strong>chilling effect</strong>. If handling or publishing such information carries the risk of severe liability, public-interest reporting may be suppressed and wrongdoing shielded.</p><h3>Part Five: Two Competing Narratives</h3><h3>The government’s position</h3><p>Supporters of the amendment present it as a matter of <strong>harmonisation</strong>. After <em>Puttaswamy</em> recognised privacy as a fundamental right, they argue, India’s older transparency law needed to be aligned with modern data-protection principles. On this view, the change is not about diluting transparency but about <strong>bringing disclosure norms into line</strong> with a constitutional right that did not exist in settled form when the RTI Act was passed in 2005.</p><h3>The critics’ position</h3><p>Critics describe the practical effect as a shift <strong>from disclosure to denial</strong>. They make two main points:</p><ul><li>Removing the internal public-interest override makes PIOs more <strong>risk-averse</strong>, especially in the shadow of data-protection liabilities. An officer who fears penalties for releasing personal data has every reason to refuse.</li><li>The overall regime becomes <strong>more restrictive and procedurally complex</strong>, undermining the spirit of the RTI Act, which was designed to empower ordinary citizens to monitor public institutions.</li></ul><p>The two narratives agree on one thing: privacy matters. They differ on whether the amendment protects it in a proportionate way or at an unacceptable cost to accountability.</p><h3>Part Six: The Puttaswamy Proportionality Test</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wExnk_TT2mTIP_wBlXnMjA.png" /></figure><p>Both sides ultimately invoke <em>Puttaswamy</em>, and the petitions lean heavily on its <strong>proportionality doctrine</strong>. Under that doctrine, a restriction on a fundamental right must satisfy four requirements:</p><ol><li>A <strong>legitimate state aim</strong>.</li><li>A <strong>rational nexus</strong> between the measure and that aim.</li><li>The use of the <strong>least restrictive means</strong>.</li><li>Adequate <strong>safeguards against abuse</strong>.</li></ol><p>Petitioners concede the first prong. Data protection is a legitimate aim. Their argument is that the amendment fails the remaining tests:</p><ul><li>A <strong>blanket exemption</strong> for personal information is more restrictive than necessary.</li><li><strong>Less restrictive alternatives already existed</strong>: the pre-amendment public-interest test within Section 8(1)(j) protected privacy while allowing disclosure where warranted.</li><li>The <strong>absence of clear standards and safeguards</strong> produces manifest arbitrariness and what petitioners call <strong>categorical opacity</strong>, which runs against <em>Puttaswamy</em>’s emphasis on <strong>contextual balancing</strong>.</li></ul><p>The heart of this argument is that privacy and transparency were previously reconciled through context-sensitive judgment, and that replacing judgment with a categorical rule is the constitutional flaw.</p><h3>Part Seven: Accountability at Stake</h3><p>Why does all this matter beyond the courtroom? Civil society and media groups point out that many of the most valuable accountability records inherently contain personal data:</p><ul><li><strong>Asset declarations</strong> of public officials.</li><li><strong>Procurement and contract files</strong>, including vendor details and signatories.</li><li><strong>Beneficiary lists</strong> for welfare schemes.</li><li><strong>Public appointments and postings</strong>.</li><li><strong>Disciplinary or misconduct records</strong>.</li></ul><p>These are the documents through which citizens detect <strong>conflicts of interest, corruption, favouritism and leakages</strong> in public expenditure. A beneficiary list, for example, is how a village can discover that a ration scheme is reaching ghost names. A procurement file is how a journalist can find that a contract went to a relative of an official. If such records are systematically treated as exempt because they are “personal,” the cases RTI was built to illuminate become much harder to uncover.</p><h3>Part Eight: Where the Law Stands</h3><p>Based on the sources reviewed for this article, the position is as follows:</p><ul><li>The DPDP Act’s amendment to <strong>Section 8(1)(j)</strong> is <strong>in force</strong>.</li><li>The Supreme Court has <strong>issued notice</strong> and referred the constitutional questions to a <strong>larger Bench</strong>, but has <strong>not granted a stay</strong>.</li><li>The <strong>Section 8(2)</strong> public-interest override in the RTI Act remains <strong>unchanged</strong>, though whether it can effectively counter denials under the amended Section 8(1)(j) is itself part of the live controversy.</li></ul><p>Since the matter is pending, developments after the dates of those sources may have occurred, and readers should check the Supreme Court’s records for the latest status.</p><pre>#!/usr/bin/env python3<br>&quot;&quot;&quot;<br>Install:   pip install torch &quot;keras&gt;=3&quot; numpy boto3 sagemaker   (boto3/sagemaker only for --live AWS)<br>Examples:<br>    python rti_dpdp_engine.py --mode train --epochs 8 --model-dir ./model<br>    python rti_dpdp_engine.py --mode analyze --model-dir ./model \<br>        --text &quot;Copy of asset declaration filed by Shri Rakesh Sharma PAN ABCDE1234F&quot;<br>    python rti_dpdp_engine.py --mode impact --model-dir ./model<br>    python rti_dpdp_engine.py --mode aws-train --role-arn arn:aws:iam::123456789012:role/SageMakerRole<br>&quot;&quot;&quot;<br>from __future__ import annotations<br><br>import argparse<br>import json<br>import logging<br>import os<br>import random<br>import re<br>import time<br>import uuid<br>from collections import Counter<br>from dataclasses import asdict, dataclass, field<br>from decimal import Decimal<br>from pathlib import Path<br>from typing import Any, Dict, List, Optional, Tuple<br><br>os.environ.setdefault(&quot;KERAS_BACKEND&quot;, &quot;torch&quot;)  # Keras 3 can run on the PyTorch backend<br><br>import numpy as np<br>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>from torch.utils.data import DataLoader, TensorDataset<br><br>import keras<br>from keras import layers<br><br>logging.basicConfig(level=logging.INFO, format=&quot;%(asctime)s %(levelname)s %(message)s&quot;)<br>log = logging.getLogger(&quot;rti-dpdp&quot;)<br><br>DECISIONS = [&quot;DISCLOSE&quot;, &quot;PARTIAL_REDACT&quot;, &quot;WITHHOLD&quot;]<br>CATEGORIES = [&quot;asset_declaration&quot;, &quot;procurement&quot;, &quot;beneficiary_list&quot;,<br>              &quot;appointment&quot;, &quot;disciplinary&quot;, &quot;private_life&quot;]<br>PAD, UNK = 0, 1<br><br><br># --------------------------------------------------------------------------- config<br>@dataclass<br>class Config:<br>    max_len: int = 48<br>    emb_dim: int = 96<br>    hidden: int = 96<br>    batch_size: int = 64<br>    epochs: int = 8<br>    lr: float = 2e-3<br>    n_train: int = 6000<br>    seed: int = 42<br>    public_interest_threshold: float = 0.70   # pre-DPDP &quot;larger public interest&quot;<br>    s82_threshold: float = 0.90               # post-DPDP s.8(2) override (very high bar in practice)<br>    loss_pi_weight: float = 1.5<br><br><br>def set_seed(seed: int) -&gt; None:<br>    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)<br><br><br># --------------------------------------------------------------------------- PII detection<br>PII_PATTERNS = {<br>    &quot;AADHAAR&quot;: re.compile(r&quot;\b\d{4}\s?\d{4}\s?\d{4}\b&quot;),<br>    &quot;PAN&quot;: re.compile(r&quot;\b[A-Z]{5}\d{4}[A-Z]\b&quot;),<br>    &quot;EMAIL&quot;: re.compile(r&quot;\b[\w.+-]+@[\w-]+\.[\w.]+\b&quot;),<br>    &quot;PHONE&quot;: re.compile(r&quot;(?:\+91[\-\s]?)?\b[6-9]\d{9}\b&quot;),<br>}<br>NAME_RE = re.compile(r&quot;\b(?:Shri|Smt|Dr|Mr|Ms|Mrs)\.?\s+[A-Z][a-z]+(?:\s+[A-Z][a-z]+)?&quot;)<br><br><br>def mask_pii(text: str) -&gt; Tuple[str, List[Dict[str, str]]]:<br>    &quot;&quot;&quot;Replace identifiers with tags (so models never see raw IDs) and report what was found.&quot;&quot;&quot;<br>    found: List[Dict[str, str]] = []<br>    for label, rx in PII_PATTERNS.items():<br>        for m in rx.finditer(text):<br>            found.append({&quot;type&quot;: label, &quot;source&quot;: &quot;regex&quot;, &quot;span&quot;: m.group(0)})<br>        text = rx.sub(f&quot; &lt;{label.lower()}&gt; &quot;, text)<br>    for m in NAME_RE.finditer(text):<br>        found.append({&quot;type&quot;: &quot;NAME&quot;, &quot;source&quot;: &quot;regex&quot;, &quot;span&quot;: m.group(0)})<br>    text = NAME_RE.sub(&quot; &lt;name&gt; &quot;, text)<br>    return re.sub(r&quot;\s+&quot;, &quot; &quot;, text).strip(), found<br><br><br># --------------------------------------------------------------------------- vocabulary<br>class Vocab:<br>    TOKEN_RE = re.compile(r&quot;&lt;[a-z]+&gt;|[a-z0-9_]+&quot;)<br><br>    def __init__(self, itos: Optional[List[str]] = None):<br>        self.itos = itos or [&quot;&lt;pad&gt;&quot;, &quot;&lt;unk&gt;&quot;]<br>        self.stoi = {t: i for i, t in enumerate(self.itos)}<br><br>    def tokenize(self, text: str) -&gt; List[str]:<br>        return self.TOKEN_RE.findall(text.lower())<br><br>    def build(self, texts: List[str], max_size: int = 20000, min_freq: int = 1) -&gt; &quot;Vocab&quot;:<br>        counts = Counter(t for x in texts for t in self.tokenize(x))<br>        for tok, c in counts.most_common(max_size):<br>            if c &gt;= min_freq and tok not in self.stoi:<br>                self.stoi[tok] = len(self.itos); self.itos.append(tok)<br>        return self<br><br>    def encode(self, text: str, max_len: int) -&gt; List[int]:<br>        ids = [self.stoi.get(t, UNK) for t in self.tokenize(text)][:max_len]<br>        return ids + [PAD] * (max_len - len(ids))<br><br>    def __len__(self) -&gt; int:<br>        return len(self.itos)<br><br><br># --------------------------------------------------------------------------- synthetic data<br>NAMES = [&quot;Rakesh Sharma&quot;, &quot;Anita Rao&quot;, &quot;Imran Khan&quot;, &quot;Lakshmi Nair&quot;, &quot;Suresh Patil&quot;, &quot;Meera Iyer&quot;]<br>ROLES = [&quot;Joint Secretary&quot;, &quot;District Collector&quot;, &quot;Block Officer&quot;, &quot;Chief Engineer&quot;, &quot;Tehsildar&quot;]<br>SCHEMES = [&quot;ration card&quot;, &quot;pension&quot;, &quot;housing&quot;, &quot;scholarship&quot;, &quot;LPG subsidy&quot;]<br>TEMPLATES = {<br>    &quot;asset_declaration&quot;: [&quot;copy of annual asset declaration filed by {t} {n}&quot;,<br>                          &quot;immovable property return of {r} {t} {n}&quot;],<br>    &quot;procurement&quot;: [&quot;tender file and vendor signatories for road contract approved by {r}&quot;,<br>                    &quot;procurement records and contract awardee details signed by {t} {n}&quot;],<br>    &quot;beneficiary_list&quot;: [&quot;list of {s} beneficiaries with names and addresses in the district&quot;,<br>                         &quot;beneficiary names for {s} scheme sanctioned by {r}&quot;],<br>    &quot;appointment&quot;: [&quot;appointment and posting order of {t} {n} as {r}&quot;,<br>                    &quot;transfer and promotion records for the post of {r}&quot;],<br>    &quot;disciplinary&quot;: [&quot;disciplinary inquiry report against {t} {n}&quot;,<br>                     &quot;misconduct complaint and action taken on {r}&quot;],<br>    &quot;private_life&quot;: [&quot;medical records and family details of {t} {n}&quot;,<br>                     &quot;personal bank statement and marital status of {t} {n}&quot;],<br>}<br>PI_BASE = {&quot;asset_declaration&quot;: .85, &quot;procurement&quot;: .92, &quot;beneficiary_list&quot;: .78,<br>           &quot;appointment&quot;: .72, &quot;disciplinary&quot;: .60, &quot;private_life&quot;: .08}<br>PII_SNIPPETS = [&quot;PAN ABCDE1234F&quot;, &quot;mobile 9876543210&quot;, &quot;email officer@gov.in&quot;, &quot;aadhaar 1234 5678 9012&quot;]<br><br><br>def gen_dataset(n: int, seed: int):<br>    rng = random.Random(seed)<br>    texts, y_cls, y_pi, y_cat = [], [], [], []<br>    for _ in range(n):<br>        cats = rng.sample(CATEGORIES, k=rng.choice([1, 1, 2]))<br>        parts = [rng.choice(TEMPLATES[c]).format(t=rng.choice([&quot;Shri&quot;, &quot;Smt&quot;, &quot;Dr&quot;]), n=rng.choice(NAMES),<br>                                                 r=rng.choice(ROLES), s=rng.choice(SCHEMES)) for c in cats]<br>        if rng.random() &lt; 0.35:<br>            parts.append(rng.choice(PII_SNIPPETS))<br>        texts.append(&quot; and &quot;.join(parts))<br>        pi = float(np.clip(np.mean([PI_BASE[c] for c in cats]) + rng.gauss(0, 0.05), 0, 1))<br>        if &quot;private_life&quot; in cats and len(cats) == 1:<br>            pi = min(pi, 0.2)<br>        label = 0 if pi &gt;= 0.80 else (1 if pi &gt;= 0.50 else 2)<br>        y_cls.append(label); y_pi.append(pi)<br>        y_cat.append([1.0 if c in cats else 0.0 for c in CATEGORIES])<br>    return texts, np.array(y_cls), np.array(y_pi, dtype=&quot;float32&quot;), np.array(y_cat, dtype=&quot;float32&quot;)<br><br><br># --------------------------------------------------------------------------- PyTorch model<br>class AttnBiLSTM(nn.Module):<br>    &quot;&quot;&quot;Multi-task: 3-way disclosure decision + public-interest regression, with token attention.&quot;&quot;&quot;<br>    def __init__(self, vocab_size: int, emb: int, hidden: int, n_cls: int = 3, dropout: float = 0.3):<br>        super().__init__()<br>        self.emb = nn.Embedding(vocab_size, emb, padding_idx=PAD)<br>        self.lstm = nn.LSTM(emb, hidden, batch_first=True, bidirectional=True)<br>        self.attn = nn.Linear(2 * hidden, 1)<br>        self.drop = nn.Dropout(dropout)<br>        self.cls = nn.Linear(2 * hidden, n_cls)<br>        self.pi = nn.Linear(2 * hidden, 1)<br><br>    def forward(self, x: torch.Tensor):<br>        mask = x.ne(PAD)<br>        h, _ = self.lstm(self.drop(self.emb(x)))<br>        scores = self.attn(h).squeeze(-1).masked_fill(~mask, -1e9)<br>        w = F.softmax(scores, dim=-1)<br>        ctx = self.drop((h * w.unsqueeze(-1)).sum(1))<br>        return self.cls(ctx), torch.sigmoid(self.pi(ctx)).squeeze(-1), w<br><br><br>def train_torch(cfg: Config, X: np.ndarray, y_cls: np.ndarray, y_pi: np.ndarray, vocab_size: int):<br>    device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>    use_amp = device == &quot;cuda&quot;<br>    n_val = int(0.15 * len(X))<br>    tt = lambda a, dt: torch.tensor(a, dtype=dt)<br>    tr = TensorDataset(tt(X[n_val:], torch.long), tt(y_cls[n_val:], torch.long), tt(y_pi[n_val:], torch.float32))<br>    va = TensorDataset(tt(X[:n_val], torch.long), tt(y_cls[:n_val], torch.long), tt(y_pi[:n_val], torch.float32))<br>    tl, vl = DataLoader(tr, cfg.batch_size, shuffle=True), DataLoader(va, cfg.batch_size)<br><br>    model = AttnBiLSTM(vocab_size, cfg.emb_dim, cfg.hidden).to(device)<br>    opt = torch.optim.AdamW(model.parameters(), lr=cfg.lr, weight_decay=1e-4)<br>    sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=cfg.lr, total_steps=cfg.epochs * len(tl))<br>    scaler = torch.amp.GradScaler(&quot;cuda&quot;, enabled=use_amp)<br>    best, best_state = 0.0, None<br><br>    for ep in range(1, cfg.epochs + 1):<br>        model.train(); total = 0.0<br>        for xb, yc, yp in tl:<br>            xb, yc, yp = xb.to(device), yc.to(device), yp.to(device)<br>            opt.zero_grad(set_to_none=True)<br>            with torch.autocast(device_type=device, enabled=use_amp):<br>                logits, pi, _ = model(xb)<br>                loss = F.cross_entropy(logits, yc) + cfg.loss_pi_weight * F.mse_loss(pi.float(), yp)<br>            scaler.scale(loss).backward()<br>            scaler.unscale_(opt); nn.utils.clip_grad_norm_(model.parameters(), 1.0)<br>            scaler.step(opt); scaler.update(); sched.step()<br>            total += loss.item()<br>        model.eval(); correct = n = 0; mae = 0.0<br>        with torch.no_grad():<br>            for xb, yc, yp in vl:<br>                logits, pi, _ = model(xb.to(device))<br>                correct += (logits.argmax(-1).cpu() == yc).sum().item(); n += len(yc)<br>                mae += (pi.cpu() - yp).abs().sum().item()<br>        acc = correct / n<br>        log.info(&quot;torch epoch %d/%d loss=%.4f val_acc=%.3f val_pi_mae=%.3f&quot;, ep, cfg.epochs, total / len(tl), acc, mae / n)<br>        if acc &gt;= best:<br>            best, best_state = acc, {k: v.detach().cpu().clone() for k, v in model.state_dict().items()}<br>    model.load_state_dict(best_state)<br>    return model.cpu().eval(), best<br><br><br># --------------------------------------------------------------------------- Keras model<br>def build_keras(vocab_size: int, max_len: int, n_labels: int) -&gt; keras.Model:<br>    inp = keras.Input(shape=(max_len,), dtype=&quot;int32&quot;)<br>    x = layers.SpatialDropout1D(0.2)(layers.Embedding(vocab_size, 64)(inp))<br>    pools = [layers.GlobalMaxPooling1D()(layers.Conv1D(64, k, padding=&quot;same&quot;, activation=&quot;relu&quot;)(x)) for k in (2, 3, 4)]<br>    x = layers.Dropout(0.3)(layers.Dense(64, activation=&quot;relu&quot;)(layers.Concatenate()(pools)))<br>    out = layers.Dense(n_labels, activation=&quot;sigmoid&quot;, name=&quot;categories&quot;)(x)<br>    m = keras.Model(inp, out, name=&quot;record_category_cnn&quot;)<br>    m.compile(optimizer=keras.optimizers.Adam(1e-3), loss=&quot;binary_crossentropy&quot;,<br>              metrics=[keras.metrics.AUC(name=&quot;auc&quot;), keras.metrics.BinaryAccuracy(name=&quot;acc&quot;)])<br>    return m<br><br><br># --------------------------------------------------------------------------- AWS gateway<br>class AWSGateway:<br>    &quot;&quot;&quot;Thin boto3 wrapper. dry_run=True prints what *would* happen, so the file runs without credentials.&quot;&quot;&quot;<br>    def __init__(self, region=&quot;ap-south-1&quot;, bucket=&quot;rti-dpdp-artifacts&quot;, table=&quot;rti_dpdp_audit&quot;,<br>                 queue_url: str = &quot;&quot;, dry_run: bool = True):<br>        self.region, self.bucket, self.table, self.queue_url = region, bucket, table, queue_url<br>        self.dry_run = dry_run<br>        self._c: Dict[str, Any] = {}<br>        if not dry_run:<br>            import boto3  # lazy: only needed for live mode<br>            self._boto3 = boto3<br><br>    def _client(self, name: str):<br>        if name not in self._c:<br>            self._c[name] = self._boto3.client(name, region_name=self.region)<br>        return self._c[name]<br><br>    def _dry(self, action: str, **kw) -&gt; None:<br>        log.info(&quot;[AWS dry-run] %s %s&quot;, action, json.dumps(kw, default=str)[:300])<br><br>    def upload_dir(self, local: Path, prefix: str) -&gt; str:<br>        uri = f&quot;s3://{self.bucket}/{prefix}&quot;<br>        if self.dry_run:<br>            self._dry(&quot;s3.upload&quot;, src=str(local), dst=uri); return uri<br>        s3 = self._client(&quot;s3&quot;)<br>        for p in local.rglob(&quot;*&quot;):<br>            if p.is_file():<br>                s3.upload_file(str(p), self.bucket, f&quot;{prefix}/{p.relative_to(local)}&quot;,<br>                               ExtraArgs={&quot;ServerSideEncryption&quot;: &quot;aws:kms&quot;})<br>        return uri<br><br>    def download_dir(self, prefix: str, local: Path) -&gt; None:<br>        if self.dry_run:<br>            self._dry(&quot;s3.download&quot;, src=prefix, dst=str(local)); return<br>        s3 = self._client(&quot;s3&quot;)<br>        for page in s3.get_paginator(&quot;list_objects_v2&quot;).paginate(Bucket=self.bucket, Prefix=prefix):<br>            for o in page.get(&quot;Contents&quot;, []):<br>                dst = local / Path(o[&quot;Key&quot;]).relative_to(prefix); dst.parent.mkdir(parents=True, exist_ok=True)<br>                s3.download_file(self.bucket, o[&quot;Key&quot;], str(dst))<br><br>    def comprehend_pii(self, text: str) -&gt; List[Dict[str, str]]:<br>        if self.dry_run:<br>            self._dry(&quot;comprehend.detect_pii_entities&quot;, chars=len(text)); return []<br>        res = self._client(&quot;comprehend&quot;).detect_pii_entities(Text=text[:4500], LanguageCode=&quot;en&quot;)<br>        return [{&quot;type&quot;: e[&quot;Type&quot;], &quot;source&quot;: &quot;comprehend&quot;,<br>                 &quot;span&quot;: text[e[&quot;BeginOffset&quot;]:e[&quot;EndOffset&quot;]]} for e in res[&quot;Entities&quot;]]<br><br>    def audit(self, item: Dict[str, Any]) -&gt; None:<br>        &quot;&quot;&quot;Immutable-style audit trail: every decision is recorded (a &#39;safeguard&#39; in the proportionality sense).&quot;&quot;&quot;<br>        item = json.loads(json.dumps(item, default=str), parse_float=Decimal)<br>        if self.dry_run:<br>            self._dry(&quot;dynamodb.put_item&quot;, table=self.table, id=item.get(&quot;request_id&quot;)); return<br>        self._boto3.resource(&quot;dynamodb&quot;, region_name=self.region).Table(self.table).put_item(Item=item)<br><br>    def metric(self, name: str, value: float, unit: str = &quot;Count&quot;) -&gt; None:<br>        if self.dry_run:<br>            self._dry(&quot;cloudwatch.put_metric_data&quot;, name=name, value=value); return<br>        self._client(&quot;cloudwatch&quot;).put_metric_data(Namespace=&quot;RTI-DPDP&quot;, MetricData=[<br>            {&quot;MetricName&quot;: name, &quot;Value&quot;: value, &quot;Unit&quot;: unit}])<br><br>    def enqueue(self, text: str) -&gt; None:<br>        body = json.dumps({&quot;request_id&quot;: str(uuid.uuid4()), &quot;text&quot;: text})<br>        if self.dry_run:<br>            self._dry(&quot;sqs.send_message&quot;, body=body); return<br>        self._client(&quot;sqs&quot;).send_message(QueueUrl=self.queue_url, MessageBody=body)<br><br>    def sagemaker_train(self, role_arn: str, epochs: int, instance=&quot;ml.g4dn.xlarge&quot;) -&gt; None:<br>        if self.dry_run:<br>            self._dry(&quot;sagemaker.PyTorch.fit&quot;, role=role_arn, instance=instance, spot=True, epochs=epochs); return<br>        from sagemaker.pytorch import PyTorch<br>        src = Path(__file__).resolve()<br>        est = PyTorch(entry_point=src.name, source_dir=str(src.parent), role=role_arn,<br>                      framework_version=&quot;2.3&quot;, py_version=&quot;py311&quot;, instance_count=1, instance_type=instance,<br>                      hyperparameters={&quot;mode&quot;: &quot;train&quot;, &quot;epochs&quot;: epochs}, use_spot_instances=True,<br>                      max_run=3600, max_wait=7200, output_path=f&quot;s3://{self.bucket}/models/&quot;)<br>        est.fit(wait=True)  # requirements.txt next to this file must list keras&gt;=3<br>        log.info(&quot;model artifact: %s&quot;, est.model_data)<br><br>    def sagemaker_deploy(self, role_arn: str, model_s3_uri: str, instance=&quot;ml.m5.large&quot;) -&gt; None:<br>        if self.dry_run:<br>            self._dry(&quot;sagemaker.PyTorchModel.deploy&quot;, model=model_s3_uri, instance=instance); return<br>        from sagemaker.pytorch import PyTorchModel<br>        src = Path(__file__).resolve()<br>        PyTorchModel(model_data=model_s3_uri, role=role_arn, entry_point=src.name, source_dir=str(src.parent),<br>                     framework_version=&quot;2.3&quot;, py_version=&quot;py311&quot;).deploy(<br>            initial_instance_count=1, instance_type=instance)<br><br><br># --------------------------------------------------------------------------- pipeline<br>@dataclass<br>class Analysis:<br>    request_id: str<br>    masked_text: str<br>    pii: List[Dict[str, str]]<br>    categories: Dict[str, float]<br>    torch_probs: Dict[str, float]<br>    public_interest: float<br>    identifiable: bool<br>    pre_dpdp: str<br>    post_dpdp: str<br>    drift: bool<br>    proportionality: Dict[str, Any]<br>    top_tokens: List[Tuple[str, float]] = field(default_factory=list)<br><br><br>class RTIPipeline:<br>    def __init__(self, cfg: Config, vocab: Vocab, tmodel: AttnBiLSTM, kmodel: keras.Model,<br>                 aws: Optional[AWSGateway] = None):<br>        self.cfg, self.vocab, self.tm, self.km, self.aws = cfg, vocab, tmodel, kmodel, aws<br><br>    # ---- policy engines (the heart of the article) ----<br>    def decide_pre(self, probs: np.ndarray, pi: float) -&gt; str:<br>        &quot;&quot;&quot;Old s.8(1)(j): privacy exemption BUT &#39;larger public interest&#39; can justify disclosure.&quot;&quot;&quot;<br>        d = int(probs.argmax())<br>        if d == 2 and pi &gt;= self.cfg.public_interest_threshold:<br>            d = 1<br>        return DECISIONS[d]<br><br>    def decide_post(self, identifiable: bool, pi: float, invoke_82: bool = True) -&gt; str:<br>        &quot;&quot;&quot;Amended s.8(1)(j): anything that &#39;relates to personal information&#39; is exempt.<br>        Only the general s.8(2) override (a very high bar) can still open the record - and only partially here.&quot;&quot;&quot;<br>        if not identifiable:<br>            return &quot;DISCLOSE&quot;<br>        return &quot;PARTIAL_REDACT&quot; if (invoke_82 and pi &gt;= self.cfg.s82_threshold) else &quot;WITHHOLD&quot;<br><br>    def proportionality(self, identifiable: bool, pre: str, post: str, pi: float) -&gt; Dict[str, Any]:<br>        &quot;&quot;&quot;Heuristic mapping of the four Puttaswamy prongs onto this record.&quot;&quot;&quot;<br>        prongs = {<br>            &quot;1_legitimate_aim&quot;: True,  # data protection is accepted as legitimate by all sides<br>            &quot;2_rational_nexus&quot;: identifiable,  # withholding only makes sense if personal data is present<br>            &quot;3_least_restrictive_means&quot;: not (post == &quot;WITHHOLD&quot; and pre != &quot;WITHHOLD&quot;),<br>            &quot;4_safeguards&quot;: pi &gt;= self.cfg.s82_threshold or pre == post,<br>        }<br>        failed = [k for k, v in prongs.items() if not v]<br>        return {&quot;prongs&quot;: prongs, &quot;passes&quot;: not failed, &quot;failed&quot;: failed,<br>                &quot;note&quot;: &quot;Post-DPDP blanket denial where redaction would suffice fails prong 3.&quot; if &quot;3_least_restrictive_means&quot; in failed else &quot;&quot;}<br><br>    # ---- inference ----<br>    def analyze(self, text: str) -&gt; Analysis:<br>        masked, pii = mask_pii(text)<br>        if self.aws is not None:<br>            pii += self.aws.comprehend_pii(text)<br>        ids = self.vocab.encode(masked, self.cfg.max_len)<br>        with torch.no_grad():<br>            logits, pi, w = self.tm(torch.tensor([ids]))<br>        probs = F.softmax(logits, -1)[0].numpy(); pi = float(pi[0])<br>        cat = self.km.predict(np.array([ids], dtype=&quot;int32&quot;), verbose=0)[0]<br>        cats = {c: float(p) for c, p in zip(CATEGORIES, cat)}<br>        identifiable = bool(pii) or any(p &gt;= 0.5 for p in cat)<br>        pre, post = self.decide_pre(probs, pi), self.decide_post(identifiable, pi)<br>        toks = self.vocab.tokenize(masked)[: self.cfg.max_len]<br>        top = sorted(zip(toks, w[0][: len(toks)].tolist()), key=lambda t: -t[1])[:5]<br>        a = Analysis(str(uuid.uuid4()), masked, pii, cats, dict(zip(DECISIONS, map(float, probs))), pi,<br>                     identifiable, pre, post, pre != post, self.proportionality(identifiable, pre, post, pi), top)<br>        if self.aws is not None:<br>            self.aws.audit(asdict(a))<br>            self.aws.metric(&quot;DenialDrift&quot;, 1.0 if a.drift else 0.0)<br>        return a<br><br>    def impact_analysis(self, n: int = 500, seed: int = 7) -&gt; Dict[str, Any]:<br>        &quot;&quot;&quot;Quantify the critics&#39; claim: how many requests flip from disclosure to denial after the amendment?&quot;&quot;&quot;<br>        texts, *_ = gen_dataset(n, seed)<br>        res = [self.analyze(t) for t in texts]<br>        pre = Counter(r.pre_dpdp for r in res); post = Counter(r.post_dpdp for r in res)<br>        deny = lambda c: c[&quot;WITHHOLD&quot;] / n<br>        return {&quot;n&quot;: n, &quot;pre_dpdp&quot;: dict(pre), &quot;post_dpdp&quot;: dict(post),<br>                &quot;denial_rate_pre&quot;: round(deny(pre), 3), &quot;denial_rate_post&quot;: round(deny(post), 3),<br>                &quot;flipped_to_denial&quot;: sum(1 for r in res if r.pre_dpdp != &quot;WITHHOLD&quot; and r.post_dpdp == &quot;WITHHOLD&quot;),<br>                &quot;fail_proportionality&quot;: sum(1 for r in res if not r.proportionality[&quot;passes&quot;])}<br><br>    # ---- persistence ----<br>    def save(self, d: str) -&gt; None:<br>        p = Path(d); p.mkdir(parents=True, exist_ok=True)<br>        torch.save(self.tm.state_dict(), p / &quot;decision_model.pt&quot;)<br>        self.km.save(p / &quot;category_model.keras&quot;)<br>        (p / &quot;vocab.json&quot;).write_text(json.dumps(self.vocab.itos))<br>        (p / &quot;config.json&quot;).write_text(json.dumps(asdict(self.cfg)))<br><br>    @classmethod<br>    def load(cls, d: str, aws: Optional[AWSGateway] = None) -&gt; &quot;RTIPipeline&quot;:<br>        p = Path(d); cfg = Config(**json.loads((p / &quot;config.json&quot;).read_text()))<br>        vocab = Vocab(json.loads((p / &quot;vocab.json&quot;).read_text()))<br>        tm = AttnBiLSTM(len(vocab), cfg.emb_dim, cfg.hidden)<br>        tm.load_state_dict(torch.load(p / &quot;decision_model.pt&quot;, map_location=&quot;cpu&quot;)); tm.eval()<br>        return cls(cfg, vocab, tm, keras.saving.load_model(p / &quot;category_model.keras&quot;), aws)<br><br><br>def train_all(cfg: Config, model_dir: str, aws: Optional[AWSGateway] = None) -&gt; RTIPipeline:<br>    set_seed(cfg.seed)<br>    texts, y_cls, y_pi, y_cat = gen_dataset(cfg.n_train, cfg.seed)<br>    masked = [mask_pii(t)[0] for t in texts]<br>    vocab = Vocab().build(masked)<br>    X = np.array([vocab.encode(t, cfg.max_len) for t in masked], dtype=&quot;int64&quot;)<br>    tm, acc = train_torch(cfg, X, y_cls, y_pi, len(vocab))<br>    km = build_keras(len(vocab), cfg.max_len, len(CATEGORIES))<br>    hist = km.fit(X.astype(&quot;int32&quot;), y_cat, validation_split=0.15, epochs=cfg.epochs, batch_size=cfg.batch_size,<br>                  verbose=0, callbacks=[keras.callbacks.EarlyStopping(patience=2, restore_best_weights=True)])<br>    log.info(&quot;keras val_auc=%.3f | torch val_acc=%.3f&quot;, max(hist.history[&quot;val_auc&quot;]), acc)<br>    pipe = RTIPipeline(cfg, vocab, tm, km, aws)<br>    pipe.save(model_dir)<br>    if aws is not None:<br>        aws.upload_dir(Path(model_dir), f&quot;models/{int(time.time())}&quot;)<br>    return pipe<br><br><br># --------------------------------------------------------------------------- AWS Lambda (SQS trigger)<br>_PIPE: Optional[RTIPipeline] = None<br><br><br>def lambda_handler(event, context=None):<br>    &quot;&quot;&quot;SQS -&gt; Lambda. Env: MODEL_S3_PREFIX, ARTIFACT_BUCKET, AUDIT_TABLE. Models cached across warm invocations.&quot;&quot;&quot;<br>    global _PIPE<br>    if _PIPE is None:<br>        aws = AWSGateway(bucket=os.environ[&quot;ARTIFACT_BUCKET&quot;], table=os.environ.get(&quot;AUDIT_TABLE&quot;, &quot;rti_dpdp_audit&quot;),<br>                         dry_run=False)<br>        aws.download_dir(os.environ[&quot;MODEL_S3_PREFIX&quot;], Path(&quot;/tmp/model&quot;))<br>        _PIPE = RTIPipeline.load(&quot;/tmp/model&quot;, aws)<br>    out = []<br>    for rec in event.get(&quot;Records&quot;, []):<br>        a = _PIPE.analyze(json.loads(rec[&quot;body&quot;])[&quot;text&quot;])<br>        out.append({&quot;request_id&quot;: a.request_id, &quot;pre&quot;: a.pre_dpdp, &quot;post&quot;: a.post_dpdp, &quot;drift&quot;: a.drift})<br>    return {&quot;statusCode&quot;: 200, &quot;body&quot;: json.dumps(out)}<br><br><br># --------------------------------------------------------------------------- SageMaker inference contract<br>def model_fn(model_dir):           return RTIPipeline.load(model_dir)<br>def input_fn(body, content_type):  return json.loads(body)[&quot;text&quot;]<br>def predict_fn(text, model):       return asdict(model.analyze(text))<br>def output_fn(pred, accept):       return json.dumps(pred, default=str), &quot;application/json&quot;<br><br><br># --------------------------------------------------------------------------- CLI<br>def main() -&gt; None:<br>    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)<br>    ap.add_argument(&quot;--mode&quot;, default=&quot;train&quot;,<br>                    choices=[&quot;train&quot;, &quot;analyze&quot;, &quot;impact&quot;, &quot;enqueue&quot;, &quot;aws-train&quot;, &quot;aws-deploy&quot;])<br>    ap.add_argument(&quot;--model-dir&quot;, default=os.environ.get(&quot;SM_MODEL_DIR&quot;, &quot;./model&quot;))<br>    ap.add_argument(&quot;--epochs&quot;, type=int, default=8)<br>    ap.add_argument(&quot;--text&quot;, default=&quot;&quot;)<br>    ap.add_argument(&quot;--role-arn&quot;, default=&quot;&quot;)<br>    ap.add_argument(&quot;--model-s3-uri&quot;, default=&quot;&quot;)<br>    ap.add_argument(&quot;--live&quot;, action=&quot;store_true&quot;, help=&quot;make real AWS calls (default: dry-run)&quot;)<br>    ap.add_argument(&quot;--region&quot;, default=&quot;ap-south-1&quot;)<br>    ap.add_argument(&quot;--bucket&quot;, default=&quot;rti-dpdp-artifacts&quot;)<br>    ap.add_argument(&quot;--queue-url&quot;, default=&quot;&quot;)<br>    args, _ = ap.parse_known_args()   # SageMaker may pass extra args<br><br>    aws = AWSGateway(args.region, args.bucket, queue_url=args.queue_url, dry_run=not args.live)<br>    cfg = Config(epochs=args.epochs)<br><br>    if args.mode == &quot;train&quot;:<br>        train_all(cfg, args.model_dir, aws if args.live else None)<br>    elif args.mode == &quot;analyze&quot;:<br>        pipe = RTIPipeline.load(args.model_dir, aws)<br>        print(json.dumps(asdict(pipe.analyze(args.text)), indent=2, default=str))<br>    elif args.mode == &quot;impact&quot;:<br>        print(json.dumps(RTIPipeline.load(args.model_dir).impact_analysis(), indent=2))<br>    elif args.mode == &quot;enqueue&quot;:<br>        aws.enqueue(args.text)<br>    elif args.mode == &quot;aws-train&quot;:<br>        aws.sagemaker_train(args.role_arn, args.epochs)<br>    elif args.mode == &quot;aws-deploy&quot;:<br>        aws.sagemaker_deploy(args.role_arn, args.model_s3_uri)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h3>Conclusion: Recalibrating Rights Without Losing Accountability</h3><p>The DPDP-RTI conflict is a defining moment for Indian democracy. Privacy is a constitutionally mandated goal, and few would argue that citizens’ personal data should be freely available. But transparency is the engine of accountability, and the RTI Act has been one of the most widely used tools for holding power to account.</p><p>The question before the Court is not whether to choose privacy over transparency or the reverse. It is whether the current amendment strikes a <strong>proportionate balance</strong> or creates a <strong>structural bias towards secrecy</strong>. A constitutionally sound resolution is likely to require one of two things: either restoring a <strong>tailored public-interest test</strong> within the exemption framework, or embedding <strong>clear safeguards, standards and oversight</strong> so that data protection does not become a shield against legitimate scrutiny.</p><p>How the Supreme Court answers will shape not only the fate of one clause, but the kind of relationship Indian citizens will have with their government in the digital age.</p><blockquote><em>💻 </em><strong><em>HIRING: Tech Talent — Multiple Roles</em></strong></blockquote><blockquote>💰 Earn $50–$150/hour<br>🚀 Developers, AI Engineers, Data Scientists, QA Testers &amp; More!</blockquote><blockquote><a href="https://refer.micro1.ai/referral/jobs?referralCode=4d8f644a-04a3-4fac-82e4-c4c1626a9009&amp;utm_source=referral&amp;utm_medium=share&amp;utm_campaign=job_referral">👉 <strong>APPLY NOW</strong></a></blockquote><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc034d77abdf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/privacy-vs-transparency-the-dpdp-act-the-rti-act-and-indias-constitutional-balance-bc034d77abdf">Privacy vs. Transparency: The DPDP Act, the RTI Act, and India’s Constitutional Balance</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Local AI Stack: A Developer’s Guide to Ollama, LM Studio, llama.cpp,]]></title>
            <link>https://medium.com/codetodeploy/navigating-the-local-ai-stack-a-developers-guide-to-ollama-lm-studio-llama-cpp-392796637d2e?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/392796637d2e</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[lm-studio]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[onnx]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 11:24:58 GMT</pubDate>
            <atom:updated>2026-08-27T14:19:15.262Z</atom:updated>
            <content:encoded><![CDATA[<h3>Navigating the Local AI Stack: A Developer’s Guide to Ollama, LM Studio, llama.cpp, Hugging Face, and ONNX</h3><h4>Introduction:</h4><p>The local AI landscape has evolved from an experimental hobbyist ecosystem into a production-ready alternative to cloud-hosted APIs. Running large language models (LLMs) locally eliminates API latency, drastically cuts cloud inference costs, guarantees complete data privacy, and enables offline capability.</p><blockquote><em>💥 </em><strong><em>Master AI &amp; Tech Skills </em></strong><br> 💥 Crack Every Tech Interview with Confidence<br> 🔥 Up to <strong>70% OFF</strong> — Limited-Time Offer<br> <a href="https://www.educative.io/unlimited?aff=xkRD"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><p>At the core of this ecosystem sit three dominant tools: <strong>llama.cpp</strong>, <strong>Ollama</strong>, and <strong>LM Studio</strong>. While all three enable local inference, they serve distinct architectural layers and workflows.</p><p>Choosing the right toolchain for running AI models locally isn’t about finding a single “best” option — it’s about matching technologies to your workflow, hardware, and deployment goals. The modern local AI ecosystem is layered: <strong>Ollama</strong> and <strong>LM Studio</strong> are polished wrappers that simplify model management; <strong>llama.cpp</strong> is the high-performance inference engine powering many of these tools; <strong>Hugging Face</strong> is the central repository where models live; and <strong>ONNX</strong> is the interoperability standard that lets you deploy models across diverse hardware and frameworks.</p><h3>Core Distinctions at a Glance</h3><p>For years, building with large language models meant sending prompts to a cloud API, waiting for a response, and paying for every token. That model remains valuable when developers need frontier-scale intelligence, global availability, or managed infrastructure. But it is no longer the only practical way to build AI applications.</p><p>A powerful local AI ecosystem has emerged — one that allows developers to download open-weight models, run inference directly on personal computers or private servers, expose local API endpoints, and build applications without continuously transmitting sensitive information to external providers.</p><p>At the center of this ecosystem are three important technologies:</p><ul><li><strong>Ollama</strong>, which makes local model management and application integration simple.</li><li><strong>LM Studio</strong>, which provides an accessible graphical environment for discovering, testing, and serving models.</li><li><strong>llama.cpp</strong>, the high-performance inference engine that gives developers precise control over hardware utilization, quantization, memory, and execution.</li></ul><p>These tools overlap, but they are not interchangeable. Each serves a different layer of the local AI stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*uXLgIIF7yc-yCbiGgFroHg.png" /></figure><p>This matrix clarifies a common confusion: <strong>Ollama, LM Studio, and llama.cpp are runtimes</strong> (they execute models), while <strong>Hugging Face is a platform</strong> (it hosts models), and <strong>ONNX is a format + runtime ecosystem</strong> (it enables portability)</p><h3>Deep Dive: Understanding Each Layer of the Stack</h3><h3>1. Ollama — The Terminal-First Automation Runtime</h3><p><strong>What it is:</strong> Ollama is a background service that wraps <strong>llama.cpp</strong> and exposes an <strong>OpenAI-compatible API</strong> on your localhost. It manages model downloads, versions, and runtime configuration via simple CLI commands.datacamp+3</p><p><strong>How it works:</strong></p><ul><li>Under the hood, Ollama uses <strong>llama.cpp</strong> for inference.</li><li>It maintains a curated library of models (Llama, Mistral, Gemma, Qwen, etc.) and handles quantization presets.</li><li>It runs as a daemon and serves requests over HTTP, making it easy to integrate with Python, JavaScript, or any language that can call REST APIs.datacamp+2</li></ul><p><strong>Interface:</strong></p><ul><li>Primarily <strong>CLI</strong> (ollama run, ollama pull, ollama list)</li><li>Optional desktop app and robust <strong>REST API</strong> (OpenAI-compatible endpoints)</li></ul><p><strong>Best for:</strong></p><ul><li>Developers building applications that need a local LLM backend</li><li>Scripting, prototyping, and CI/CD pipelines</li><li>Teams that want an OpenAI-equivalent API without cloud dependenciesdatacamp+2</li></ul><p><strong>Why it matters in 2026:</strong> Ollama has become the de facto standard for local LLM development due to its simplicity, cross-platform support, and strong ecosystem integrations (e.g., Open WebUI, LangChain, LlamaIndex)</p><h3>2. LM Studio — The Visual ChatGPT Alternative</h3><p><strong>What it is:</strong> LM Studio is a polished <strong>desktop GUI application</strong> that lets you discover, download, and chat with local models — completely offline. It bundles <strong>llama.cpp</strong> (and sometimes other backends like MLX) under a user-friendly interface.</p><p><strong>How it works:</strong></p><ul><li>Provides a built-in <strong>model browser</strong> connected to Hugging Face and other sources.</li><li>Offers sliders for context length, temperature, top-p, and other inference parameters.</li><li>Can run in <strong>local server mode</strong> to expose an API, but its primary strength is the interactive chat UI.techsy+1</li></ul><p><strong>Interface:</strong></p><ul><li>High-fidelity <strong>desktop GUI</strong> (Windows, macOS, Linux)</li><li>No terminal required; ideal for non-technical users</li></ul><p><strong>Best for:</strong></p><ul><li>Writers, researchers, and analysts who want a ChatGPT-like experience offline</li><li>Rapid model evaluation and comparison without writing code</li><li>Privacy-sensitive workflows where data must never leave the device data.</li></ul><p><strong>Limitations:</strong></p><ul><li>Less suited for headless automation or production API serving compared to Ollama or vLLM.</li><li>Some advanced configuration options are abstracted away, which can frustrate power users.</li></ul><h3>3. llama.cpp — The Engine Room</h3><p><strong>What it is:</strong> <strong>llama.cpp</strong> is the legendary open-source <strong>C/C++ inference engine</strong> that made running large language models on consumer hardware feasible. It introduced the <strong>GGUF</strong> format, enabling efficient quantization and execution on CPUs, Apple Silicon, and GPUs.tribuneindia+2<a href="https://www.youtube.com/watch?v=pyFRIlk4se0">youtube</a></p><p><strong>How it works:</strong></p><ul><li>Compiles directly to <strong>bare-metal</strong> code, optimizing for the specific hardware (CPU, Metal, CUDA, Vulkan, etc.).</li><li>Supports advanced features like <strong>layer offloading</strong> (splitting models between GPU and CPU), custom quantization schemes, and multi-backend inference.</li></ul><p><strong>Interface:</strong></p><ul><li>Raw <strong>C++ binaries</strong> and command-line flags</li><li>Can be embedded into custom servers or applications</li><li>Often used indirectly via Ollama, LM Studio, or other wrappers</li></ul><p><strong>Best for:</strong></p><ul><li>Performance engineers who need <strong>maximum throughput</strong> and minimal latency</li><li>Edge deployments on CPU-only or resource-constrained devices</li><li>Custom quantization experiments and fine-grained control over inference parameters</li></ul><p><strong>Why it’s foundational:</strong> Most local LLM tools (including Ollama and LM Studio) rely on <strong>llama.cpp</strong> under the hood. Understanding it gives you leverage to optimize beyond what wrappers expose.</p><h3>4. Hugging Face — The Model Hub &amp; Ecosystem Platform</h3><p><strong>What it is:</strong> Hugging Face is the <strong>“GitHub of Machine Learning”</strong> — a platform hosting hundreds of thousands of models, datasets, and demo applications. It is not a runtime; it’s where you <strong>find, evaluate, and download</strong> models.</p><p><strong>How it works:</strong></p><ul><li>Provides <strong>model cards</strong>, research papers, and community fine-tunes.</li><li>Offers Python libraries (transformers, diffusers) for loading and running models.</li><li>Integrates with local runtimes: LM Studio, Ollama, and others can pull models directly from Hugging Face.</li></ul><p><strong>Interface:</strong></p><ul><li>Web platform with search, filters, and leaderboards</li><li>Python ecosystem for programmatic access and pipelines</li></ul><p><strong>Best for:</strong></p><ul><li>Discovering state-of-the-art models and community variants</li><li>Downloading raw weights (Safetensors, PyTorch, GGUF) for local execution</li><li>Building initial AI pipelines and experimenting with fine-tuning<a href="https://pinggy.io/blog/top_5_local_llm_tools_and_models/">pinggy</a></li></ul><p><strong>Strategic role:</strong> In a local AI workflow, Hugging Face is your <strong>source of truth</strong> for models. You browse there, then run locally via Ollama, LM Studio, or llama.cpp</p><h3>5. ONNX — The Universal Model Interchange Format</h3><p><strong>What it is:</strong> <strong>ONNX (Open Neural Network Exchange)</strong> is an open standard developed by Microsoft and partners to enable <strong>model portability</strong> across frameworks (PyTorch, TensorFlow, scikit-learn, etc.) and hardware (Intel CPUs, AMD GPUs, mobile NPUs).</p><p><strong>How it works:</strong></p><ul><li>You train a model in your preferred framework, then <strong>export to ONNX</strong> (.onnx file).</li><li>The <strong>ONNX Runtime</strong> executes the model with hardware-specific optimizations (e.g., DirectML, TensorRT, OpenVINO).</li></ul><p><strong>Scope:</strong></p><ul><li>Supports <strong>all ML models</strong>: transformers (BERT, LLMs), CNNs (ResNet), classical ML (XGBoost), and more.</li><li>Unlike <strong>llama.cpp</strong>, which is specialized for autoregressive LLMs in GGUF, ONNX is <strong>framework-agnostic</strong>.</li></ul><p><strong>Best for:</strong></p><ul><li>Enterprise deployments requiring <strong>cross-platform consistency</strong></li><li>Edge computing on mobile, IoT, or specialized accelerators</li><li>Computer vision, speech, and traditional ML workloads alongside LLMs</li></ul><p><strong>When to choose ONNX:</strong> If you’re building a production app that must run the same model on Windows, Linux, macOS, and mobile — especially for non-LLM workloads — ONNX is the strategic choice.</p><h3>Decision Framework: Which Tool Fits Your Use Case?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*NuFd-g9ExzsvR-Iw9qEotg.png" /></figure><h3>Practical Workflows: Putting It All Together</h3><h3>Workflow A: The Non-Coder’s Path to Local AI</h3><ol><li><strong>Install LM Studio</strong> → browse models → click to download → start chatting.</li><li>Use the built-in <strong>local server mode</strong> if you need an API later.datacamp+2</li></ol><h3>Workflow B: The Developer’s Fast Track</h3><ol><li><strong>Install Ollama</strong> → ollama pull llama3.2 → ollama run llama3.2.</li><li>Integrate via <strong>OpenAI-compatible API</strong> in your Python/JS app.</li><li>Optionally, add <strong>Open WebUI</strong> for a browser-based chat interface.</li></ol><h3>Workflow C: The Performance Engineer’s Route</h3><ol><li>Clone and build <strong>llama.cpp</strong> from source for your hardware (CPU, Metal, CUDA).</li><li>Convert or download <strong>GGUF</strong> models from Hugging Face.</li><li>Tune quantization, context length, and layer offloading for maximum tokens/sec.</li></ol><h3>Workflow D: The Enterprise Deployment</h3><ol><li>Train or fine-tune in <strong>PyTorch/TensorFlow</strong> → export to <strong>ONNX</strong>.</li><li>Optimize with <strong>ONNX Runtime</strong> for target hardware (Intel, AMD, mobile).</li><li>Deploy across platforms with consistent performance and tooling.</li></ol><pre>pip install tensorflow scikit-learn pandas numpy requests sentence-transformers fastapi uvicorn pydantic</pre><pre>from __future__ import annotations<br><br>import os<br>import json<br>import time<br>import logging<br>from dataclasses import dataclass, asdict<br>from typing import Any, Literal<br><br>import numpy as np<br>import pandas as pd<br>import requests<br>import tensorflow as tf<br><br>from pydantic import BaseModel<br>from sklearn.cluster import KMeans<br>from sklearn.preprocessing import StandardScaler<br>from sentence_transformers import SentenceTransformer<br><br><br>logging.basicConfig(<br>    level=logging.INFO,<br>    format=&quot;%(asctime)s | %(levelname)s | %(message)s&quot;<br>)<br>logger = logging.getLogger(&quot;local-ai-lab&quot;)<br><br><br>Provider = Literal[&quot;lm_studio&quot;, &quot;ollama&quot;, &quot;llama_cpp&quot;]<br><br><br>@dataclass<br>class LocalLLMConfig:<br>    provider: Provider = &quot;lm_studio&quot;<br>    base_url: str = &quot;http://localhost:1234&quot;<br>    model: str = &quot;local-model&quot;<br>    timeout_seconds: int = 120<br>    temperature: float = 0.3<br>    max_tokens: int = 300<br><br><br>class PromptRequest(BaseModel):<br>    prompt: str<br>    system_prompt: str = &quot;You are a precise local AI assistant.&quot;<br>    metadata: dict[str, Any] = {}<br><br><br>class LocalLLMClient:<br>    &quot;&quot;&quot;<br>    Unified adapter for popular local LLM servers.<br><br>    LM Studio:<br>      base_url = http://localhost:1234<br>      endpoint = /v1/chat/completions<br><br>    Ollama:<br>      base_url = http://localhost:11434<br>      endpoint = /api/chat<br><br>    llama.cpp server:<br>      commonly exposes an OpenAI-compatible endpoint:<br>      base_url = http://localhost:8080<br>      endpoint = /v1/chat/completions<br>    &quot;&quot;&quot;<br><br>    def __init__(self, config: LocalLLMConfig):<br>        self.config = config<br><br>    def generate(self, prompt: str, system_prompt: str) -&gt; str:<br>        if self.config.provider in (&quot;lm_studio&quot;, &quot;llama_cpp&quot;):<br>            return self._generate_openai_compatible(prompt, system_prompt)<br><br>        if self.config.provider == &quot;ollama&quot;:<br>            return self._generate_ollama(prompt, system_prompt)<br><br>        raise ValueError(f&quot;Unsupported provider: {self.config.provider}&quot;)<br><br>    def _generate_openai_compatible(self, prompt: str, system_prompt: str) -&gt; str:<br>        url = f&quot;{self.config.base_url.rstrip(&#39;/&#39;)}/v1/chat/completions&quot;<br><br>        payload = {<br>            &quot;model&quot;: self.config.model,<br>            &quot;messages&quot;: [<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>            ],<br>            &quot;temperature&quot;: self.config.temperature,<br>            &quot;max_tokens&quot;: self.config.max_tokens,<br>        }<br><br>        response = requests.post(<br>            url,<br>            json=payload,<br>            timeout=self.config.timeout_seconds,<br>        )<br>        response.raise_for_status()<br><br>        data = response.json()<br>        return data[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;].strip()<br><br>    def _generate_ollama(self, prompt: str, system_prompt: str) -&gt; str:<br>        url = f&quot;{self.config.base_url.rstrip(&#39;/&#39;)}/api/chat&quot;<br><br>        payload = {<br>            &quot;model&quot;: self.config.model,<br>            &quot;stream&quot;: False,<br>            &quot;options&quot;: {<br>                &quot;temperature&quot;: self.config.temperature,<br>                &quot;num_predict&quot;: self.config.max_tokens,<br>            },<br>            &quot;messages&quot;: [<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>            ],<br>        }<br><br>        response = requests.post(<br>            url,<br>            json=payload,<br>            timeout=self.config.timeout_seconds,<br>        )<br>        response.raise_for_status()<br><br>        data = response.json()<br>        return data[&quot;message&quot;][&quot;content&quot;].strip()<br><br><br>class PromptEmbeddingEngine:<br>    def __init__(self, model_name: str = &quot;all-MiniLM-L6-v2&quot;):<br>        logger.info(&quot;Loading embedding model: %s&quot;, model_name)<br>        self.model = SentenceTransformer(model_name)<br><br>    def embed(self, texts: list[str]) -&gt; np.ndarray:<br>        vectors = self.model.encode(<br>            texts,<br>            normalize_embeddings=True,<br>            show_progress_bar=False,<br>        )<br>        return np.asarray(vectors, dtype=np.float32)<br><br><br>class PromptClusterer:<br>    def __init__(self, n_clusters: int = 4):<br>        self.n_clusters = n_clusters<br>        self.scaler = StandardScaler()<br>        self.model = KMeans(<br>            n_clusters=n_clusters,<br>            random_state=42,<br>            n_init=&quot;auto&quot;,<br>        )<br><br>    def fit_predict(self, embeddings: np.ndarray) -&gt; np.ndarray:<br>        scaled = self.scaler.fit_transform(embeddings)<br>        return self.model.fit_predict(scaled)<br><br><br>class AnomalyAutoencoder:<br>    &quot;&quot;&quot;<br>    Learns normal prompt-embedding patterns.<br>    High reconstruction error indicates a potentially unusual prompt.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, input_dim: int):<br>        self.input_dim = input_dim<br>        self.scaler = StandardScaler()<br><br>        inputs = tf.keras.Input(shape=(input_dim,), name=&quot;embedding&quot;)<br><br>        x = tf.keras.layers.Dense(128, activation=&quot;relu&quot;)(inputs)<br>        x = tf.keras.layers.Dropout(0.10)(x)<br>        x = tf.keras.layers.Dense(48, activation=&quot;relu&quot;)(x)<br>        latent = tf.keras.layers.Dense(16, activation=&quot;relu&quot;, name=&quot;latent&quot;)(x)<br><br>        x = tf.keras.layers.Dense(48, activation=&quot;relu&quot;)(latent)<br>        x = tf.keras.layers.Dense(128, activation=&quot;relu&quot;)(x)<br>        outputs = tf.keras.layers.Dense(input_dim, activation=&quot;linear&quot;)(x)<br><br>        self.model = tf.keras.Model(inputs=inputs, outputs=outputs)<br>        self.model.compile(<br>            optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),<br>            loss=&quot;mse&quot;,<br>        )<br><br>        self.threshold: float | None = None<br><br>    def fit(self, embeddings: np.ndarray, epochs: int = 30) -&gt; None:<br>        scaled = self.scaler.fit_transform(embeddings)<br><br>        self.model.fit(<br>            scaled,<br>            scaled,<br>            epochs=epochs,<br>            batch_size=min(16, len(scaled)),<br>            validation_split=0.2 if len(scaled) &gt;= 10 else 0.0,<br>            verbose=0,<br>        )<br><br>        reconstruction = self.model.predict(scaled, verbose=0)<br>        errors = np.mean(np.square(scaled - reconstruction), axis=1)<br><br>        # 95th percentile: top 5% most unusual training-like observations<br>        self.threshold = float(np.percentile(errors, 95))<br>        logger.info(&quot;Anomaly threshold learned: %.6f&quot;, self.threshold)<br><br>    def score(self, embedding: np.ndarray) -&gt; tuple[float, bool]:<br>        if self.threshold is None:<br>            raise RuntimeError(&quot;Train the autoencoder before scoring prompts.&quot;)<br><br>        scaled = self.scaler.transform(embedding.reshape(1, -1))<br>        reconstruction = self.model.predict(scaled, verbose=0)<br><br>        error = float(np.mean(np.square(scaled - reconstruction)))<br>        return error, error &gt; self.threshold<br><br><br>class LocalAIIntelligenceLab:<br>    def __init__(self, llm_config: LocalLLMConfig):<br>        self.llm = LocalLLMClient(llm_config)<br>        self.embedder = PromptEmbeddingEngine()<br>        self.clusterer = PromptClusterer(n_clusters=4)<br>        self.anomaly_model: AnomalyAutoencoder | None = None<br>        self.history: list[dict[str, Any]] = []<br><br>    def train(self, historical_prompts: list[str]) -&gt; pd.DataFrame:<br>        if len(historical_prompts) &lt; 8:<br>            raise ValueError(&quot;Provide at least 8 prompts for meaningful local training.&quot;)<br><br>        embeddings = self.embedder.embed(historical_prompts)<br>        clusters = self.clusterer.fit_predict(embeddings)<br><br>        self.anomaly_model = AnomalyAutoencoder(input_dim=embeddings.shape[1])<br>        self.anomaly_model.fit(embeddings)<br><br>        training_report = pd.DataFrame({<br>            &quot;prompt&quot;: historical_prompts,<br>            &quot;cluster&quot;: clusters,<br>        })<br><br>        return training_report<br><br>    def analyze_and_generate(self, request: PromptRequest) -&gt; dict[str, Any]:<br>        if self.anomaly_model is None:<br>            raise RuntimeError(&quot;Call train() before analyzing new prompts.&quot;)<br><br>        start = time.perf_counter()<br><br>        embedding = self.embedder.embed([request.prompt])[0]<br>        anomaly_score, is_anomaly = self.anomaly_model.score(embedding)<br><br>        response = self.llm.generate(<br>            prompt=request.prompt,<br>            system_prompt=request.system_prompt,<br>        )<br><br>        latency_ms = round((time.perf_counter() - start) * 1000, 2)<br><br>        record = {<br>            &quot;timestamp&quot;: pd.Timestamp.utcnow().isoformat(),<br>            &quot;prompt&quot;: request.prompt,<br>            &quot;response&quot;: response,<br>            &quot;anomaly_score&quot;: round(anomaly_score, 6),<br>            &quot;is_anomalous&quot;: is_anomaly,<br>            &quot;latency_ms&quot;: latency_ms,<br>            &quot;provider&quot;: self.llm.config.provider,<br>            &quot;model&quot;: self.llm.config.model,<br>            &quot;metadata&quot;: request.metadata,<br>        }<br><br>        self.history.append(record)<br>        return record<br><br>    def save_history(self, filename: str = &quot;local_ai_history.json&quot;) -&gt; None:<br>        with open(filename, &quot;w&quot;, encoding=&quot;utf-8&quot;) as file:<br>            json.dump(self.history, file, indent=2, ensure_ascii=False)<br><br>    def analytics(self) -&gt; pd.DataFrame:<br>        if not self.history:<br>            return pd.DataFrame()<br><br>        frame = pd.DataFrame(self.history)<br>        return frame[<br>            [<br>                &quot;timestamp&quot;,<br>                &quot;provider&quot;,<br>                &quot;model&quot;,<br>                &quot;latency_ms&quot;,<br>                &quot;anomaly_score&quot;,<br>                &quot;is_anomalous&quot;,<br>            ]<br>        ]<br><br><br>if __name__ == &quot;__main__&quot;:<br>    config = LocalLLMConfig(<br>        provider=os.getenv(&quot;LOCAL_LLM_PROVIDER&quot;, &quot;lm_studio&quot;),<br>        base_url=os.getenv(&quot;LOCAL_LLM_BASE_URL&quot;, &quot;http://localhost:1234&quot;),<br>        model=os.getenv(&quot;LOCAL_LLM_MODEL&quot;, &quot;local-model&quot;),<br>    )<br><br>    lab = LocalAIIntelligenceLab(config)<br><br>    training_prompts = [<br>        &quot;Explain Python decorators with a short example.&quot;,<br>        &quot;Write unit tests for a FastAPI endpoint.&quot;,<br>        &quot;Compare PostgreSQL and MongoDB for analytics.&quot;,<br>        &quot;Explain vector embeddings in machine learning.&quot;,<br>        &quot;How does a Solidity reentrancy attack work?&quot;,<br>        &quot;Draft a secure smart contract review checklist.&quot;,<br>        &quot;Describe how Kubernetes services route traffic.&quot;,<br>        &quot;Help optimize a Pandas DataFrame transformation.&quot;,<br>        &quot;Explain TensorFlow model overfitting.&quot;,<br>        &quot;Create a React component for a transaction dashboard.&quot;,<br>        &quot;What is model quantization for local LLM inference?&quot;,<br>        &quot;Explain retrieval augmented generation architecture.&quot;,<br>    ]<br><br>    report = lab.train(training_prompts)<br><br>    print(&quot;\nTraining clusters:&quot;)<br>    print(report[[&quot;prompt&quot;, &quot;cluster&quot;]].to_string(index=False))<br><br>    result = lab.analyze_and_generate(<br>        PromptRequest(<br>            prompt=(<br>                &quot;Design a privacy-first RAG system that indexes local smart &quot;<br>                &quot;contract audit reports and identifies security patterns.&quot;<br>            ),<br>            system_prompt=(<br>                &quot;You are a senior AI and blockchain security engineer. &quot;<br>                &quot;Give practical, technically precise guidance.&quot;<br>            ),<br>            metadata={&quot;domain&quot;: &quot;blockchain-security&quot;}<br>        )<br>    )<br><br>    print(&quot;\nGenerated analysis:&quot;)<br>    print(json.dumps(result, indent=2, ensure_ascii=False))<br><br>    lab.save_history()<br>    print(&quot;\nRuntime analytics:&quot;)<br>    print(lab.analytics().to_string(index=False))</pre><h3>Conclusion: Match the Tool to Your Constraints</h3><p>The “best” choice depends on three axes:</p><ul><li><strong>Technical expertise:</strong> CLI comfort vs. GUI preference</li><li><strong>Hardware:</strong> CPU-only, Apple Silicon, or NVIDIA GPU</li><li><strong>Deployment goal:</strong> Personal experimentation, app development, or production serving</li></ul><p><strong>Thumb Rule for 2026:</strong></p><ul><li>Start with <strong>Ollama</strong> if you’re a developer — it’s the fastest path to a working local API.</li><li>Choose <strong>LM Studio</strong> if you want a ChatGPT-like desktop experience without touching the terminal.</li><li>Go deep with <strong>llama.cpp</strong> if you need every last bit of performance or run on constrained hardware.</li><li>Use <strong>Hugging Face</strong> as your model source, regardless of runtime.</li><li>Adopt <strong>ONNX</strong> when cross-platform deployment or non-LLM workloads are in scope.</li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=392796637d2e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/navigating-the-local-ai-stack-a-developers-guide-to-ollama-lm-studio-llama-cpp-392796637d2e">Navigating the Local AI Stack: A Developer’s Guide to Ollama, LM Studio, llama.cpp,</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Medallion Architecture: An End-to-End Guide to Data Engineering and Multi-Cloud…]]></title>
            <link>https://medium.com/codetodeploy/mastering-medallion-architecture-an-end-to-end-guide-to-data-engineering-and-multi-cloud-068c54ef8267?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/068c54ef8267</guid>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Sat, 25 Jul 2026 16:50:46 GMT</pubDate>
            <atom:updated>2026-07-29T07:46:25.378Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong><em>Mastering Medallion Architecture:</em></strong><em> </em>An End-to-End Guide to Data Engineering and Multi-Cloud Lakehouses Across AWS, Azure, and Google Cloud</h3><p>Introduction:</p><p>The medallion architecture is a multi-hop data design pattern that improves data structure, quality, and semantic value as information moves through Bronze, Silver, and Gold layers. Bronze preserves source evidence and replayability. Silver resolves structural and entity-level truth. Gold expresses metrics, features, and data products in forms optimized for decisions. The pattern is most powerful in a lakehouse, where low-cost object storage, open table formats, transactional guarantees, scalable compute, and warehouse-style governance converge.</p><blockquote><strong><em>💥 Master Any Skills in 3 Months</em></strong><br><em> </em>📚 Up to <strong>50% OFF</strong> Premium Courses<br> ⏰ Limited-Time Offer<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><p>Its simplicity is deceptive. Successful implementations do more than copy data between three locations. They define contracts for ingestion, schema evolution, data quality, privacy, lineage, reconciliation, serving latency, ownership, and recovery. They also support both batch and streaming paths without creating two incompatible systems.</p><p><strong>Architectural outcome </strong>A well-designed medallion platform makes every downstream number traceable to source evidence, every failed transformation replayable, and every business-facing dataset explicitly owned and measurable.</p><h3>The Problem the Pattern Solves</h3><p>A data lake can accept structured, semi-structured, and unstructured information at enormous scale, but storage alone does not produce shared meaning. Without progressive refinement, the lake becomes a collection of files with inconsistent schemas, duplicate entities, unclear provenance, uncontrolled sensitive data, and unpredictable query costs. A traditional warehouse offers stronger structure but may impose early modeling decisions, expensive data movement, and limited support for high-volume logs, media, and machine-learning workloads.</p><p>The lakehouse addresses this tension by bringing transactional table behavior and governance to object storage while retaining independent compute. The medallion pattern supplies the workflow discipline: land faithfully, conform carefully, and publish intentionally. Databricks describes this as progressive quality improvement; Google Cloud frames it as a flexible blueprint rather than a rigid product; Microsoft’s Azure Databricks guidance emphasizes layer-specific validation and modeling.</p><h3>The Trust Gradient: Bronze → Silver → Gold</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*fstNr7aKwOtFbYE2dqvvxQ.png" /><figcaption>Table Depicting the Role of Bronze, Silver and Gold</figcaption></figure><p>Bronze: Preserve the Evidence</p><p>Bronze is an append-oriented system of record for received data. It should retain the source payload or a lossless representation, the source identity, event or extraction time, ingestion time, batch/run identifier, file path or message coordinates, checksum, schema version, and quarantine status. The purpose is fast ingestion, forensic traceability, and deterministic replay — not analytical convenience.</p><p>· Accept databases through snapshots or CDC; SaaS applications through APIs; events through Kafka-, Kinesis-, Event Hubs-, or Pub/Sub-class services; and files through managed transfer or object storage.</p><p>· Separate malformed or policy-violating records from accepted raw data without silently dropping them.</p><p>· Use encryption, immutability controls where required, retention policies, and restricted access because Bronze often contains the broadest sensitive-data footprint.</p><p>· Partition by stable operational keys such as ingestion date and source — not by high-cardinality fields that create small files.</p><h3>Silver: Establish Reusable Truth</h3><p>Silver is where raw observations become dependable enterprise entities and events. Processing commonly includes parsing, schema enforcement, type normalization, unit and time-zone conversion, null handling, deduplication, CDC merge logic, reference-data enrichment, master-data matching, late-arriving event handling, and privacy transformations. The result should be reusable across multiple use cases, not shaped around one dashboard.</p><p>· Quarantine failed records with explicit reason codes and preserve a remediation path.</p><p>· Apply deterministic keys, survivorship rules, and slowly changing dimension logic where history matters.</p><p>· Publish data-quality metrics: completeness, validity, uniqueness, referential integrity, freshness, and volume deviation.</p><p>· Avoid over-curating. Project-specific calculations generally belong in Gold; Silver should remain broadly reusable.</p><h3>Gold: Publish Business Meaning</h3><p>Gold converts conformed data into decision-ready products. Typical structures include Kimball-style star schemas, denormalized marts, materialized aggregates, executive KPIs, customer 360 views, operational serving tables, and machine-learning feature sets. Each product needs a business owner, technical owner, metric definition, refresh objective, access policy, lineage, and quality service-level objective.</p><p><strong>A crucial distinction </strong>Silver answers, “What happened, consistently?” Gold answers, “What does it mean for this business decision?”</p><h3>An End-to-End Reference Flow</h3><p>1. Discover and classify sources. Inventory owners, interfaces, data sensitivity, change rates, schemas, retention, contractual restrictions, and recovery requirements.</p><p>2. Ingest idempotently. Assign immutable ingestion identifiers, capture checksums and watermarks, and make retries safe.</p><p>3. Land in Bronze. Preserve the source representation and operational metadata; register assets in the catalog.</p><p>4. Validate and quarantine. Run structural, policy, and malware/content checks before wider processing.</p><p>5. Build Silver incrementally. Parse, standardize, deduplicate, merge CDC, reconcile counts, and publish reusable entities.</p><p>6. Build Gold as products. Apply business logic, dimensional models, aggregates, semantic definitions, and ML feature logic.</p><p>7. Serve through fit-for-purpose engines. Use SQL warehouses, serverless query, BI semantic layers, feature stores, APIs, vector retrieval, or reverse ETL.</p><p>8. Observe and govern continuously. Monitor freshness, quality, lineage, cost, access, drift, and business adoption.</p><h3>Batch, Streaming, and the Lambda Fallacy</h3><p>Medallion does not imply batch-only processing. The same logical layers can be implemented with micro-batches or continuous streams. Bronze can capture immutable events; Silver can maintain streaming tables with deduplication, watermarking, and stateful joins; Gold can incrementally refresh aggregates and materialized views. Databricks documents streaming tables and materialized views as a unified pipeline approach, while AWS Glue supports continuous streaming ETL from Kinesis, Kafka, and Amazon MSK.[1][8]</p><p>The goal is not to build separate “real-time” and “historical” truth systems. Instead, use one event model, one catalog, consistent transformation logic, and explicit latency classes. A daily finance close and a sub-minute fraud signal can share Bronze evidence and Silver identities while publishing different Gold products.</p><h3>Cross-Cloud Technology Map</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*4W4DNcxWNRhx_r-H_cn0Ag.png" /></figure><h3>AWS Implementation Blueprint</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NC3baJP-t_UOoCsPy1_vSQ.png" /></figure><p>A modern AWS implementation can keep all three logical layers on Amazon S3 using Apache Iceberg, Delta Lake, or Hudi, or combine S3-based Bronze with Redshift-managed Silver and Gold. AWS Prescriptive Guidance recommends distinct raw, stage, and analytics data layers, and newer AWS lakehouse capabilities unify access across S3 data lakes and Redshift.</p><h3>Ingestion and Bronze</h3><p>· Amazon DMS captures database changes; Amazon AppFlow ingests supported SaaS data; DataSync and Transfer Family handle files; Kinesis Data Streams, Amazon MSK, and Amazon Data Firehose ingest events.</p><p>· Amazon S3 stores immutable payloads with versioning, server-side encryption using AWS KMS, Object Lock where mandated, and lifecycle transitions to colder tiers.</p><p>· EventBridge routes object and application events; Step Functions orchestrates multi-stage workflows. Lambda is useful for lightweight validation and routing, while high-volume transformations belong in Glue, EMR, or a streaming engine.</p><h3>Silver Processing and Transactional Tables</h3><p>AWS Glue provides serverless Spark ETL, crawlers, job bookmarks, data-quality capabilities, and streaming ETL. Amazon EMR offers deeper runtime control and broad open-source framework support. Store Silver as Parquet-backed open tables when updates, deletes, time travel, schema evolution, and concurrent writes matter.</p><p>Apache Iceberg and Delta Lake are separate choices. AWS Glue supports both, and Iceberg tables registered in the Glue Data Catalog can be queried and written by multiple AWS engines, including current Amazon Redshift capabilities.[7] Choose one canonical format per data domain unless a tested interoperability requirement justifies otherwise.</p><h3>Gold Serving</h3><p>· Amazon Athena provides serverless SQL over cataloged lake data.</p><p>· Amazon Redshift or Redshift Serverless serves high-concurrency BI, dimensional models, materialized views, and governed warehouse workloads; Redshift Spectrum and Iceberg integration bridge warehouse and lake.</p><p>· Amazon QuickSight consumes curated metrics; SageMaker AI consumes governed training and feature datasets; Amazon Bedrock applications can use Gold documents, embeddings, and entitlement-filtered retrieval corpora.</p><h3>Governance and Operations</h3><p>The AWS Glue Data Catalog is the technical catalog; Lake Formation applies fine-grained permissions across cataloged resources; Amazon DataZone supports discovery, subscription, ownership, and lineage-oriented workflows. IAM, KMS, CloudTrail, Macie, CloudWatch, VPC endpoints, and multi-account landing-zone controls complete the security and operational plane.</p><h3>Azure Databricks and Microsoft Blueprint</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Ypbl54OKoKhU-drTSaHcQ.png" /></figure><p>On Azure, ADLS Gen2 commonly supplies object storage while Azure Databricks provides ingestion, Spark processing, Delta Lake tables, SQL serving, and Unity Catalog governance. Azure Data Factory can integrate enterprise sources and orchestrate hybrid movement; Event Hubs supports event ingestion; Microsoft Purview contributes broader catalog and governance integration. Microsoft’s guidance recommends raw append-oriented Bronze, validated and deduplicated Silver, and enriched or aggregated Gold.</p><p>· Bronze: Auto Loader or streaming connectors land source-faithful Delta tables with ingestion metadata.</p><p>· Silver: Lakeflow pipelines or Spark jobs enforce schemas, expectations, CDC merges, deduplication, and entity conformance.</p><p>· Gold: Databricks SQL warehouses, materialized views, semantic models, and Power BI serve consumption workloads.</p><p>· Governance: Unity Catalog centralizes catalogs, schemas, tables, volumes, functions, models, privileges, auditability, and lineage across workspaces.</p><p>Where Microsoft Fabric is the strategic analytics surface, OneLake and Fabric lakehouses can express the same layering logic. The architecture remains a pattern: logical layer boundaries and contracts matter more than whether each layer is a separate account, container, catalog, schema, or database.</p><h3>Google Cloud Blueprint</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O9-1MjqvcHjdf2lc-EHrow.png" /></figure><p>Google Cloud describes medallion architecture as a flexible framework that can be implemented through lakehouse, warehouse, or hybrid technologies.A common design lands raw data in Cloud Storage or BigQuery, transforms with Dataflow, Dataproc, BigQuery SQL, or Dataform, governs with Dataplex Universal Catalog and IAM, and serves through BigQuery and Looker.</p><p>· Bronze: Datastream captures CDC, Pub/Sub receives events, Storage Transfer Service moves objects, and Cloud Storage provides durable landing zones.</p><p>· Silver: Dataflow handles streaming and batch pipelines; Dataproc runs Spark; BigQuery SQL and Dataform implement ELT, assertions, and dependency-aware transformations.</p><p>· Gold: BigQuery tables, materialized views, BI Engine, and Looker semantic models provide governed, high-performance consumption.</p><p>· AI: Vertex AI trains and serves models from governed datasets, while vector search and generative AI workflows consume curated, permission-aware corpora.</p><p>· Governance: Dataplex Universal Catalog, Sensitive Data Protection, IAM Conditions, CMEK, Audit Logs, and organization policies control discovery, classification, access, and evidence.</p><h3>The Open Table Format Decision</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*JVHoVj7UD67TcgyQcs4CrQ.png" /></figure><p>Table format is not the architecture. It is one implementation mechanism. The decisive questions are: Which engines must read and write? How are concurrent commits coordinated? What are the recovery objectives? Which deletion and privacy semantics are required? Who operates compaction, snapshot expiration, orphan cleanup, statistics, and small-file control?</p><h3>Data Quality as a Layer Gate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*uuP0NgAD2mYIG8ERa-ddPQ.png" /></figure><p>Quality should be expressed as code and emitted as data. Store results, thresholds, tested columns, sample failures, rule versions, run identifiers, and lineage links. Distinguish record-level rejects from dataset-level failure. A pipeline can quarantine 0.01% of malformed telemetry and proceed, but a 12% revenue reconciliation gap should stop publication.</p><h3>Security, Privacy, and Governance by Design</h3><p>· Identity first: use workload identities and short-lived credentials; prohibit shared access keys.</p><p>· Least privilege by layer: Bronze access is narrow; Silver access is domain-scoped; Gold access follows approved products and business purposes.</p><p>· Protect sensitive attributes: classify automatically, tokenize or mask in Silver, enforce column/row policies, and propagate tags through lineage.</p><p>· Encrypt everywhere: provider-managed or customer-managed keys, private networking, controlled egress, and audited cross-account or cross-project sharing.</p><p>· Design deletion deliberately: map subject-deletion requirements through snapshots, backups, derived tables, features, indexes, and model artifacts.</p><p>· Treat governance as a control plane spanning all layers — not a fourth folder added after deployment.</p><h3>Metadata, Lineage, and Data Contracts</h3><p>Technical metadata answers where and how; business metadata answers what and why; operational metadata answers whether it is healthy. A production platform needs all three. Catalog tables and files, but also owners, domains, sensitivity, quality scores, freshness, costs, consumers, policies, transformation versions, and deprecation status.</p><p>A source contract should define schema, semantic meaning, keys, event-time behavior, update and delete semantics, expected volume, compatibility rules, producer SLO, and escalation owner. A Gold product contract adds business definitions, intended use, serving interface, refresh objective, and consumer-facing change policy.</p><h3>Orchestration, Idempotency, and Recovery</h3><p>Retries are normal; duplicates and partial failures are inevitable. Each hop should therefore be idempotent. Use source offsets or watermarks, immutable run IDs, deterministic merge keys, atomic table commits, and control tables that record input ranges, output versions, counts, checksums, status, and error reasons.</p><p>· Backfill from Bronze without overwriting evidence.</p><p>· Recompute Silver deterministically for a defined source range.</p><p>· Publish Gold atomically, preferably with versioned tables/views or replace-where semantics.</p><p>· Keep the last known good publication available during failures.</p><p>· Test restore, replay, and schema rollback — not merely backup creation.</p><h3>Performance and FinOps</h3><p>· Prefer columnar storage such as Parquet; compress appropriately and collect table statistics.</p><p>· Control file sizes through compaction; excessive tiny objects damage planning and scan performance.</p><p>· Partition on frequently filtered, moderately cardinal attributes; use clustering, sorting, or liquid/adaptive strategies where supported.</p><p>· Process changes incrementally through CDC, bookmarks, change feeds, or merge logic.</p><p>· Separate storage from elastic compute and apply workload isolation, autoscaling, concurrency controls, and query limits.</p><p>· Track cost per domain, pipeline, Gold product, query, and model. Unit economics reveal unused data products and runaway transformations.</p><p>· Apply lifecycle policies to raw evidence only after retention, replay, legal, and model-reproducibility requirements are resolved.</p><h3>Analytics, Machine Learning, and Generative AI</h3><p>Medallion architecture creates a natural bridge between analytics and AI. Bronze preserves training provenance. Silver supplies stable entities, events, labels, and privacy-safe attributes. Gold supplies reusable feature tables, training sets, evaluation cohorts, vector-ready documents, and business metrics.</p><p>For classical ML, prevent training-serving skew by versioning feature logic and reusing it for offline and online computation. Record dataset and table snapshots, code versions, feature definitions, model parameters, evaluation results, and approvals. For generative AI, treat document parsing, chunking, classification, redaction, embedding, and entitlement metadata as Silver transformations; publish Gold retrieval collections tailored to applications. Retrieval must enforce source permissions at query time, not merely during ingestion.</p><h3>Data Mesh Compatibility</h3><p>Medallion and data mesh solve different problems and can coexist. Medallion defines progressive refinement; data mesh defines decentralized ownership, data-as-a-product, federated governance, and self-service infrastructure. A domain may own its Bronze inputs, Silver canonical entities, and several Gold products, while the platform team supplies standardized ingestion, quality, catalog, policy, observability, and serving capabilities.[1]</p><h3>What Commonly Goes Wrong</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*zqpCPNEmxjGAABiVY8EL5w.png" /></figure><h3>A Pragmatic Delivery Roadmap</h3><p>9. Choose one valuable data product and trace it backward to two or three source systems.</p><p>10. Define layer contracts, quality rules, lineage, SLOs, access policy, and recovery objectives before selecting every service.</p><p>11. Build the platform thin slice: ingestion, Bronze evidence, one Silver entity, one Gold product, catalog, policy, and observability.</p><p>12. Automate infrastructure, tests, deployments, schema compatibility, and rollback through CI/CD and infrastructure as code.</p><p>13. Prove replay, late data, duplicates, deletion, access revocation, and cost behavior with failure drills.</p><p>14. Productize reusable capabilities: connectors, table templates, quality libraries, policy tags, orchestration patterns, and dashboards.</p><p>15. Scale by domain with federated ownership and platform guardrails; measure adoption and retire unused products.</p><h3>Production Readiness Checklist</h3><p>· □ Every dataset has a business owner, technical owner, classification, retention rule, and consumer.</p><p>· □ Bronze can be replayed without rereading the source; duplicate ingestion is harmless.</p><p>· □ Silver has deterministic keys, CDC semantics, reconciliation, quarantine, and measurable quality.</p><p>· □ Gold metrics have documented definitions, SLOs, lineage, access policies, and acceptance tests.</p><p>· □ Schema evolution and breaking-change procedures are automated and rehearsed.</p><p>· □ Open-table maintenance and small-file control are scheduled and monitored.</p><p>· □ Cross-layer lineage includes source coordinates, code version, run ID, and output table version.</p><p>· □ Disaster recovery, backfill, deletion, and last-known-good serving have been tested.</p><p>· □ Cost, freshness, quality, and usage are visible by domain and product.</p><p>· □ ML and GenAI artifacts preserve dataset versions, permissions, evaluation evidence, and model lineage.</p><h3>Contemplative Perspective: Architecture as an Epistemology</h3><p>The deeper value of medallion architecture is not that data becomes cleaner as it moves rightward. It is that an organization makes explicit how it knows what it claims to know. Bronze preserves observation. Silver reconciles observation into shared facts. Gold turns those facts into situated meaning for a decision. Each transition adds assumptions — and therefore must also add evidence, ownership, and tests.</p><p>This perspective changes the design question. Instead of asking, “Which service belongs in each layer?” ask, “What claim may consumers safely make after this layer, and what proof supports that claim?” That question prevents Bronze from becoming a forgotten archive, Silver from becoming an opaque transformation maze, and Gold from becoming a marketplace of ungoverned numbers.</p><h3>Advanced Python Programming</h3><p>Repository contains source code and Data Set for Medallion Architecture Using GCP,Microsoft Azure,AWS <a href="https://github.com/drraghavendra/Medallion-AI-Multi-Cloud-Architecture">https://github.com/drraghavendra/Medallion-AI-Multi-Cloud-Architecture</a></p><h3>Conclusion</h3><p>Medallion architecture endures because it provides a simple vocabulary for a difficult organizational problem: converting diverse, changing, and imperfect source data into trusted intelligence without destroying evidence or agility. Its Bronze, Silver, and Gold layers work across Databricks, AWS, Azure, Google Cloud, warehouses, lakehouses, batch systems, streams, analytics, ML, and generative AI.</p><p>The winning implementation is not the one with the most services. It is the one whose trust boundaries are explicit, whose transformations are replayable, whose metrics are owned, whose policies follow the data, and whose operational cost is observable. Build the layers as contracts, choose technologies to enforce those contracts, and treat every published dataset as a product with a promise.</p><p><strong>Final takeaway </strong>Raw data is not yet truth; cleaned data is not yet meaning; and business meaning is not trustworthy until its lineage, assumptions, quality, and ownership are visible.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/bhuwan-kc-583505319/?isSelfProfile=true"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=068c54ef8267" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/mastering-medallion-architecture-an-end-to-end-guide-to-data-engineering-and-multi-cloud-068c54ef8267">Mastering Medallion Architecture: An End-to-End Guide to Data Engineering and Multi-Cloud…</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Quantum Machine: Why Consciousness Remains the Final Frontier of Physics.]]></title>
            <link>https://blog.cubed.run/beyond-the-quantum-machine-why-consciousness-remains-the-final-frontier-of-physics-273a10a9b126?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/273a10a9b126</guid>
            <category><![CDATA[conscious-observer]]></category>
            <category><![CDATA[quantum-mechanics]]></category>
            <category><![CDATA[consciousness]]></category>
            <category><![CDATA[quantumwaveofprobability]]></category>
            <category><![CDATA[physical-measuring-device]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Fri, 03 Jul 2026 16:25:00 GMT</pubDate>
            <atom:updated>2026-07-06T14:36:31.094Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong><em>The Ultimate Cosmic Catch-22: Why Physics Needs the Mind to Make Sense of Matter.</em></strong></h4><h4>Introduction:</h4><p>Among the deepest questions confronting modern science lies a paradox that refuses to disappear despite a century of extraordinary progress in quantum theory:</p><blockquote>Quantum mechanics contains no mathematical equation for consciousness, yet consciousness appears inseparable from our interpretation of quantum reality.</blockquote><p>This statement is not an established scientific fact — it reflects one of several philosophical interpretations of quantum mechanics — but it exposes one of the most profound conceptual tensions in physics.</p><p>Quantum mechanics has transformed civilization. It explains the behavior of atoms, predicts chemical reactions, powers lasers, semiconductors, MRI scanners, and quantum computers with astonishing precision. Its mathematical framework is arguably the most successful theory ever created.</p><p>Yet beneath this extraordinary predictive success lies an unsettling mystery.</p><p>The equations describe how probabilities evolve with exquisite accuracy, but they remain remarkably silent about one deceptively simple question:</p><h3>The Measurement Problem: Nature’s Stage Fright</h3><p>To understand why this paradox exists, we have to look at the foundational rules of quantum physics.</p><p>In the macroscopic world — the world of coffee mugs, cars, and planets — things exist in definite places at definite times. But down in the subatomic realm, particles behave like ghosts. Before we look at an electron, it doesn’t exist in a single spot; it exists as a wave of mathematical probabilities, described perfectly by the Schrödinger equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*Mzd8cL1dS2iPKaOByz6Ghw.png" /></figure><p>This equation has survived every experimental test with extraordinary accuracy.</p><p>It predicts interference.</p><p>It predicts tunneling.</p><p>It predicts atomic structure.</p><p>It predicts superconductivity.</p><p>Yet despite its immense power, it leaves one fundamental question unanswered.</p><p>The equation continuously evolves probabilities — but nowhere does it specify how one particular possibility becomes the unique reality we eventually observe.</p><p>When a measurement occurs, the wave function appears to “collapse” from many possibilities into a single observed outcome.</p><p>This transition is not described by the Schrödinger equation itself.</p><p>It is simply introduced as a separate postulate.</p><p>This discontinuity is known as <strong>the Measurement Problem</strong>, and it remains one of the deepest unresolved conceptual questions in modern physics.</p><p>This equation can track how those probabilities evolve over time with pinpoint accuracy. But it has a fatal flaw: <strong>it never dictates how a probability becomes a reality.</strong> The moment we measure that electron, the wave of probabilities instantly “collapses” into a single, concrete point.</p><blockquote><strong><em>The Big Question:</em></strong><em> What constitutes a “measurement”? Is it a Geiger counter? A camera? Or does it require a conscious mind to look at the results and say, </em>“Ah, there it is”<em>?</em></blockquote><h4><strong>How does possibility become actuality?</strong></h4><p>At that boundary — where mathematics meets experience — the observer unexpectedly enters the story.</p><p>For over a century, physicists, philosophers, neuroscientists, and mathematicians have wrestled with this puzzle. Despite thousands of papers and numerous interpretations, there is still no universally accepted explanation.</p><p>This is why consciousness remains one of the last great frontiers — not merely of neuroscience or philosophy, but potentially of fundamental physics itself</p><h3>Why Consciousness “Makes Sense” of the Math</h3><p>Early pioneers of quantum mechanics, like <strong>John von Neumann</strong> and <strong>Eugene Wigner</strong>, realized that if you follow the mathematics strictly, physical instruments are <em>also</em> made of atoms. Therefore, the camera or the Geiger counter should also exist in a blur of quantum probabilities.</p><p>Following this logic, the entire universe remains a ghost world of probabilities until the chain finally hits something that is inherently <em>not</em> a mechanical quantum object: <strong>a conscious mind.</strong> In this interpretation (often called the von Neumann-Wigner interpretation), consciousness is the ultimate catalyst. It is the only thing that can bridge the gap between abstract mathematical possibilities and the hard, physical reality we experience daily. Without an observer to make sense of the data, the universe remains an unrendered video game waiting for a player to join.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*obXlj6QygPdoo-uH-Tjuaw.png" /></figure><h3>What Exactly Counts as a Measurement?</h3><p>The natural next question seems obvious.</p><p>What actually constitutes a measurement?</p><p>Is it:</p><ul><li>a photon striking a detector?</li><li>a Geiger counter registering radiation?</li><li>a CCD camera recording an image?</li><li>a computer storing information?</li><li>or does measurement become complete only when a conscious observer becomes aware of the outcome?</li></ul><p>Surprisingly, quantum theory itself does not provide an unambiguous answer.</p><p>The mathematics predicts probabilities with astonishing precision, but it remains silent about where, when, or why observation transitions into experienced reality.</p><p>That silence has inspired some of the most fascinating ideas in twentieth-century science.</p><h3>The Observer Enters the Equation</h3><p>Early pioneers including John von Neumann and Eugene Wigner explored an extraordinary possibility.</p><p>Suppose we examine the measurement process step by step.</p><p>An electron interacts with a detector.</p><p>The detector interacts with electronic circuits.</p><p>The circuits influence a computer.</p><p>The computer stores data.</p><p>The data appears on a screen.</p><p>A scientist observes the screen.</p><p>Every physical component in this chain is itself composed of atoms governed by quantum mechanics.</p><p>If each component obeys the same quantum laws, then each should also evolve into a quantum superposition.</p><p>Nothing within the chain seems capable of terminating the process.</p><p>The question naturally arises:</p><p><strong>Where does the chain actually end?</strong></p><p>Von Neumann suggested that the chain could continue indefinitely until it encounters something fundamentally different from ordinary physical systems:</p><p><strong>a conscious observer.</strong></p><p>In what later became known as the <strong>von Neumann–Wigner interpretation</strong>, consciousness was proposed — not as another quantum object — but as the event that transforms mathematical possibilities into experienced reality.</p><p>Whether this interpretation is correct remains highly controversial, but it continues to stimulate philosophical debate because it directly confronts the observer’s role instead of attempting to remove it.</p><h3>Consciousness as the Bridge Between Mathematics and Experience</h3><p>This perspective proposes something remarkably unusual.</p><p>Quantum mechanics may successfully describe every possible outcome.</p><p>Consciousness, however, is where one outcome becomes an experience.</p><p>The distinction is subtle but profound.</p><p>Physics can calculate probabilities.</p><p>It can predict amplitudes.</p><p>It can describe interactions.</p><p>But the equations themselves do not explain why one specific reality is experienced rather than another.</p><p>In this philosophical view, consciousness is not merely another physical process occurring inside the universe.</p><p>It becomes the interface through which abstract mathematical descriptions acquire experiential meaning.</p><p>Whether this reflects physical reality or simply the limits of current theory remains an open question.</p><h3>Why Many Physicists Resist This Idea</h3><p>Not surprisingly, introducing consciousness into fundamental physics has generated considerable discomfort.</p><p>Science has traditionally sought explanations that are objective, measurable, and reproducible.</p><p>Consciousness appears to resist all three.</p><p>Unlike mass or charge, consciousness has:</p><ul><li>no universally accepted definition,</li><li>no agreed physical unit,</li><li>no governing equation,</li><li>and no direct method of experimental measurement.</li></ul><p>Even Albert Einstein expressed skepticism toward interpretations that appeared to give observation a privileged role, famously asking whether the Moon ceases to exist when no one is looking.</p><p>His objection reflected a broader philosophical commitment to an objective reality existing independently of observers.</p><p>That commitment continues to influence much of contemporary physics.</p><h3>Alternative Interpretations Without Consciousness</h3><p>Because of these conceptual challenges, physicists have developed numerous interpretations that attempt to explain quantum phenomena without assigning a special role to conscious observers.</p><p>Among the most influential are:</p><h3>The Many-Worlds Interpretation</h3><p>Rather than collapsing, the wave function continues evolving indefinitely.</p><p>Every possible outcome is realized, each within a different branch of an ever-expanding multiverse.</p><p>Observation merely identifies which branch an observer occupies.</p><p>No collapse occurs.</p><p>No special role for consciousness is required.</p><p>The cost is accepting an unimaginably vast number of parallel realities.</p><h3>Decoherence Theory</h3><p>Another influential approach argues that interactions with the surrounding environment rapidly destroy observable quantum coherence.</p><p>Macroscopic objects therefore appear classical long before human observation occurs.</p><p>Decoherence explains why quantum behavior becomes effectively hidden at everyday scales.</p><p>However, many researchers note that while decoherence explains why interference disappears, it does not fully resolve why a single definite outcome is experienced instead of multiple possibilities.</p><h3>Objective Collapse Models</h3><p>Some physicists propose that wave functions collapse spontaneously according to new physical laws, independent of observation.</p><p>These models attempt to replace the measurement postulate with experimentally testable mechanisms.</p><p>If confirmed, they would eliminate the need for consciousness in explaining collapse.</p><p>To date, decisive experimental evidence remains elusive.</p><h3>The Deeper Philosophical Challenge</h3><p>Regardless of which interpretation ultimately proves correct, one remarkable fact remains.</p><p>Quantum mechanics is extraordinarily successful at predicting observations.</p><p>Yet it remains surprisingly reticent about the nature of observation itself.</p><p>This raises an intriguing philosophical possibility.</p><p>Perhaps consciousness is not simply another object waiting to be explained.</p><p>Perhaps it represents an entirely different category of phenomenon — one that cannot be fully understood using concepts originally developed to describe matter alone.</p><p>If that is true, then physics and consciousness research may eventually require a common conceptual framework extending beyond both current quantum theory and current neuroscience.</p><h3>A Contemplative Perspective</h3><p>There is a profound irony at the heart of modern science.</p><p>Humanity set out to understand the universe as an objective machine governed by elegant mathematical laws.</p><p>The deeper we explored, the more those laws revealed extraordinary precision.</p><p>Yet at the smallest scales, the theory itself seems to gesture back toward the observer asking the questions.</p><p>Whether consciousness actively influences physical reality or merely accompanies it remains unresolved.</p><p>Either way, every scientific theory is ultimately formulated, interpreted, and understood through conscious experience.</p><p>The observer can be mathematically postponed, philosophically minimized, or experimentally abstracted — but never entirely eliminated.</p><h3>Conclusion: Beyond the Quantum Machine</h3><p>Quantum mechanics has revealed a universe far stranger than classical intuition ever imagined. It describes a reality where certainty gives way to probability, where particles behave like waves, and where observation occupies a uniquely enigmatic position. Yet despite its extraordinary predictive power, the theory does not provide a universally accepted explanation for how potential outcomes become the singular reality we experience.</p><p>Consciousness remains equally mysterious. We still lack a comprehensive scientific theory explaining subjective awareness, and there is no consensus that consciousness causes wave-function collapse. At the same time, no interpretation has completely dissolved the conceptual puzzles surrounding measurement and observation. The result is not evidence that consciousness governs the universe, but rather a reminder that our deepest theories remain incomplete.</p><p>Perhaps the next revolution in physics will emerge not from building larger particle accelerators or more powerful quantum computers alone, but from developing a richer understanding of the relationship between information, observation, and conscious experience. It may require ideas that bridge quantum theory, neuroscience, information theory, mathematics, and philosophy into a unified framework.</p><p>Until then, consciousness stands as one of science’s most compelling frontiers — not because it has already been proven to shape reality, but because it exposes the limits of our current understanding. The greatest mystery may not be hidden in distant galaxies or inside exotic particles. It may reside in the very awareness through which every experiment is conceived, every equation is interpreted, and every scientific discovery becomes meaningful.</p><p>In the end, the quest is not merely to understand the universe. It is to understand the observer capable of asking what the universe truly is.</p><h4>Before you go</h4><p><em>Thousands of developers share what they’re building, learning, and discovering across our publications every month. One account connects you to our entire network of publications and communities.</em></p><p><a href="https://plainenglish.io/write-for-us"><strong><em>Explore more at plainenglish.io</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=273a10a9b126" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/beyond-the-quantum-machine-why-consciousness-remains-the-final-frontier-of-physics-273a10a9b126">Beyond the Quantum Machine: Why Consciousness Remains the Final Frontier of Physics.</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet ThunderID—WSO2’s Radical Blueprint for Next-Gen Identity]]></title>
            <link>https://medium.com/codetodeploy/meet-thunderid-wso2s-radical-blueprint-for-next-gen-identity-f3f0f01500ca?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3f0f01500ca</guid>
            <category><![CDATA[identity-platform]]></category>
            <category><![CDATA[thunderid]]></category>
            <category><![CDATA[wso2]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Fri, 03 Jul 2026 04:53:36 GMT</pubDate>
            <atom:updated>2026-07-06T10:13:23.945Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>The Next Frontier of IAM: Securing AI and Quantum Eras with WSO2’s ThunderID</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hn_D3Y4NGwPsNKEKVhfcYA.png" /><figcaption>ThunderID — WSO2’s Radical Blueprint for Next-Gen Identity</figcaption></figure><h4>Introduction:</h4><p><a href="https://wso2.com/library/conference/2026/05/meet-thunderid-an-identity-core-for-the-future"><strong>ThunderID</strong> </a>is a high-performance, open-source identity stack built from the ground up for modern development teams. It redefines access management by treating <strong>humans, AI agents, and programmatic workloads</strong> as first-class citizens.</p><p>With ThunderID, you aren’t locked into rigid authentication structures. Instead, you can build, scale, and secure your ecosystem using <strong>fully composable identity flows</strong> that adapt to any architecture without sacrificing speed or security. The ground beneath Identity and Access Management (IAM) is shifting faster than traditional infrastructure can handle. We aren’t just authenticating humans clicking buttons anymore; we are managing autonomous AI agents spinning up API calls, securing systems against the looming specter of post-quantum decryption, and shifting toward user-centric decentralized wallets.</p><blockquote><em>💥 </em><strong><em>Master Any Skills in 3 Months</em></strong><br> 📚 Up to 50% OFF Premium Courses<br> ⏳ Limited-Time Offer<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><p>Recognizing that tomorrow’s threats can&#39;t be solved by throwing patches at yesterday’s foundations, WSO2 has taken a bold step. Pulling from their Asgardio heritage, they have engineered a ground-up rewrite of their identity core.</p><p>Enter ThunderID—an open-source, ultra-lightweight, and blazing-fast engine built completely in Go</p><h4>The Pillars of Tomorrow&#39;s Identity</h4><p>ThunderID isn&#39;t just an evolutionary update; it is a architectural rebellion against bloated identity stacks. It anchors itself on four fundamental shifts:</p><p>Agent-Native Identity: AI agents aren’t simple service accounts, nor are they standard human users . They require distinct identities anchored cryptographically, utilizing time-bound, permission-bound scopes that respect strict user consent ThunderID builds tracing directly into its token architecture to monitor what agents execute across your infrastructure.</p><p>Post-Quantum Safety by Design: The threat of &quot;harvest now, decrypt later&quot; is a looming reality ThunderID bakes crypto-agility directly into its runtime . Rather than hardcoding algorithms, its entire configuration is fully agile, defaulting to NIST-approved post-quantum algorithms while supporting a hybrid coexistence mode so enterprises can migrate without sudden disruptions</p><p>Decentralized Identity (DID): With the EU’s eIDAS and global digital wallet movements picking up massive speed, identity is returning to the hands of users Built heavily on OpenID standards, ThunderID is engineered to issue verifiable credentials and seamlessly integrate decentralized identities into standard enterprise registration journeys .</p><p>Lightweight, Headless Performance: Stripped of unnecessary legacy weight, ThunderID’s entire headless runtime occupies a microscopic 35MB memory footprint (around 25MB without UI elements) and starts up in sub-second times (400–500 milliseconds). Running entirely stateless by design, it allows organizations to scale up to millions of users in high-density cloud-native environments at a fraction of standard infrastructure costs.</p><h4>Built for the Modern Developer Ecosystem</h4><p>WSO2 understood that developers do not want to constantly write custom code; they want to configure flows . ThunderID delivers a visual, no-code GUI builder for drag-and-drop orchestration of user journeys—such as implementing Passkeys instantly ,. It prioritizes GitOps, enabling full declarative imports and exports of identity configurations so your team can easily track changes and run continuous integration pipelines .</p><p>Even more uniquely, it packs a native Model Context Protocol (MCP) server directly into the runtime. This means your AI coding co-pilots can actively use the system’s tools to build, register, and safely embed registration and login journeys straight into your codebase.</p><h4>Where is ThunderID Heading?</h4><p>ThunderID isn’t just a concept—it is already the foundation for massive real-world projects like the MOSIP network’s eSignet 2.0, powering national-level digital identity infrastructure across dozens of countries With its 1.0 release slated for July 2026, WSO2 is also working to donate ThunderID to the Open Wallet Foundation to champion global decentralized ecosystems,</p><p>For existing WSO2 Identity Server users, fear not: WSO2 will continue to maintain and evolve the standard identity server with no forced migrationsBut for greenfield projects, agentic workflows, and cloud-native scaling, ThunderID is the fast, secure, and open-source lightning strike the industry has been waiting for.</p><p>&quot;The landscape of digital identity is fractured. Traditional access management systems were built for humans typing passwords. But today? We are dealing with autonomous AI agents triggering APIs, the impending threat of quantum computing cracks, and a global shift toward decentralized citizen wallets. Enter WSO2’s answer to the future: ThunderID.&quot;</p><h4>What IS ThunderID?</h4><p>&quot;Born out of WSO2’s Asgardio heritage, ThunderID is a ground-up rewrite of the identity core engineered completely in Go. It’s ultra-lightweight, blazing-fast, and designed to be open-source under the Apache 2.0 license. We’re talking about a microscopic 35-megabyte runtime footprint that starts up in under half a second. It strips out the bloat to give you a highly scalable, headless-first architecture built for high-density cloud deployments.&quot;</p><h4>KEY FEATURES</h4><blockquote>&quot;ThunderID delivers on three major fronts. First, it is Agent-Native. It recognizes AI agents as distinct identities, locking them down with cryptographically anchored credentials and user-consented, time-bound privileges.</blockquote><blockquote>Second, it brings Post-Quantum Security by design. With its crypto-agile framework, it uses NIST-approved algorithms to guard against &#39;harvest now, decrypt later&#39; threats, allowing classic and quantum proof algorithms to smoothly coexist during migration.</blockquote><blockquote>And third, it embraces Decentralized Identity, allowing companies to issue verifiable credentials and hook directly into global digital identity wallets.&quot;</blockquote><h4>ThunderId Simulation Engine Using Go Language</h4><pre>package main<br><br>import (<br> &quot;context&quot;<br> &quot;crypto/ed25519&quot;<br> &quot;crypto/rand&quot;<br> &quot;errors&quot;<br> &quot;fmt&quot;<br> &quot;log&quot;<br> &quot;sync&quot;<br> &quot;time&quot;<br>)<br><br>// IdentityType defines the category of entity requesting access.<br>type IdentityType string<br><br>const (<br> IdentityHuman IdentityType = &quot;HUMAN&quot;<br> IdentityAgent IdentityType = &quot;AI_AGENT&quot; // First-class Agent Identity [00:09:57]<br>)<br><br>// CryptoAlgorithm models ThunderID&#39;s crypto-agility.<br>type CryptoAlgorithm string<br><br>const (<br> AlgClassicEd25519 CryptoAlgorithm = &quot;Ed25519&quot;<br> AlgPostQuantumMLDSA CryptoAlgorithm = &quot;ML-DSA-65&quot; // NIST-approved PQC algorithm [00:12:24]<br>)<br><br>// Identity represents the actor trying to authenticate.<br>type Identity struct {<br> ID        string<br> Type      IdentityType<br> PublicKey []byte<br> Alg       CryptoAlgorithm<br>}<br><br>// AuthContext captures runtime metadata for flow orchestration [00:18:40].<br>type AuthContext struct {<br> ID         string<br> Subject    Identity<br> Scopes     []string<br> OnBehalfOf string // Tracing delegation for AI Agents [00:11:01]<br> CreatedAt  time.Time<br>}<br><br>// Token represents a stateless, cryptographically anchored token issued by the Gateway [00:10:26], [00:17:01].<br>type Token struct {<br> TokenString string<br> Algorithm   CryptoAlgorithm<br> ExpiresAt   time.Time<br>}<br><br>// FlowOrchestrator manages high-throughput identity evaluations.<br>type FlowOrchestrator struct {<br> mu            sync.RWMutex<br> activeFlows   map[string]*AuthContext<br> tokenPool     sync.Pool<br> cryptoSetting CryptoAlgorithm<br>}<br><br>// NewFlowOrchestrator instantiates a highly scalable, stateless platform service.<br>func NewFlowOrchestrator(defaultAlg CryptoAlgorithm) *FlowOrchestrator {<br> return &amp;FlowOrchestrator{<br>  activeFlows:   make(map[string]*AuthContext),<br>  cryptoSetting: defaultAlg,<br>  tokenPool: sync.Pool{<br>   New: func() interface{} {<br>    return &amp;Token{}<br>   },<br>  },<br> }<br>}<br><br>// Authenticate executes an atomic identity evaluation pipeline.<br>func (fo *FlowOrchestrator) Authenticate(ctx context.Context, authCtx *AuthContext) (*Token, error) {<br> // Enforce context deadlines to maintain sub-millisecond SLAs [00:23:53]<br> select {<br> case &lt;-ctx.Done():<br>  return nil, ctx.Err()<br> default:<br> }<br><br> // Step 1: Evaluate Identity Primitives<br> if authCtx.Subject.ID == &quot;&quot; {<br>  return nil, errors.New(&quot;malformed identity: subject ID missing&quot;)<br> }<br><br> // Step 2: Handle Agent Identity Scopes &amp; Traceability Natively [00:11:01]<br> if authCtx.Subject.Type == IdentityAgent {<br>  if authCtx.OnBehalfOf == &quot;&quot; {<br>   return nil, errors.New(&quot;agent identity execution rejected: missing human user delegation anchor&quot;)<br>  }<br>  log.Printf(&quot;[ThunderID Engine] Tracing Agent Action: ID %s acting on behalf of %s\n&quot;, authCtx.Subject.ID, authCtx.OnBehalfOf)<br> }<br><br> // Step 3: Crypto-Agility Validation Step [00:11:57]<br> fo.mu.RLock()<br> currentAlg := fo.cryptoSetting<br> fo.mu.RUnlock()<br><br> log.Printf(&quot;[Gateway Runtime] Validating signatures utilizing %s pipeline...\n&quot;, currentAlg)<br><br> // Step 4: Token Generation utilizing sync.Pool for low memory allocation footprints [00:15:52]<br> tkn := fo.tokenPool.Get().(*Token)<br> tkn.Algorithm = currentAlg<br> tkn.ExpiresAt = time.Now().Add(1 * time.Hour)<br> tkn.TokenString = fmt.Sprintf(&quot;tndr_jwt.%s.%d&quot;, authCtx.Subject.ID, time.Now().UnixNano())<br><br> return tkn, nil<br>}<br><br>// UpdateCryptoAlgorithm hot-swaps cryptographic suites without service restarts.<br>func (fo *FlowOrchestrator) UpdateCryptoAlgorithm(newAlg CryptoAlgorithm) {<br> fo.mu.Lock()<br> defer fo.mu.Unlock()<br> log.Printf(&quot;[Control Plane] Hot-swapping cryptographic suite to: %s\n&quot;, newAlg)<br> fo.cryptoSetting = newAlg<br>}<br><br>func main() {<br> log.Println(&quot;⚡ Initializing ThunderID Gateway Runtime...&quot;)<br> start := time.Now()<br><br> // 1. Simulate the sub-second fast boot time of Go runtime [00:23:15]<br> time.Sleep(15 * time.Millisecond) <br> log.Printf(&quot;⚡ ThunderID started successfully in %v (Headless Mode Enabled)\n&quot;, time.Since(start))<br><br> // 2. Mock crypto-anchored hardware keys [00:10:26]<br> pubKey, _, err := ed25519.GenerateKey(rand.Reader)<br> if err != nil {<br>  log.Fatalf(&quot;Failed to generate hardware keys: %v&quot;, err)<br> }<br><br> // 3. Initialize Engine with default configuration<br> engine := NewFlowOrchestrator(AlgClassicEd25519)<br><br> // 4. Create an Autonomous AI Agent Identity Profile [00:09:57]<br> aiAgentContext := &amp;AuthContext{<br>  ID: &quot;flow_ctx_001&quot;,<br>  Subject: Identity{<br>   ID:        &quot;Customer-Concierge-AI&quot;,<br>   Type:      IdentityAgent,<br>   PublicKey: pubKey,<br>   Alg:       AlgClassicEd25519,<br>  },<br>  Scopes:     []string{&quot;flights:book&quot;, &quot;profile:read&quot;},<br>  OnBehalfOf: &quot;User_404_Passenger&quot;, // Cryptographically delegated user anchor [00:10:17]<br>  CreatedAt:  time.Now(),<br> }<br><br> // 5. Execute Concurrent Authentication Streams to benchmark performance<br> var wg sync.WaitGroup<br> ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)<br> defer cancel()<br><br> log.Println(&quot;\n--- Scenario 1: Executing Agent-Native Auth Pipelines ---&quot;)<br> wg.Add(1)<br> go func() {<br>  defer wg.Done()<br>  token, err := engine.Authenticate(ctx, aiAgentContext)<br>  if err != nil {<br>   log.Printf(&quot;Authentication failed: %v\n&quot;, err)<br>   return<br>  }<br>  log.Printf(&quot;SUCCESS: Issued Token -&gt; %s (Valid Until: %s)\n&quot;, token.TokenString, token.ExpiresAt.Format(time.Kitchen))<br> }()<br> wg.Wait()<br><br> // 6. Demonstrate Crypto-Agility: Shift to Post-Quantum Era on the fly [00:11:57]<br> log.Println(&quot;\n--- Scenario 2: Migrating Core to Post-Quantum Security ---&quot;)<br> engine.UpdateCryptoAlgorithm(AlgPostQuantumMLDSA)<br><br> wg.Add(1)<br> go func() {<br>  defer wg.Done()<br>  token, err := engine.Authenticate(ctx, aiAgentContext)<br>  if err != nil {<br>   log.Printf(&quot;Authentication failed: %v\n&quot;, err)<br>   return<br>  }<br>  log.Printf(&quot;SUCCESS: Issued PQC Token -&gt; %s [Algorithm: %s]\n&quot;, token.TokenString, token.Algorithm)<br> }()<br> wg.Wait()<br>}</pre><h4>THE DEVELOPER EXPERIENCE &amp; CONCLUSION</h4><p>For developers, ThunderID turns customization into configuration. It features a stunning visual drag-and-drop GUI builder for user journeys, native GitOps support, and a built-in Model Context Protocol server. This allows AI co-pilots to literally write your identity integration for you. Already powering national digital IDs through the MOSIP network, ThunderID is dropping its 1.0 release in July 2026. Step into the future of identity. Open, fast, and ready for what’s next.&quot;</p><p>References</p><ol><li><a href="https://github.com/thunder-id/thunderid">https://github.com/thunder-id/thunderid</a></li></ol><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/bhuwan-kc-583505319/?isSelfProfile=true"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3f0f01500ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/meet-thunderid-wso2s-radical-blueprint-for-next-gen-identity-f3f0f01500ca">Meet ThunderID—WSO2’s Radical Blueprint for Next-Gen Identity</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracing Structural Emergence: A Lens on End-to-End Gradient Descent in Deep Networks]]></title>
            <link>https://medium.com/codetodeploy/tracing-structural-emergence-a-lens-on-end-to-end-gradient-descent-in-deep-networks-6bdab982fac9?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bdab982fac9</guid>
            <category><![CDATA[aws-sagemaker]]></category>
            <category><![CDATA[end-to-end-gradient]]></category>
            <category><![CDATA[deeplinearlygated-network]]></category>
            <category><![CDATA[relu-network]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 10:47:09 GMT</pubDate>
            <atom:updated>2026-07-01T11:03:25.575Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xHVUdxI_bGSXoeihbNHv8A.png" /><figcaption>Where Every Gradient Shapes a Lens, Every Lens Reveals Structure, and Every Layer Brings Intelligence to Life.</figcaption></figure><h4>Introduction :</h4><p>The Intent and Objective of the article is to provide End-to-end gradient-based optimization of multi-layer neural architectures stands as a defining empirical triumph of modern machine learning. Across diverse domains — ranging from computer vision and natural language processing to complex scientific discovery — deep networks trained via backpropagation routinely extract intricate, task-relevant abstractions directly from high-dimensional data, largely rendering classical hand-engineered feature pipelines obsolete.</p><blockquote><em>💥 </em><strong><em>Save Up to 85% OFF</em></strong><br> 📚 Premium Courses <br> ⏳ Limited-Time Offer | 🎓 Upgrade Your Skills<br><a href="https://trk.udemy.com/zz4NBO"><em>👉 </em><strong><em>Enroll Now &amp; Start Learning</em></strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><p>Yet, beneath this profound empirical success lies a persistent theoretical void: a foundational, mechanistic understanding of the <em>why</em>, <em>when</em>, and <em>how</em> of deep learning remains elusive. While classical statistical learning theory often struggles to explain the generalization capabilities of overparameterized models, the core mysteries of deep feature learning can be distilled into three fundamental questions:</p><ul><li><strong>The Mechanistic “Why” (Optimization &amp; Geometry):</strong> Why does gradient descent reliably navigate non-convex loss landscapes to uncover highly structured, hierarchical representations, rather than becoming trapped in pathological local minima or overfitting to stochastic noise?</li><li><strong>The Regime “When” (Feature Learning vs. Memorization):</strong> At what thresholds of data scale, model capacity, and initialization does a network transition from mere kernel-like memorization (e.g., the Lazy Training/Neural Tangent Kernel regime) into a dynamic, rich feature-learning regime?</li><li><strong>The Structural “How” (Inter-layer Coordination):</strong> How do individual layers coordinate across deep hierarchies to build cohesive, compositional representations, and what principles govern the efficient propagation of representation gradients throughout the network?</li></ul><p>Existing perspectives — representational power of deep architectures, input-space folding, optimization landscape conditioning, the neural tangent kernel (NTK) regime, and hierarchical feature learning — each illuminate important facets. However, they often fall short in explaining the interactive, co-adaptivedynamics that drive feature emergence across layers. In this article, we explore a fresh narrative centered on gated deep networks, where layers function as adaptive “lenses” that importance-weight the input for one another. This viewpoint, supported by theoretical analysis and experiments on the Deep Linearly Gated Network (DLGN), reveals a powerful positive-feedback mechanism underlying feature learning. We will journey through the ideas, mathematics, empirics, and real-world implications, weaving in contemplative reflections on what this means for understanding intelligence itself.</p><h4>The Puzzle of Feature Learning</h4><p>Deep networks succeed because they learn *features* — task-relevant patterns or abstractions — rather than treating inputs as fixed vectors. In the lazy training/NTK regime, networks behave like kernel methods with fixed random features, achieving good performance but limited adaptability. In the rich/feature-learning regime (typical with larger learning rates, appropriate initialization, or specific architectures), parameters evolve significantly, allowing the network to discover and refine representations dynamically.</p><h4>Traditional views highlight:</h4><blockquote>Depth and compositionality: Deeper networks can represent more complex functions.<br>Optimization benefits: Depth can improve conditioning in certain cases.<br>Hierarchical learning: Lower layers capture edges/textures; higher layers capture objects/concepts.</blockquote><p>These are insightful but often descriptive. They do not fully explain the cooperative dynamics between layers during training. How does a layer “know” which features to prioritize when the rest of the network is also changing?</p><h4>The Lens Hypothesis: Layers as Adaptive Importance Weighters</h4><p>Consider a gated deep network (e.g., with ReLU or similar piecewise-linear activations). From the perspective of any single layer, the preceding and subsequent layers collectively act as a **lens**. This lens modulates the input data by assigning *importance weights* or effective attention to different regions or directions in the input space. Features that might be subtle or localized — those lacking global scope — are amplified or highlighted through this modulation, making them easier for the current layer to detect and specialize in.</p><p>Gradient descent then updates the parameters of each layer to better fit the importance-weighted version of the data as seen through the lenses formed by the other layers. Crucially, each neuron’s parameters play a dual role:<br>1. As a separatoror classifier in the space lensed by the rest of the network.<br>2. As a component of the lensthat influences how other layers perceive the data.</p><p>This duality creates a positive feedback loop: Better lenses lead to better separators, which in turn refine the lenses further. Over time, this bootstrapping process drives the emergence of hierarchical, task-relevant structure.</p><p>Contemplative aside: This lens metaphor evokes optical systems or perceptual psychology. Just as the human visual cortex has reciprocal connections and top-down modulation that sharpen attention to salient features, deep networks develop an internal “attention economy” through gradients. It suggests feature learning is not purely bottom-up nor top-down, but deeply interdependent — a distributed computation where no single layer has global responsibility.</p><h4>Deep Linearly Gated Networks (DLGNs) as a Testbed</h4><p>To make this concrete, consider the Deep Linearly Gated Network (DLGN), an architecture that blends deep linear networks with ReLU-like gating. In DLGNs, each layer applies linear transformations followed by gating mechanisms that create piecewise-linear decision boundaries, but with a structure that facilitates exact analysis.</p><blockquote>Theoretical analysis of DLGNs under gradient descent yields striking predictions:</blockquote><blockquote>Control over trajectories from initialization: The dynamics can be characterized such that the evolution of weights aligns predictably with the lens-induced importance weighting. Starting from small random initialization, gradients steer layers toward specializing in progressively more abstract features.</blockquote><blockquote>Hierarchical feature discovery: Early layers tend to capture broad, low-frequency patterns (acting as coarse lenses), while deeper layers refine these into sharper, higher-order features. The positive feedback accelerates this hierarchy formation in the feature-learning regime.</blockquote><p>Mathematically, one can derive how the effective input to a layer \( l \) is transformed by the product of weight matrices and gating functions from other layers, leading to a reweighted loss landscape. Updates to layer \( l \) minimize a loss on this reweighted data, and because of the dual role, improvements propagate bidirectionally.</p><p>Empirical validation on DLGNs confirms these predictions. Training trajectories match theoretical forecasts remarkably well, with visualizations showing clear progression from generic to specialized features. Comparisons with standard ReLU networks reveal qualitative similarities in feature emergence, suggesting the lens mechanism is not an artifact of the simplified DLGN but a broader principle.</p><h4>Real-Time Applications and Use Cases</h4><p>This narrative has profound practical implications beyond theory:</p><p>1. Interpretability and Debugging:<br>By modeling layers as lenses, we can develop tools to visualize “importance-weighted” inputs for each layer. For instance, in computer vision, saliency maps or activation atlases could be augmented with lens-effect decompositions to trace how early layers bias later ones. This aids debugging misclassifications in medical imaging (e.g., why a model overlooks subtle tumors) or autonomous driving (detecting edge cases in rare weather).</p><p>2. Efficient Training and Architecture Design:<br>Understanding positive feedback suggests targeted regularization or initialization schemes that amplify lens formation early on. In resource-constrained settings (mobile AI, edge devices), architectures could explicitly incorporate gating to promote faster hierarchical learning, reducing training time and energy use.<br>For large language models, the lens view reframes attention mechanisms: Multi-head attention already resembles dynamic lensing; deeper integration of this principle could improve long-context reasoning.</p><p>3. Transfer Learning and Few-Shot Adaptation:<br>Pre-trained lenses (from a source domain) can be fine-tuned to reweight data in a target domain. This explains why foundation models adapt quickly — existing hierarchical features act as powerful initial lenses. Applications include rapid customization for personalized medicine or low-resource languages.</p><p>4. Scientific Discovery:<br>In physics or biology, where data is noisy and structure is hierarchical (e.g., particle interactions or protein folding), the lens mechanism helps models discover latent variables without exhaustive search. Real-time use cases: accelerating simulation surrogate models or analyzing genomic sequences by iteratively refining importance weights on variants.</p><p>5. Robustness and Safety:<br>Positive feedback can amplify both good and bad structures. By monitoring lens stability during training, we can detect adversarial vulnerabilities or dataset biases early. Contemplatively, this raises questions about AI alignment: How do we ensure the lenses align with human values rather than spurious correlations?</p><p>Case Study Sketch: Imagine training a DLGN-inspired model for climate modeling. Raw satellite data is high-dimensional and noisy. Early layers learn broad seasonal lenses, weighting atmospheric patterns; deeper layers specialize in extreme events (hurricanes, heatwaves). Gradients refine these interactively, leading to more accurate forecasts than purely linear or kernel-based methods.</p><h4>Python program for Advanced Deep Linearly Gated Network Implementation with Keras/TensorFlow, Lens Mechanism Analysis, and AWS Integration.</h4><pre><br>import os<br>import json<br>import boto3<br>import numpy as np<br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras import layers, mixed_precision<br>import matplotlib.pyplot as plt<br>from sklearn.datasets import fetch_openml<br>from sklearn.model_selection import train_test_split<br><br># ==============================<br># AWS Configuration<br># ==============================<br>AWS_REGION = &quot;us-east-1&quot;  # Change as needed<br>S3_BUCKET = &quot;your-dlgn-bucket&quot;  # Create this bucket first<br>s3_client = boto3.client(&quot;s3&quot;, region_name=AWS_REGION)<br><br># For SageMaker (uncomment and configure when running on SageMaker)<br># import sagemaker<br># from sagemaker.tensorflow import TensorFlow<br><br># Enable mixed precision for optimization<br>policy = mixed_precision.Policy(&#39;mixed_float16&#39;)<br>mixed_precision.set_global_policy(policy)<br><br># ==============================<br># Custom Gated Layer (DLGN-inspired)<br># ==============================<br>class GatedLinearLayer(layers.Layer):<br>    &quot;&quot;&quot;<br>    Custom layer simulating DLGN: Linear transformation + gating (ReLU-like)<br>    with built-in lens tracking (activation statistics for importance weighting).<br>    &quot;&quot;&quot;<br>    def __init__(self, units, name=&quot;gated_linear&quot;, **kwargs):<br>        super().__init__(name=name, **kwargs)<br>        self.units = units<br>        self.dense = layers.Dense(units, activation=None, kernel_initializer=&quot;he_normal&quot;)<br>        self.gate = layers.Activation(&quot;relu&quot;)  # Core gating for piecewise linearity<br><br>    def call(self, inputs, training=False):<br>        x = self.dense(inputs)<br>        gated = self.gate(x)<br>        # Lens tracking: Record mean activation as proxy for importance weight<br>        if training:<br>            self.add_metric(tf.reduce_mean(gated), name=f&quot;{self.name}_lens_mean&quot;)<br>        return gated<br><br>    def get_config(self):<br>        config = super().get_config()<br>        config.update({&quot;units&quot;: self.units})<br>        return config<br><br><br># ==============================<br># DLGN Model Builder<br># ==============================<br>def build_dlgn_model(input_shape, num_classes=10, depth=4, width=128):<br>    &quot;&quot;&quot;Build hierarchical DLGN with multiple gated layers.&quot;&quot;&quot;<br>    inputs = keras.Input(shape=input_shape, name=&quot;input&quot;)<br>    x = layers.Flatten()(inputs) if len(input_shape) &gt; 1 else inputs<br><br>    # Progressive hierarchical layers (simulating lens refinement)<br>    for i in range(depth):<br>        x = GatedLinearLayer(width, name=f&quot;gated_layer_{i}&quot;)(x)<br>        # Optional: Add residual for better gradient flow (optimization)<br>        if i &gt; 0:<br>            x = layers.Add()([x, layers.Dense(width, activation=None)(x)])<br><br>    outputs = layers.Dense(num_classes, activation=&quot;softmax&quot;, dtype=&quot;float32&quot;)(x)<br>    model = keras.Model(inputs, outputs, name=&quot;DLGN_Lens_Model&quot;)<br><br>    # Compile with optimized optimizer<br>    optimizer = keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)<br>    model.compile(<br>        optimizer=optimizer,<br>        loss=&quot;sparse_categorical_crossentropy&quot;,<br>        metrics=[&quot;accuracy&quot;]<br>    )<br>    return model<br><br><br># ==============================<br># Lens Analysis Callback<br># ==============================<br>class LensAnalysisCallback(keras.callbacks.Callback):<br>    &quot;&quot;&quot;Monitor positive feedback and lens effects during training.&quot;&quot;&quot;<br>    def __init__(self, log_dir=&quot;./lens_logs&quot;):<br>        super().__init__()<br>        self.log_dir = log_dir<br>        os.makedirs(log_dir, exist_ok=True)<br>        self.lens_history = []<br><br>    def on_epoch_end(self, epoch, logs=None):<br>        # Extract lens metrics (importance weights proxy)<br>        layer_metrics = {}<br>        for layer in self.model.layers:<br>            if hasattr(layer, &quot;metrics&quot;) and layer.name.startswith(&quot;gated&quot;):<br>                for m in self.model.metrics:<br>                    if layer.name in m.name:<br>                        layer_metrics[layer.name] = float(m.result().numpy())<br><br>        self.lens_history.append({<br>            &quot;epoch&quot;: epoch,<br>            &quot;lens_means&quot;: layer_metrics,<br>            &quot;val_acc&quot;: logs.get(&quot;val_accuracy&quot;)<br>        })<br><br>        # Save for visualization<br>        with open(f&quot;{self.log_dir}/lens_history_epoch_{epoch}.json&quot;, &quot;w&quot;) as f:<br>            json.dump(self.lens_history[-1], f, indent=2)<br><br>        print(f&quot;Epoch {epoch}: Lens means - {layer_metrics}&quot;)<br><br><br># ==============================<br># Data Pipeline (Optimized)<br># ==============================<br>def load_and_preprocess_data():<br>    &quot;&quot;&quot;Load MNIST (or similar) with tf.data pipeline.&quot;&quot;&quot;<br>    # For demo; replace with your dataset<br>    X, y = fetch_openml(&quot;mnist_784&quot;, version=1, return_X_y=True, as_frame=False)<br>    X = X.astype(&quot;float32&quot;) / 255.0<br>    y = y.astype(&quot;int32&quot;)<br><br>    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)<br><br>    # tf.data for high-performance input pipeline<br>    train_ds = tf.data.Dataset.from_tensor_slices((X_train, y_train)) \<br>        .shuffle(10000).batch(512).prefetch(tf.data.AUTOTUNE)<br>    val_ds = tf.data.Dataset.from_tensor_slices((X_val, y_val)) \<br>        .batch(512).prefetch(tf.data.AUTOTUNE)<br><br>    return train_ds, val_ds, (28, 28, 1)  # MNIST shape<br><br><br># ==============================<br># AWS S3 Utilities<br># ==============================<br>def upload_to_s3(local_path, s3_key):<br>    &quot;&quot;&quot;Upload model/checkpoints to S3.&quot;&quot;&quot;<br>    try:<br>        s3_client.upload_file(local_path, S3_BUCKET, s3_key)<br>        print(f&quot;Uploaded {local_path} to s3://{S3_BUCKET}/{s3_key}&quot;)<br>    except Exception as e:<br>        print(f&quot;S3 upload failed: {e}&quot;)<br><br><br>def download_from_s3(s3_key, local_path):<br>    &quot;&quot;&quot;Download from S3.&quot;&quot;&quot;<br>    try:<br>        s3_client.download_file(S3_BUCKET, s3_key, local_path)<br>        print(f&quot;Downloaded s3://{S3_BUCKET}/{s3_key} to {local_path}&quot;)<br>    except Exception as e:<br>        print(f&quot;S3 download failed: {e}&quot;)<br><br><br># ==============================<br># SageMaker Training Wrapper (for cloud scaling)<br># ==============================<br>def create_sagemaker_estimator(entry_point=&quot;train.py&quot;, role_arn=None):<br>    &quot;&quot;&quot;Example SageMaker TensorFlow estimator setup.&quot;&quot;&quot;<br>    # Requires SageMaker SDK and proper IAM role<br>    # estimator = TensorFlow(<br>    #     entry_point=entry_point,<br>    #     source_dir=&quot;.&quot;,<br>    #     role=role_arn,<br>    #     instance_count=1,<br>    #     instance_type=&quot;ml.p3.2xlarge&quot;,<br>    #     framework_version=&quot;2.12&quot;,<br>    #     py_version=&quot;py310&quot;,<br>    #     distribution={&quot;parameter_server&quot;: {&quot;enabled&quot;: True}},  # or Horovod<br>    #     hyperparameters={&quot;epochs&quot;: 10, &quot;batch_size&quot;: 512}<br>    # )<br>    # return estimator<br>    print(&quot;SageMaker estimator configuration ready (uncomment for use).&quot;)<br>    pass<br><br><br># ==============================<br># Main Training &amp; Analysis<br># ==============================<br>def main():<br>    # Data<br>    train_ds, val_ds, input_shape = load_and_preprocess_data()<br><br>    # Model<br>    model = build_dlgn_model(input_shape)<br>    model.summary()<br><br>    # Callbacks<br>    callbacks = [<br>        keras.callbacks.ModelCheckpoint(&quot;dlgn_best.keras&quot;, save_best_only=True),<br>        keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),<br>        LensAnalysisCallback(),<br>        keras.callbacks.TensorBoard(log_dir=&quot;./tensorboard_logs&quot;)<br>    ]<br><br>    # Train<br>    history = model.fit(<br>        train_ds,<br>        validation_data=val_ds,<br>        epochs=20,<br>        callbacks=callbacks<br>    )<br><br>    # Save and upload to S3<br>    model.save(&quot;dlgn_final.keras&quot;)<br>    upload_to_s3(&quot;dlgn_final.keras&quot;, &quot;models/dlgn_final.keras&quot;)<br><br>    # Visualize Lens Evolution<br>    visualize_lens_evolution(&quot;./lens_logs&quot;)<br><br>    print(&quot;Training completed with lens analysis. Model ready for inference/AWS deployment.&quot;)<br><br><br>def visualize_lens_evolution(log_dir):<br>    &quot;&quot;&quot;Plot lens importance weights over epochs.&quot;&quot;&quot;<br>    # Implementation: aggregate JSON logs and plot<br>    plt.figure(figsize=(12, 6))<br>    # (Add plotting logic based on saved JSONs)<br>    plt.title(&quot;DLGN Lens Importance Evolution (Positive Feedback)&quot;)<br>    plt.xlabel(&quot;Epoch&quot;)<br>    plt.ylabel(&quot;Mean Activation (Importance Proxy)&quot;)<br>    # plt.legend()<br>    plt.savefig(&quot;lens_evolution.png&quot;)<br>    upload_to_s3(&quot;lens_evolution.png&quot;, &quot;visuals/lens_evolution.png&quot;)<br>    print(&quot;Lens visualization saved and uploaded.&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>   <br>    main()<br><br>   </pre><h4>Broader Contemplations on Intelligence</h4><p>This lens-and-feedback perspective invites deeper reflection. Traditional symbolic AI struggled with feature engineering precisely because it lacked this emergent, co-adaptive process. Neural networks succeed by *embodying* computation in parameters that serve multiple roles simultaneously — mirroring how biological neurons participate in many circuits.</p><p>Yet mysteries remain: How does this scale to truly general intelligence? What happens in very deep or very wide regimes? Does the mechanism break under certain distributions (e.g., heavy-tailed data)? The partial extension to ReLU networks is encouraging, hinting at universality, but rigorous proofs across architectures are needed.</p><p>Philosophically, it blurs the line between optimization and learning. Gradient descent is not merely minimizing loss; it is orchestrating a society of lenses that collectively bootstrap understanding from data. This resonates with theories of predictive coding in neuroscience and self-organizing systems in complexity science.</p><h4>Conclusion: Toward a Unified Theory</h4><p>The success of end-to-end training is no longer a black box. By viewing layers as mutual lenses that enable localized feature discovery through positive feedback, we gain predictive power (as in DLGN trajectories) and actionable insights for applications. This narrative complements existing theories while addressing their gaps, particularly the interactive nature of hierarchical learning.</p><p>Future work should extend the analysis to broader architectures, quantify lens quality metrics, and explore interventions that harness this mechanism deliberately. As we continue this journey, the lens hypothesis not only explains past successes but lights the path toward more interpretable, efficient, and capable AI systems — bringing us closer to understanding the universal principles of intelligence.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/bhumika-kc-3784391b9/?isSelfProfile=true"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bdab982fac9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/tracing-structural-emergence-a-lens-on-end-to-end-gradient-descent-in-deep-networks-6bdab982fac9">Tracing Structural Emergence: A Lens on End-to-End Gradient Descent in Deep Networks</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Journey of RNA Sequencing: From Bulk Transcriptomes to Single-Cell Revelations — An End-to-End…]]></title>
            <link>https://medium.com/codetodeploy/the-journey-of-rna-sequencing-from-bulk-transcriptomes-to-single-cell-revelations-an-end-to-end-46f241229f1d?source=rss-9b994a01c790------2</link>
            <guid isPermaLink="false">https://medium.com/p/46f241229f1d</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[variational-autoencoder]]></category>
            <category><![CDATA[rna-sequencing]]></category>
            <category><![CDATA[transcriptome]]></category>
            <dc:creator><![CDATA[Drraghavendra]]></dc:creator>
            <pubDate>Fri, 26 Jun 2026 06:57:46 GMT</pubDate>
            <atom:updated>2026-06-27T04:32:33.562Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Journey of RNA Sequencing: From Bulk Transcriptomes to Single-Cell Revelations — An End-to-End Exploration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqFUJfeopa9pGj8J7HudMQ.png" /><figcaption>RNA Sequencing: Illuminating the Hidden Cellular Universe Where Every Transcript Shapes Health, Disease, and the Future of Precision Medicine</figcaption></figure><h4>I)Introduction :</h4><p>RNA sequencing (RNA-seq) stands as one of the most transformative technologies in modern biology. It profiles the *transcriptome* — the complete set of RNA transcripts produced by the genome in a cell or population of cells — revealing which genes are actively expressed, at what levels, and under what conditions. This provides a dynamic snapshot of cellular activity far beyond what DNA sequencing alone can offer, as it captures the functional output of the genome.</p><blockquote><strong><em>💰 High-Paying Remote Tech Jobs</em></strong><br> 🌎 Work From Anywhere<br> 🚀 Multiple Openings Available<br><strong> </strong><a href="https://refer.micro1.ai/referral/jobs?referralCode=4d8f644a-04a3-4fac-82e4-c4c1626a9009&amp;linkId=7f8b2be4-3caa-439d-b902-e4c2dc87c1a9&amp;utm_source=referral&amp;utm_medium=share&amp;utm_campaign=job_referral"><strong>👉 Apply Now</strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkQFf0NUnzoFbVEVQCF5WQ.png" /></figure><p>Traditional bulk RNA-seq averages gene expression across millions of cells in a tissue sample, offering robust insights into overall patterns but masking cellular heterogeneity. Single-cell RNA sequencing (scRNA-seq), by contrast, dissects expression profiles at the individual cell level, often for thousands to millions of cells. This leap unlocks unprecedented resolution into cellular diversity, rare cell populations, developmental trajectories, disease mechanisms, and evolutionary dynamics within tissues.</p><p>Contemplating this evolution feels profound: Biology has long grappled with the complexity of multicellular life. Bulk methods gave us the “average citizen” view of a tissue; scRNA-seq lets us meet every unique individual cell, revealing hidden stories of cooperation, rebellion, and specialization. It shifts our perspective from population statistics to a census of cellular identities.</p><h4>II)The Comprehensive Workflow: From Sample to Insight</h4><p>The journey begins with experimental design. Researchers isolate RNA from samples (tissues, cells, or biofluids), often enriching for poly(A) mRNA or using total RNA approaches. For scRNA-seq, cells are dissociated and captured — commonly via microfluidic droplet-based platforms like those from 10x Genomics, which barcode individual cells with unique molecular identifiers (UMIs) to quantify transcripts accurately.</p><p>Library preparation involves reverse transcription to cDNA, amplification, and sequencing on high-throughput platforms (e.g., Illumina). Raw data processing uses tools like **Cell Ranger** (from 10x Genomics) for alignment to a reference genome, quality control, and generation of gene-barcode matrices.</p><p>Downstream analysis in R with the Seurat package (or Python alternatives like Scanpy) involves normalization, dimensionality reduction (e.g., PCA, t-SNE, UMAP refers to Uniform Manifold Approximation and Projection), clustering to identify cell types, differential expression testing, and trajectory inference. Seurat v5 enhances this with scalable “sketch”-based analysis for massive datasets (millions of cells), multimodal integration (e.g., with ATAC-seq), and spatial transcriptomics support.</p><p>Thoughtfully, each step demands trade-offs: High cell throughput vs. read depth; capturing rare cells vs. computational burden. Contemplating these choices reminds us that data generation is easy — insight requires rigorous QC, batch correction, and biological validation.</p><h4>III)Real-Time Applications and Use Cases: Where RNA-seq Meets the Clinic and Beyond</h4><p>i)Oncology and Precision Medicine<br>Tumors are ecosystems of diverse cells. scRNA-seq maps **tumor heterogeneity**, identifying malignant clones, cancer stem cells driving metastasis, and immune infiltrates in the tumor microenvironment. This reveals how subpopulations interact and evade therapy.</p><p>In drug resistance tracking, longitudinal scRNA-seq of biopsies shows clonal evolution — e.g., emergence of resistant populations under proteasome inhibitors or targeted therapies. Clinicians can pivot treatments dynamically.</p><p>Gene fusions and biomarkers: RNA-seq excels at detecting chimeric transcripts and splicing variants that DNA sequencing misses, informing diagnostics and therapies (e.g., in leukemias or sarcomas).</p><p>Contemplation: Cancer’s lethality often stems from its adaptability. scRNA-seq exposes this at single-cell resolution, paving the way for truly personalized oncology — treating the “many diseases” within one tumor.</p><p>ii. Immunology and Autoimmune Diseases<br>scRNA-seq profiles immune cell subtypes with exquisite detail, distinguishing T-helper 17 cells, exhausted T cells, or novel regulatory populations during infections or inflammation.</p><p>In autoimmunity (e.g., multiple sclerosis, rheumatoid arthritis), it uncovers pathogenic signatures and dysregulated pathways, identifying new therapeutic targets like specific cytokine networks or cell states.</p><p>Real-time impact: Monitoring immune responses in blood or tissues for vaccine development, immunotherapy optimization (e.g., CAR-T), or predicting flares in autoimmune conditions.</p><p>iii.) Neuroscience and Neurodegeneration<br>The brain’s complexity — thousands of neuronal and glial subtypes — benefits immensely from cell atlases built via scRNA-seq. These map cellular taxonomy across regions and conditions.</p><p>In Alzheimer’s and Parkinson’s, scRNA-seq traces early molecular changes in vulnerable cell types, glial responses, and resilience factors. Large initiatives like the Seattle Alzheimer’s Disease Brain Cell Atlas (SEA-AD) integrate transcriptomics with pathology for insights into progression and cognitive resilience.</p><p>Thought process: Neurodegenerative diseases progress silently for years. Single-cell views let us detect “outlier” cells at the onset, potentially enabling earlier interventions and understanding how protein aggregates or inflammation disrupt specific networks.</p><p>iv.) Developmental Biology and Stem Cell Research<br>scRNA-seq illuminates **lineage tracing** and differentiation trajectories. By profiling embryonic cells over time, researchers reconstruct gene regulatory networks governing pluripotency, organogenesis, and fate decisions.</p><p>Applications include optimizing stem cell therapies, modeling congenital disorders, and understanding reprogramming in regenerative medicine. It reveals transient states and decision points invisible in bulk data.</p><p>v.)Public Health and Infectious Diseases<br>During COVID-19, RNA-seq enabled rapid viral genome tracking, host response profiling, and wastewater-based epidemiology to monitor community spread and variants, even in asymptomatic populations. It supported RNA vaccine design by characterizing immune responses.</p><p>Broader uses: Metatranscriptomics for pathogen-host interactions, outbreak surveillance, and studying chronic infections (e.g., chikungunya).</p><h4>IV) Platforms, Tools, and the Future</h4><blockquote>Capture Platforms: 10x Genomics Chromium for high-throughput droplet-based scRNA-seq; alternatives for spatial or multiomic profiling.<br>Analysis: Cell Ranger → Seurat (or equivalents) for clustering, visualization, and integration.<br>Emerging: Spatial transcriptomics combines location with expression; multimodal omics; AI-driven interpretation for massive datasets.</blockquote><p>Challenges remain — cost, technical variability, data integration, and ethical considerations around human samples — but the field advances rapidly toward routine clinical use.</p><h4>V)Advanced Python Program: scRNA-seq Analysis Pipeline with Keras VariationalAutoEncoder and AWS Integration</h4><pre>import os<br>import json<br>import boto3<br>import numpy as np<br>import pandas as pd<br>import scanpy as sc<br>import anndata<br>import tensorflow as tf<br>from tensorflow.keras import layers, Model, backend as K<br>from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>from sklearn.preprocessing import StandardScaler<br>import umap<br>import sagemaker<br>from sagemaker.tensorflow import TensorFlow<br><br># ========================= CONFIG =========================<br>S3_BUCKET = &quot;your-scrna-bucket&quot;  # Change this<br>DATA_KEY = &quot;data/pbmc_3k_raw.h5ad&quot;  # Example: 10x PBMC dataset or your data<br>MODEL_KEY = &quot;models/scrna_vae.keras&quot;<br>LATENT_DIM = 32<br>EPOCHS = 100<br>BATCH_SIZE = 128<br>USE_SAGEMAKER = False  # Set True for cloud training<br><br># AWS Clients<br>s3 = boto3.client(&#39;s3&#39;)<br>sagemaker_session = sagemaker.Session() if USE_SAGEMAKER else None<br><br># ====================== VAE MODEL ======================<br>class Sampling(layers.Layer):<br>    &quot;&quot;&quot;Reparameterization trick for VAE.&quot;&quot;&quot;<br>    def call(self, inputs):<br>        z_mean, z_log_var = inputs<br>        batch = tf.shape(z_mean)[0]<br>        dim = tf.shape(z_mean)[1]<br>        epsilon = tf.random.normal(shape=(batch, dim))<br>        return z_mean + tf.exp(0.5 * z_log_var) * epsilon<br><br>def build_vae(input_dim, latent_dim=LATENT_DIM):<br>    # Encoder<br>    inputs = layers.Input(shape=(input_dim,))<br>    h = layers.Dense(512, activation=&#39;relu&#39;)(inputs)<br>    h = layers.BatchNormalization()(h)<br>    h = layers.Dropout(0.2)(h)<br>    h = layers.Dense(256, activation=&#39;relu&#39;)(h)<br>    z_mean = layers.Dense(latent_dim)(h)<br>    z_log_var = layers.Dense(latent_dim)(h)<br>    z = Sampling()([z_mean, z_log_var])<br>    <br>    encoder = Model(inputs, [z_mean, z_log_var, z], name=&#39;encoder&#39;)<br>    <br>    # Decoder<br>    latent_inputs = layers.Input(shape=(latent_dim,))<br>    h = layers.Dense(256, activation=&#39;relu&#39;)(latent_inputs)<br>    h = layers.Dense(512, activation=&#39;relu&#39;)(h)<br>    outputs = layers.Dense(input_dim, activation=&#39;sigmoid&#39;)(h)  # or linear for counts<br>    <br>    decoder = Model(latent_inputs, outputs, name=&#39;decoder&#39;)<br>    <br>    # VAE<br>    vae_outputs = decoder(encoder(inputs)[2])<br>    vae = Model(inputs, vae_outputs, name=&#39;vae&#39;)<br>    <br>    # Loss<br>    reconstruction_loss = tf.reduce_mean(tf.keras.losses.mse(inputs, vae_outputs)) * input_dim<br>    kl_loss = -0.5 * tf.reduce_mean(1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var))<br>    vae.add_loss(reconstruction_loss + kl_loss)<br>    vae.compile(optimizer=&#39;adam&#39;)<br>    <br>    return vae, encoder, decoder<br><br># ===================== DATA HANDLING =====================<br>def load_data_from_s3(bucket, key, local_path=&quot;data.h5ad&quot;):<br>    &quot;&quot;&quot;Download from S3 and load with Scanpy.&quot;&quot;&quot;<br>    os.makedirs(os.path.dirname(local_path), exist_ok=True)<br>    s3.download_file(bucket, key, local_path)<br>    adata = sc.read_h5ad(local_path)<br>    return adata<br><br>def preprocess_adata(adata, n_top_genes=2000):<br>    sc.pp.filter_cells(adata, min_genes=200)<br>    sc.pp.filter_genes(adata, min_cells=3)<br>    sc.pp.normalize_total(adata, target_sum=1e4)<br>    sc.pp.log1p(adata)<br>    sc.pp.highly_variable_genes(adata, n_top_genes=n_top_genes)<br>    adata = adata[:, adata.var.highly_variable]<br>    sc.pp.scale(adata, max_value=10)<br>    return adata<br><br># ===================== TRAINING =====================<br>def train_vae(adata, latent_dim=LATENT_DIM):<br>    X = adata.X.toarray() if hasattr(adata.X, &#39;toarray&#39;) else adata.X<br>    scaler = StandardScaler()<br>    X_scaled = scaler.fit_transform(X)<br>    <br>    input_dim = X_scaled.shape[1]<br>    vae, encoder, decoder = build_vae(input_dim, latent_dim)<br>    <br>    callbacks = [<br>        EarlyStopping(monitor=&#39;loss&#39;, patience=15, restore_best_weights=True),<br>        ModelCheckpoint(&#39;best_vae.keras&#39;, save_best_only=True, monitor=&#39;loss&#39;)<br>    ]<br>    <br>    history = vae.fit(X_scaled, X_scaled, epochs=EPOCHS, batch_size=BATCH_SIZE,<br>                      validation_split=0.1, callbacks=callbacks)<br>    <br>    # Latent representations<br>    z_mean, _, _ = encoder.predict(X_scaled)<br>    return vae, encoder, z_mean, history, scaler<br><br># ===================== VISUALIZATION =====================<br>def visualize_latent(z, adata, title=&quot;scRNA-seq Latent Space&quot;):<br>    reducer = umap.UMAP(random_state=42)<br>    embedding = reducer.fit_transform(z)<br>    <br>    plt.figure(figsize=(10, 8))<br>    sc.pl.embedding(adata, basis=None, color=&#39;leiden&#39; if &#39;leiden&#39; in adata.obs else None, <br>                   show=False)  # Optional clustering<br>    sns.scatterplot(x=embedding[:,0], y=embedding[:,1], hue=adata.obs.get(&#39;leiden&#39;), palette=&quot;tab10&quot;)<br>    plt.title(title)<br>    plt.savefig(&quot;latent_umap.png&quot;)<br>    plt.show()<br><br># ===================== AWS INTEGRATION =====================<br>def upload_to_s3(local_path, bucket, key):<br>    s3.upload_file(local_path, bucket, key)<br>    print(f&quot;Uploaded {local_path} to s3://{bucket}/{key}&quot;)<br><br>def deploy_to_sagemaker(vae_model_path=&quot;best_vae.keras&quot;):<br>    &quot;&quot;&quot;Deploy trained model to SageMaker endpoint.&quot;&quot;&quot;<br>    # Save model in TensorFlow SavedModel format for SageMaker<br>    tf.saved_model.save(vae_model_path.replace(&#39;.keras&#39;, &#39;&#39;), &#39;saved_model&#39;)<br>    upload_to_s3(&#39;saved_model&#39;, S3_BUCKET, &#39;models/saved_model.tar.gz&#39;)  # Simplified<br>    <br>    estimator = TensorFlow(<br>        entry_point=&#39;inference.py&#39;,  # Create this separately for serving<br>        source_dir=&#39;.&#39;,<br>        role=sagemaker.get_execution_role(),<br>        instance_count=1,<br>        instance_type=&#39;ml.m5.large&#39;,<br>        framework_version=&#39;2.12&#39;,<br>        py_version=&#39;py310&#39;,<br>    )<br>    # estimator.deploy(...)  # Uncomment for actual deployment<br>    print(&quot;Model ready for SageMaker deployment.&quot;)<br><br># ===================== INFERENCE =====================<br>def predict_on_new_data(encoder, new_data_path, bucket, key):<br>    adata_new = load_data_from_s3(bucket, key, &quot;new_data.h5ad&quot;)<br>    adata_new = preprocess_adata(adata_new)<br>    X_new = adata_new.X.toarray() if hasattr(adata_new.X, &#39;toarray&#39;) else adata_new.X<br>    # Scale with saved scaler (in production, load from S3)<br>    z_new, _, _ = encoder.predict(X_new)<br>    return z_new<br><br># ========================= MAIN =========================<br>if __name__ == &quot;__main__&quot;:<br>    print(&quot;Starting scRNA-seq VAE Pipeline with AWS...&quot;)<br>    <br>    # Load &amp; Preprocess<br>    adata = load_data_from_s3(S3_BUCKET, DATA_KEY)<br>    adata = preprocess_adata(adata)<br>    print(f&quot;Processed data shape: {adata.shape}&quot;)<br>    <br>    # Train<br>    vae, encoder, z_latent, history, scaler = train_vae(adata)<br>    <br>    # Visualize<br>    visualize_latent(z_latent, adata)<br>    <br>    # Save &amp; Upload<br>    vae.save(&#39;final_vae.keras&#39;)<br>    upload_to_s3(&#39;final_vae.keras&#39;, S3_BUCKET, MODEL_KEY)<br>    upload_to_s3(&#39;latent_umap.png&#39;, S3_BUCKET, &#39;results/latent_umap.png&#39;)<br>    <br>    if USE_SAGEMAKER:<br>        deploy_to_sagemaker()<br>    <br>    print(&quot;Pipeline completed successfully! Latent embeddings ready for downstream tasks (clustering, cell typing, trajectory inference).&quot;)</pre><p>Inference.py</p><pre>import json<br>import tensorflow as tf<br><br>def model_fn(model_dir):<br>    return tf.keras.models.load_model(model_dir)<br><br>def input_fn(serialized_input_data, content_type):<br>    data = json.loads(serialized_input_data)<br>    return np.array(data[&#39;instances&#39;])<br><br>def predict_fn(input_data, model):<br>    return model.predict(input_data).tolist()<br><br>def output_fn(prediction, accept):<br>    return json.dumps({&#39;predictions&#39;: prediction})</pre><h4>VI)Contemplative Close: The Broader Implications</h4><p>Embarking on an RNA-seq journey is humbling. It forces us to confront biological noise as signal: Every cell is a unique story. In oncology, it humanizes precision medicine; in neuroscience, it maps the mind’s cellular architecture; in public health, it equips us against evolving threats.</p><p>As we integrate scRNA-seq with spatial, proteomic, and longitudinal data, we edge closer to a holistic understanding of life — from embryo to disease to therapy. The technology doesn’t just profile expression; it reveals the choreography of existence.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9QTaaaxboQP_gKSLedW_w.png" /></figure><p>👉 Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></p><p>👉 Follow us: <a href="https://www.linkedin.com/in/bhumika-kc-3784391b9/?isSelfProfile=true"><strong>Linkedin</strong></a>| <a href="https://medium.com/codetodeploy"><strong>Medium</strong></a></p><p>👉 CodeToDeploy Tech Community is live on Discord — <a href="https://discord.gg/ZpwhHq6D"><strong>Join now!</strong></a></p><p><strong>Disclosure:</strong> This post includes affiliate and partnership links.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46f241229f1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codetodeploy/the-journey-of-rna-sequencing-from-bulk-transcriptomes-to-single-cell-revelations-an-end-to-end-46f241229f1d">The Journey of RNA Sequencing: From Bulk Transcriptomes to Single-Cell Revelations — An End-to-End…</a> was originally published in <a href="https://medium.com/codetodeploy">CodeToDeploy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>