<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shoa Aamir on Medium]]></title>
        <description><![CDATA[Stories by Shoa Aamir on Medium]]></description>
        <link>https://medium.com/@shoa.devs?source=rss-c9a282dbece6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nDfmDtEc0DdNt_WbHvU9ig.png</url>
            <title>Stories by Shoa Aamir on Medium</title>
            <link>https://medium.com/@shoa.devs?source=rss-c9a282dbece6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:55:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shoa.devs/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What JEV is Not?]]></title>
            <link>https://medium.com/@shoa.devs/what-jev-is-not-b32b54518f77?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b32b54518f77</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 15:09:53 GMT</pubDate>
            <atom:updated>2026-09-25T15:13:47.523Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been seeing variations of <em>“Will Jev replace LLMs?”</em> lately — Blogs, youtube thumbnails, linkedin posts… so on. Replacement of Models, LLMs and Humans have become the gimmicky scroll-stopper these days.</p><p>Why do we talk about every new AI model as if it has to replace the previous one? Bigger model. Smaller model. Faster model. Better model.</p><p>That’s quite an absurd way to frame it.</p><p>The actual frame to view JEV’s release is now -</p><blockquote><strong>LLM is built to Generate and JEV is built to Decide</strong></blockquote><h3>What really is JEV ?</h3><p>JEV is TypeSafe AI’s first System One model. Instead of generating text, it takes a state, some data or context, and returns a typed, structured decision: yes/no, pick one, or a score, each with a confidence attached.</p><p>The System One name is inspired by Daniel Kahneman’s distinction between the fast, intuitive kind of thinking, as opposed to System 2’s slow, deliberate reasoning. Human intuition doesn’t reason step by step either, it fires in one shot, and JEV’s architecture is built the same way, it uses a parallel sampler that evaluates the text context (“state”) in a single pass instead of a deliberated chain. It’s here to make the call, fast, and hand the result to your code. More on exactly how that pass works next.</p><h3>How JEV vs LLMs Respond?</h3><p>An LLM generally generates sequentially:</p><figure><img alt="LLM Response Framework" src="https://cdn-images-1.medium.com/max/1024/1*Gkmb51qv-nDNEKAzm7ElmA.png" /></figure><p>Jev’s approach is different. Its sampler produces decision outputs in parallel rather than generating a text sequence token by token. TypeSafe also built a new model architecture and a training approach called <strong>RLCD, Reinforcement Learning for Calibrated Decisions</strong>, around this decision-making objective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BqGvwLmSJh5jUGg3QOIIg.png" /></figure><p>The important part isn’t just that this can make inference fast.</p><p>It’s the philosophy behind it.</p><blockquote><strong>Don’t optimise an AI system to produce arbitrary text when the software only needs a bounded decision.</strong></blockquote><h3>Smart if-statements, not a Smarter Chatbot</h3><p>Here’s the philosophical split. An LLM’s job: give me an answer. JEV’s job: here’s the state, here are the questions, here are the possible answer types, now decide.</p><p>Your code stays in control of the workflow. TypeSafe calls this pattern smart if-statements. JEV just answers a bounded question, your code decides what happens next.</p><p>Three output types, that’s it:</p><ul><li><strong>Noul</strong>: yes/no, with a probability. “Is this fraudulent?” comes back as 0.93</li><li><strong>Choice</strong>: pick one option from a defined set, up to 255 of them, with a confidence distribution</li><li><strong>Score</strong>: rate something on a scale, with confidence</li></ul><figure><img alt="JEV Architecture" src="https://cdn-images-1.medium.com/max/1024/1*LdYaK_qZIZKvFGG2VI3Psg.png" /></figure><h3>Here’s how you use JEV in Application</h3><p>The basic input is:</p><p><strong>State</strong></p><ul><li>The information the model needs to make the decision.</li><li>Can be unstructured text or structured program state. <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev?utm_source=chatgpt.com">TypeSafe AI</a></li></ul><p><strong>Questions</strong></p><ul><li>Explicit questions you want the model to answer.</li><li>Each question has a defined output type.</li></ul><pre>State:<br>Customer was charged twice for the same order.<br><br>Questions:<br>1. Is this a duplicate charge? → Noul<br>2. What action should we take? → Choice<br>   [refund, escalate, reject]<br>3. How severe is the issue? → Score<br>   [1, 2, 3, 4, 5]</pre><p>This is important: <strong>you decompose a complicated workflow into narrow decisions rather than asking one LLM prompt to do everything.</strong> TypeSafe&#39;s workflow research explicitly uses this approach.</p><p>TypeSafe exposes a POST /v1/systemone API with the three question types.</p><pre>from typesafe_sdk import Choice, Noul, Score, TypeSafeClient<br><br>state = {<br>    &quot;ticket&quot;: &quot;I was charged twice and need the duplicate refunded today.&quot;,<br>    &quot;account_tier&quot;: &quot;business&quot;,<br>}<br><br>with TypeSafeClient() as client:<br>    response = client.system_one(<br>        state=state,<br>        questions={<br>            &quot;intent&quot;: Choice(<br>                instructions=&quot;What is the customer&#39;s main request?&quot;,<br>                criteria={<br>                    &quot;refund&quot;: &quot;The customer wants money returned.&quot;,<br>                    &quot;technical_help&quot;: &quot;The customer needs a bug or integration fixed.&quot;,<br>                    &quot;information&quot;: &quot;The customer is asking for information only.&quot;,<br>                    &quot;other&quot;: &quot;None of the other options clearly fits.&quot;,<br>                },<br>            ),<br>            &quot;is_urgent&quot;: Noul(<br>                instructions=&quot;Does the ticket explicitly communicate time pressure?&quot;<br>            ),<br>            &quot;frustration&quot;: Score(<br>                instructions=&quot;How frustrated does the customer appear?&quot;,<br>                criteria=[<br>                    &quot;Calm and neutral&quot;,<br>                    &quot;Concerned but civil&quot;,<br>                    &quot;Very angry or using strong language&quot;,<br>                ],<br>            ),<br>        },<br>    )</pre><p>Output comes back with the choice along with confidence -</p><pre>{<br>  &quot;model&quot;: &quot;jev-latest&quot;,<br>  &quot;answers&quot;: {<br>    &quot;billing&quot;: {<br>      &quot;noul&quot;: 0.98,<br>      &quot;type&quot;: &quot;noul&quot;<br>    },<br>    &quot;action&quot;: {<br>      &quot;choice&quot;: &quot;refund&quot;,<br>      &quot;probabilities&quot;: {<br>        &quot;refund&quot;: 0.91,<br>        &quot;escalate&quot;: 0.07,<br>        &quot;reject&quot;: 0.02<br>      },<br>      &quot;type&quot;: &quot;choice&quot;<br>    },<br>    &quot;severity&quot;: {<br>      &quot;score&quot;: 3.8,<br>      &quot;confidence&quot;: 0.89,<br>      &quot;type&quot;: &quot;score&quot;<br>    }<br>  },<br>  &quot;usage&quot;: {<br>    &quot;input_tokens&quot;: 120,<br>    &quot;output_tokens&quot;: 12<br>  }<br>}</pre><h3>When to use LLM vs JEV?</h3><p>JEV fits narrow, repeatable judgment calls, the kind a system needs at high volume:</p><ul><li>Classifications</li><li>Fraud and risk scoring</li><li>Guardrails, jailbreak detection, tool selections and evaluating other AI output</li><li>Routing in Multi-Agent Systems</li><li>Document tagging</li><li>Approval or review decisions</li></ul><p>An LLM stays the right tool for anything that ends in language a person reads: writing, coding, explaining, open-ended reasoning, conversation.</p><h3>What JEV is NOT?</h3><p>To repeat the actual point how it is ridiculous to compare JEV to LLMs.</p><p>JEV <strong>cannot</strong>:</p><ul><li>Write or explain anything in prose</li><li>Hold a conversation</li><li>Code</li><li>Generate anything whose structure isn’t already known upfront</li></ul><p>It’s not built for generating anything!</p><p>It’s also not obviously more accurate. It is meaningfully behind frontier models on harder tasks, one invoice-processing benchmark had it at 61.8% against a frontier model’s 79.1%. The pitch that holds up is speed and cost at comparable accuracy, not smarter decisions.</p><p>So no, JEV is not “a better LLM.” It’s a different primitive for a different part of the stack, and the two aren’t competing for the same job. A good system probably uses both: the LLM for understanding and generation, JEV for evaluation and decisions, ordinary code executing whatever comes out.</p><h3>What’s worth watching?</h3><p>There is another development that makes this space even more interesting.</p><p>Together AI released <strong>Tev1</strong>, an open-weight, Jev-inspired model built by fine-tuning Qwen3.5–4B. <a href="https://github.com/togethercomputer/tev1">Link to the OSS Repository</a></p><p>But unlike Jev, Tev retains the underlying autoregressive LLM architecture. It essentially demonstrates that an existing language model can be fine-tuned to behave like a constrained decision model.</p><p>So we now have two interesting approaches:</p><p><strong>Jev:</strong> build a model and inference stack specifically around decisions.</p><p><strong>Tev:</strong> take an existing LLM and teach it to behave like a decision model.</p><p>Worth watching where this development goes eventually…</p><h3>The bigger idea</h3><p>The bigger idea as we move forward with AI maturing is —</p><blockquote>Lets not put LLMs everywhere and be smarter engineers!</blockquote><h3>References</h3><ul><li><a href="https://anthonymaio.substack.com/p/jev-the-language-model-that-wont">Jev the Language Model that Won’t Talk</a></li><li><a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">TypeSafe AI’s Official Release Blog</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b32b54518f77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You Found Sensitive Data. Now What?]]></title>
            <link>https://medium.com/@shoa.devs/you-found-sensitive-data-now-what-b22db1ae0b47?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b22db1ae0b47</guid>
            <category><![CDATA[healthtech]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Wed, 05 Aug 2026 08:45:42 GMT</pubDate>
            <atom:updated>2026-08-05T08:45:42.086Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>An open source workflow for assessing privacy risk, optimizing de-identification, and preserving data utility.</blockquote><p>Lets say you are preparing dataset for a healthcare AI application, or analytics or just exchanging with third party.</p><p>Generally, healthcare data is multimodal mix of names, medical record numbers, dates of birth, free-text clinical notes, referral documents, and years of operational data accumulated across different systems.</p><p>And if you have had any exposure in health-tech, you’d know the first question is:</p><blockquote><strong>How sensitive is the data?</strong></blockquote><p>Today, there are excellent tools available to answer that question by detecting personally identifiable information (PII), protected health information (PHI), and other sensitive entities. Some also support calls to de-identify them.</p><p>But identifying sensitive values is only the beginning. The question requires following answers too -</p><ul><li><strong>How severe is the privacy risk?</strong></li><li><strong>Which datasets, records, or fields deserve immediate review?</strong></li><li><strong>Which anonymization strategy is suffice for compliance while preserving as much useful information as possible?</strong></li><li><strong>Can those decisions be explained and audited later?</strong></li></ul><p>Detection tells you <strong>where sensitive information exists.</strong></p><p>It doesn’t tell you <strong>how risky that information is, how it should be treated optimally, or whether the dataset is actually ready for its intended use.</strong></p><p>That’s the gap <a href="https://github.com/seiba-ai/seiba-risk-scanner.git"><strong>Seiba Risk Scanner</strong></a> is designed to support.</p><h3>Introducing Seiba Risk Scanner</h3><p><em>Extending sensitive data detection into a complete privacy risk assessment workflow</em></p><p><a href="https://github.com/seiba-ai/seiba-risk-scanner.git">Seiba Risk Scanner is an open source toolkit</a> (<a href="https://github.com/seiba-ai/seiba-risk-scanner.git">https://github.com/seiba-ai/seiba-risk-scanner.git</a>) that extends sensitive data detection into an end-to-end privacy risk assessment and de-identification workflow.</p><p>It supports both structured datasets and unstructured text, helping transform raw detections into explainable risk reports, policy-driven de-identification plans, and measurable privacy outcomes. Rather than stopping at identifying sensitive information, it helps answer the questions that follow before data moves into AI systems, analytics pipelines, or external environments.</p><p>The workflow follows six stages:</p><blockquote><strong>Identify → Quantify Risk → Explain → Optimize → Apply → Measure</strong></blockquote><ul><li><strong>Identify</strong> sensitive entities across structured and unstructured data using configurable detection backends, medical knowledge sources, and editable ontology definitions.</li><li><strong>Quantify Risk</strong> by measuring severity, exposure, high-risk records, re-identification risk, and where sensitive information is concentrated across a dataset.</li><li><strong>Explain</strong> every finding with provenance, detector evidence, confidence, and the reasoning behind recommended actions.</li><li><strong>Optimize</strong> privacy actions using configurable policies while balancing compliance requirements with downstream data utility.</li><li><strong>Apply</strong> those decisions by automatically de-identifying text, documents, and structured datasets using the selected policy.</li><li><strong>Measure</strong> the resulting privacy exposure, residual risk, and utility after the transformations have been applied</li></ul><p><a href="https://github.com/maziyarpanahi/openmed">OpenMed</a> provides the default foundation for named entity recognition, de-identification primitives, and privacy analysis. Seiba Risk Scanner extends to build the broader workflow around those capabilities by combining additional detection layers, configurable ontologies, structured data assessment, risk scoring, provenance, optimization, reporting, and execution into a single pipeline. The NER backend is swappable (spaCy, OpenMed, custom LLM or transformers), so detection is never tied to a single model.</p><p>Beyond swappable detection backends, Seiba Risk Scanner also exposes its ontology as configuration. New entity types, detection rules, severity, and default de-identification actions can be added through editable YAML without modifying the codebase.</p><h3>Walkthrough 1: Turning a Structured Export into a Risk Assessment</h3><p>Suppose you’re preparing a structured patient export for downstream analytics or sharing with another team.</p><p>Despite structured, real data is rarely clean — multiple formats, abbrevations, free-text discrepancies, and so on…</p><p>Here’s a csv excerpt example:</p><pre>rec_id,pat_nm,mrn,ssn_num,dob,phone,zip_cd,comments,status,amount<br>R-0001,&quot;Delgado, Maria&quot;,MRN-0012345,412–88–7390,1978–03–14,617–555–0143,02149,&quot;left vm, call back after 4pm&quot;,active,124.50<br>R-0002,ROBERT CHEN,0012346,XXX-XX-7391,03/15/1978,(508) 555–0188,02149–1201,&quot;pt states insurance changed&quot;,pending,0.00<br>R-0003,linda whitfield,mrn 12347,512–44–8821,16-Mar-78,+1 781 555 0122,2149,&quot;call daughter Maria Delgado at 617–555–0143&quot;,active,89.00<br>…</pre><p>Notice what’s happening here. Columns aren’t following a standardized format, SSN is partially masked, and the comments column also contains PII.</p><p>Let’s start by assessing the dataset without modifying it.</p><pre>from pathlib import Path<br><br>from seiba_risk_scanner.assessment import ReadinessAssessor, report_from_paths<br><br>report = report_from_paths(<br>    [&quot;structured_corpus/tables/dirty_intake.csv&quot;], # replace with the file path<br>    assessor=ReadinessAssessor(execute_policy=False, optimize=Privacy.BALANCED),<br>    health_context=True,<br>    out_dir=Path(&quot;reports&quot;), # replace with output dir<br>)<br><br>print(report.exposure_index)<br>print(sum(report.severity_histogram.values()))<br>print(len(report.review_queue))</pre><blockquote>Setting execute_policy=False<em> </em>tells the SDK to identify sensitive information, assess the risk, and plan de-identification actions without applying them. This gives you a chance to understand the dataset before making any changes.</blockquote><p>Running the assessment on this <a href="https://github.com/seiba-ai/seiba-risk-scanner/blob/main/examples/data/tables/dirty_intake.csv">200-row csv</a> produces:</p><pre>Exposure Index:        80.7 / 100<br>Severity Distribution: critical 1078 · high 2 · medium 213<br>Review Queue:          695 findings flagged for review<br>Re-identification:     93% of records are unique on their quasi-identifiers</pre><p><a href="https://github.com/seiba-ai/seiba-risk-scanner/blob/main/examples/outputs/reports/dirty_intake_risk.md">Link to Full Generated Report</a></p><p>The report goes beyond listing detections by showing <strong>how the privacy risk is distributed across the dataset</strong>.</p><p>The <strong>Exposure Index</strong> is a dataset-level measure of privacy exposure on a scale from <strong>0 to 100</strong>. Rather than counting how many sensitive values were found, it combines two factors: <strong>how severe the findings are</strong> and <strong>how easily records could be re-identified</strong>. This makes the score comparable across datasets regardless of their size.</p><p>In this example, <strong>84%</strong> of findings are classified as <strong>High</strong> or <strong>Critical</strong>, while <strong>93%</strong> of records are unique on combinations of <strong>quasi-identifiers</strong> such as dates, ZIP codes, and other indirect identifiers. That means many records could still be distinguished from the indirect identifiers without relying on names or other direct identifiers. Together, those factors produce an <strong>Exposure Index of 80.7</strong>.</p><p>The report also generates a <strong>heatmap</strong> that highlights where the risk is concentrated. Severity indicated is contextual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*3EVeMiVO8ODufClYeYylFg.png" /></figure><blockquote><strong>Medium &lt; High &lt; Critical </strong>represent how much exposure exists as those same identifiers may be inherently sensitive and appear alongside other strong identifiers present a higher overall privacy risk.</blockquote><p>Once you’re satisfied with the assessment, applying the planned de-identification is simply a matter of enabling policy execution. Re-running the assessment with execute_policy=True applies the planned de-identification.</p><p>The resulting dataset becomes:</p><pre>rec_id,pat_nm,mrn,ssn_num,dob,phone,zip_cd,comments,status,amount<br>R-0001,&quot;[PERSON], [PERSON]&quot;,[ID_NUM],[SSN],[DATE_OF_BIRTH],[PHONE],[ZIPCODE],&quot;left vm, call back [TIME]&quot;,active,124.50<br>R-0002,[PERSON],[ID_NUM],XXX-XX-7391,[DATE_OF_BIRTH],[PHONE],[ZIPCODE],&quot;[PERSON] insurance changed, will fax new card&quot;,pending,0.00<br>R-0003,[PERSON],mrn [ID_NUM],[SSN],[DATE_OF_BIRTH][DATE_OF_BIRTH],[PHONE],[ZIPCODE],call daughter [PERSON] at [PHONE],active,89.00</pre><pre>rec_id,pat_nm,mrn,ssn_num,dob,phone,zip_cd,comments,status,amount<br>R-0001,&quot;PERSON_3d4aa16c, PERSON_9ff18ebe&quot;,ID_NUM_723b1e2e,SSN_445ec659,DATE_OF_BIRTH_8c6621ae,PHONE_123ab500,[ZIPCODE],&quot;left vm, call back [TIME]&quot;,active,124.50<br>R-0002,PERSON_e8937f23,ID_NUM_5370e321,XXX-XX-7391,DATE_OF_BIRTH_0bf1a489,PHONE_4a26a4b8,[ZIPCODE],&quot;PERSON_fbb81ed6 insurance changed, will fax new card&quot;,pending,0.00<br>R-0003,PERSON_e301a2a1,mrn ID_NUM_5570b8ff,SSN_92fa0a81,DATE_OF_BIRTH_b17ef6d1DATE_OF_BIRTH_2eafc164,PHONE_c00ceecf,[ZIPCODE],call daughter PERSON_7d60f673 at PHONE_123ab500,active,89.00</pre><p>Notice what changed and what didn’t. Multiple MRN formats are normalized into a single replacement. The already masked SSN in row R-0002 is also preserved, cross-references like phone number from R-0001 and R-0003 <strong>comment</strong> column is preserved with same hash <strong>PHONE_123ab500</strong>.</p><p>The SDK measures the impact of those choices through report.utility.overall, a utility loss score between <strong>0</strong> and <strong>1</strong>, where <strong>0</strong> indicates no analytical value was lost and <strong>1</strong> indicates complete loss. Here the value is approx 0.59<strong> </strong>— fifty-nine percent of the analytic utility in those columns is gone after optimizing utility and privacy (discussed further later)</p><p>The workflow has now gone from identifying sensitive information, to assessing the risk, to producing a de-identified dataset that is ready for the next stage of the pipeline instead of stopping at a report.</p><h3>Walkthrough 2: Understanding Clinical Notes and Optimized De-identification</h3><p>Structured datasets are only half of the problem. Some of the most valuable healthcare information lives inside free-text clinical notes, referral letters, discharge summaries, and consultation reports, where sensitive information appears naturally alongside clinically meaningful context.</p><p>Consider this consultation note:</p><pre>RIVERBEND SPECIALTY CLINIC<br>1200 Harbor Avenue, Suite 410, Portland, OR 97205 | Tel 503-555-0142 | Fax 503-555-0198<br><br>NEUROLOGY CONSULTATION<br>Date of service: 08/01/2026<br>Re: Maya Ellison | DOB: 03/14/1978 | MRN: RB-472918 | Sex: F<br>Referring clinician: Dr. Elena Brooks, MD<br><br>HPI: There is no family history of Wilson disease. She has no history suggestive of<br>Crohn disease. There is no facial weakness to suggest Bell&#39;s palsy. The overall<br>pattern is most consistent with early Parkinson disease.</pre><p>Calling the SDK to classify entities and provide explainable anonymization plan:</p><pre>from pathlib import Path<br><br>from seiba_risk_scanner import SeibaScanner<br>from seiba_risk_scanner.assessment import ReadinessAssessor<br>from seiba_risk_scanner.assessment.optimize import Privacy<br><br>path = Path(&quot;neurology_consult.txt&quot;)<br>note = path.read_text(encoding=&quot;utf-8&quot;)<br><br>scan = SeibaScanner().classify_text(<br>    note,<br>    source_id=str(path), # passing source id and label allows provenance tracking<br>    source_label=path.name, <br>)<br><br>report = ReadinessAssessor(<br>    optimize=Privacy.BALANCED, # balanced privacy and utility optimization, it can be MAXIMUM utility or REQUIRED as per compliance<br>).assess(<br>    [scan],<br>    labels=[path.name],<br>    health_context=True,<br>)<br><br>for record in report.policy_plan.records:<br>    print(record.text, &quot;→&quot;, record.action, &quot;→&quot;, record.replacement)</pre><blockquote>One challenge in medical text is distinguishing patient identifiers from clinical terminology. In this sample note, <strong>Wilson disease</strong>, <strong>Crohn disease</strong>, <strong>Bell’s palsy</strong>, and <strong>Parkinson disease</strong> are correctly recognised as medical conditions rather than person names, while <strong>Maya Ellison</strong> and <strong>Dr. Elena Brooks</strong> are identified as people.</blockquote><blockquote>Getting this distinction wrong has real consequences. Misclassifying conditions as patient names removes clinically important information, while missing genuine names risks leaking PHI.</blockquote><p><a href="https://github.com/seiba-ai/seiba-risk-scanner/blob/main/examples/outputs/single/adv_01_neurology_consult_findings.json">Link to Full Run Output</a></p><h4>The De-Identification plan</h4><p>Once the classification is complete, the SDK generates a policy-driven de-identification plan. In this example, the default <strong>HIPAA Safe Harbor policy</strong> is used together with the <strong>Privacy.BALANCED </strong>optimization profile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CpSG3yDTYRUvo77C3zfxRw.png" /><figcaption>Smart De-identification of Data</figcaption></figure><blockquote>You’ll notice that repeated mentions of Maya Ellison resolve to the same hashed identifier (PERSON_815fd9a3). This preserves co-reference throughout the document, allowing downstream systems to recognise that multiple mentions refer to the same patient without revealing their identity.</blockquote><h4>Optimizing privacy without unnecessarily sacrificing utility</h4><p>Not every detected entity deserves the same treatment. A patient name is a direct identifier. An MRN can often be hashed while remaining linkable across a dataset. A date of birth may be generalized to a year. Whether clinical conditions should be retained or removed depends on the privacy policy and the intended use of the data.</p><p>The optimization profile determines the smartest de-identification strategy. Producing Final De-identified Output</p><p>Producing the de-identified document directly one call:</p><pre>from seiba_risk_scanner.policy import scrub_text<br><br>clean = scrub_text(note, report.policy_plan.records, str(path))<br>Path(&quot;scrubbed/neurology_consult.txt&quot;).write_text(<br>    clean,<br>    encoding=&quot;utf-8&quot;,<br>)</pre><p><a href="https://github.com/seiba-ai/seiba-risk-scanner/blob/main/examples/outputs/scrubbed/adv_01_neurology_consult.txt">Scrubbed Output </a>—</p><pre>[ORGANIZATION]<br>STREET_ADDRESS_54b46fe3, [LOCATION], [LOCATION] [ZIPCODE] | Tel PHONE_ee84ec5b | Fax PHONE_60b6ab0d<br><br>NEUROLOGY CONSULTATION<br>Date of service: [DATE]<br>Re: PERSON_815fd9a3 | DOB: DATE_OF_BIRTH_64be4a7b | MRN: ID_NUM_84bc14b5 | Sex: F<br>Referring clinician: PERSON_c7fb6b99, MD<br><br>Dear Colleague:<br><br>Thank you for asking me to evaluate PERSON_815fd9a3 for progressive tremor and gait slowing. She reports that the right hand tremor began about eighteen months ago and is now more apparent when resting. Her partner has noticed smaller handwriting and reduced arm swing. She denies loss of consciousness, recent [CONDITION], or change in vision.<br><br>HPI: There is no family history of [CONDITION]. She has no history suggestive of [CONDITION]. Examination shows masked facies, mild right-sided bradykinesia, and a resting tremor. There is no facial weakness to suggest [CONDITION]. The overall pattern is most consistent with early [CONDITION].<br><br>Assessment/[CONDITION]: I reviewed medication options and referred her to physical therapy for balance training. MRI brain without contrast and ceruloplasmin are ordered. She will return in eight weeks or sooner for worsening falls, confusion, or new focal symptoms.<br><br>Sincerely,<br><br>PERSON_c7fb6b99, MD<br>Neurology | NPI PERSON_474dd161<br>Electronically signed [DATE]<br><br>[ORGANIZATION], STREET_ADDRESS_54b46fe3, [LOCATION], [LOCATION] [ZIPCODE]<br>This consultation contains confidential health information intended only for the receiving clinician.<br><br></pre><p>De-identification methods can be overridden either permanently through editable ontology definitions or at runtime using action_overrides for dataset-specific workflows.</p><pre>report = ReadinessAssessor(<br>    action_overrides={<br>        &quot;medical_condition&quot;: &quot;keep&quot;,<br>        &quot;date_of_birth&quot;: &quot;generalize:year&quot;,<br>    }<br>).assess(<br>    [scan],<br>    labels=[path.name],<br>    health_context=True,<br>)</pre><h4>Explainability — Why de-identification decisions were made?</h4><p>Every de-identification action carries enough information to explain where it came from, why it was detected, and which policy produced the final decision.</p><p>Sample object —</p><pre>{<br>  &quot;text&quot;: &quot;Maya Ellison&quot;,<br>  &quot;entity&quot;: &quot;person_names&quot;,<br>  &quot;start&quot;: 145,<br>  &quot;end&quot;: 157,<br>  &quot;action&quot;: &quot;hash&quot;,<br>  &quot;replacement&quot;: &quot;PERSON_815fd9a3&quot;,<br>  &quot;policy_name&quot;: &quot;hipaa_safe_harbor&quot;,<br>  &quot;source&quot;: &quot;openmed_policy_class&quot;,<br>  &quot;origin&quot;: {<br>    &quot;source_id&quot;: &quot;notes/neurology_consult.txt&quot;,<br>    &quot;source_label&quot;: &quot;neurology_consult.txt&quot;,<br>    &quot;row&quot;: null,<br>    &quot;column&quot;: null<br>  }<br>}</pre><h4>Running It Across Batch Data</h4><p>The same workflow scales from individual files to large dirs containing a mix of structured and unstructured data.</p><pre>from pathlib import Path<br>from time import perf_counter<br><br>from seiba_risk_scanner import SeibaScanner<br>from seiba_risk_scanner.assessment import (<br>    ReadinessAssessor,<br>    report_from_paths,<br>)<br>from seiba_risk_scanner.assessment.optimize import Privacy<br><br>scanner = SeibaScanner()<br>scanner.classify_text(&quot;warmup&quot;)  # Load models once<br><br>started = perf_counter()<br><br>report = report_from_paths(<br>    [&quot;incoming_data/&quot;], # Replace with the dir path<br>    scanner=scanner,<br>    assessor=ReadinessAssessor(<br>        optimize=Privacy.BALANCED,<br>    ),<br>    health_context=True,<br>    out_dir=Path(&quot;reports&quot;),<br>)<br><br>elapsed = perf_counter() - started<br><br>print(f&quot;Processed {len(report.per_record)} records in {elapsed:.1f}s&quot;)<br>print(&quot;Exposure Index:&quot;, report.exposure_index)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NVf6pRkyd0K84Dk1UrNlAg.png" /><figcaption>Sensitive Data Risk Report</figcaption></figure><p><a href="https://github.com/seiba-ai/seiba-risk-scanner/blob/main/examples/outputs/reports/corpus_risk.md">Link to A Risk Readiness Report of Multiple Multimodal Sources</a></p><h3>Try It Yourself</h3><p>If you’d like to explore the project yourself:</p><ul><li>⭐ Check out the GitHub repository — <a href="https://github.com/seiba-ai/seiba-risk-scanner.git">https://github.com/seiba-ai/seiba-risk-scanner.git</a></li><li>📖 Follow the README for installation and configuration.</li><li>💻 Run the example workflows to scan both structured datasets and clinical notes — <a href="https://github.com/seiba-ai/seiba-risk-scanner/tree/main/examples">https://github.com/seiba-ai/seiba-risk-scanner/tree/main/examples</a></li></ul><p>Privacy workflows don’t stop once sensitive information has been identified.</p><p>Before data is used for AI, analytics, or collaboration, teams still need to understand where risk is concentrated, decide how to reduce that risk, preserve as much useful information as possible, and document the decisions made along the way.</p><p>As AI adoption accelerates and privacy and regulatory expectations become more demanding, the question is gradually shifting from <strong>“Did we detect sensitive data?”</strong> to <strong>“Can we justify the decisions we made before this data moved?”</strong></p><p>That’s the workflow <strong>Seiba Risk Scanner</strong> is designed to support.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b22db1ae0b47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Semantic to Time-Aware: Unstructured Data for Predictive AI Agents]]></title>
            <link>https://medium.com/@shoa.devs/from-semantic-to-time-aware-unstructured-data-for-predictive-ai-agents-08bf0896d3a9?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/08bf0896d3a9</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[healthtech]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 06:14:09 GMT</pubDate>
            <atom:updated>2026-04-01T06:14:09.482Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Semantic to Temporal: Unstructured Data for Predictive Agents</h3><p>Here’s a number worth sitting with: <strong>80% of enterprise data is unstructured.</strong> Documents, emails, call logs, clinical notes, audio recordings — a vast ocean of narrative context, and signal that predictive AI has historically been blind to.</p><p>Generative AI has changed this dynamic. It is not only a shiny new interface, but also a critical piece of infrastructure. It acts as a translation layer, converting high dimensional, noisy and unstructured data into semantically rich and structured features that predictive models, augmented with agentic reasoning, can consume.</p><p>But here is the catch: this isn’t just about running a one-time LLM extraction and calling it a day. If you are building predictive intelligent AI agents that forecast risk, personalize care, or optimize operations, the architecture for handling unstructured data needs a fundamental rethink.</p><p>The real bottleneck isn’t the text. It’s <strong>time</strong>.</p><p>In this article, I’ll walk you through the engineering shift required to move from “semantic search” to “temporal intelligence.” To ground this, we’ll use a high-stakes case study: designing a data layer for <strong>mental health risk stratification</strong>, where the cost of getting the data pipeline wrong is significant.</p><h3>Gen AI Unlock: From Blind Spot to Live Inference</h3><p>Predictive intelligence has historically ignored unstructured data because it was too noisy to standardise. Structured data gives us the <em>what</em>. Unstructured data gives us the <em>why</em> and the <em>how</em>.</p><p>In modern agentic systems, unstructured data becomes a <strong>live inference resource</strong>, retrieved, reasoned over, and weighted at prediction time, not just at pipeline level.</p><p>This shift introduces a new class of engineering problems that most teams are not yet equipped to handle.</p><h3>Real Bottleneck : Time, Not Text</h3><p>The critical bottleneck for predictive agents is <strong>temporal reasoning</strong>.</p><p>If your system does not understand <em>when</em> something happened relative to everything else, its predictions break. This is not a one-time ETL problem. It is a continuous, event-driven pipeline that must preserve temporal provenance throughout.</p><p>Here are the core challenges:</p><ol><li><strong>Temporal Reasoning</strong></li></ol><p>Unstructured data can carry two timestamps, and they are not the same.</p><ul><li><strong>event_time</strong>: When the event actually occurred</li><li><strong>record_time</strong>: When it was documented</li></ul><p>Treating record_time as truth destroys causality.</p><p><strong>2. Longitudinal Representation</strong></p><p>We have to move beyond semantic reasoning. Predictive systems require longitudinal representation, tracking how states evolve over time.</p><p>For example: how does sentiment change over the last 90 days? What is the <strong>rate of change in sentiment or negativity density</strong>?</p><p><strong>3. Temporal Leakage</strong></p><p>This is harder with unstructured data than structured data. Because unstructured data often lacks explicit timestamps, it’s easy to accidentally include future information in your training set. You might use a note written <em>after</em> a risk event to predict the event. The model looks accurate offline, but fails in real-world deployment because it’s essentially cheating. We’ll come back to exactly how to prevent this.</p><p><strong>4. Governance and Control</strong></p><p>Unstructured data introduces serious governance challenges. Poor lineage, missing timestamps, and lack of access control create both compliance and audit risks. Point-in-time correctness and traceable transformations are not just technical requirements, they directly impact regulatory alignment in domains like healthcare.</p><h3>Case Study: Designing Data Pipeline for Mental Health Risk Stratification</h3><p>Let’s move from theory to architecture.</p><p>We are designing the data layer for an AI system performing <strong>mental health risk stratification</strong>.</p><p>This is a hard problem: rare signals, highly unstructured data, and regulatory constraints.</p><h4><strong>Step-by-Step Architectural Deep Dive</strong></h4><figure><img alt="Architecting ETL for Unstructured Temporal Data" src="https://cdn-images-1.medium.com/max/1024/1*X2FkQIsBOrvOfVX1yFfnlA.png" /><figcaption>Architecting Data Layer to Process Temporal Unstructured Data</figcaption></figure><p><strong>Step 1: Temporal Ingestion Engine (The Unstructured Gateway)</strong></p><p>This is where most pipelines fail. We cannot treat a note’s filename timestamp as the truth.</p><p><strong>What the engine does:</strong></p><ul><li>Parses text for temporal anchors: <em>“three weeks ago,” “last Tuesday,” “since her father passed.”</em></li><li>Uses a fine-tuned BERT-style model to extract (anchor_phrase, relative_offset, confidence) (this requires labeled training data specific to temporal extraction)</li><li>Calculates absolute event_time using record_time as reference</li><li>Stores both timestamps and confidence</li></ul><p>Example output from a single sentence:</p><blockquote>“Patient says she stopped sleeping about two weeks ago, but today she feels slightly better.</blockquote><pre>{<br>  &quot;record_time&quot;: &quot;2025-03-31&quot;,<br>  &quot;extracted_event&quot;: &quot;sleep_disturbance_onset&quot;,<br>  &quot;event_time&quot;: &quot;2025-03-17&quot;,<br>  &quot;confidence&quot;: 0.82,<br>  &quot;original_text&quot;: &quot;stopped sleeping about two weeks ago&quot;<br>}</pre><p><strong>Step 2: Longitudinal Feature Builder (Structured + Unstructured Merge)</strong></p><p>We align structured and unstructured signals along a <strong>shared timeline</strong>.</p><p><strong>Unstructured-derived features:</strong></p><ul><li><strong>Sentiment velocity:</strong> rate of change in sentiment over time</li><li><strong>Ideation density:</strong> mentions per 1000 words</li><li><strong>Therapeutic alliance signal</strong></li><li><strong>Narrative vs structured mismatch score</strong></li></ul><p>The mismatch score captures when structured assessments indicate stability, but narrative data shows deterioration. These contradictions often surface earlier and act as strong early risk signals.</p><p><strong>Key principle:</strong> every feature must be computed only from events where event_time &lt;= prediction timestamp.</p><p><strong>Step 3: Point-in-Time Feature Store (The Anti-Leakage Shield)</strong></p><p>This is the most under-engineered component in most AI systems, and is the direct solution to the temporal leakage problem introduced earlier.</p><p>A point-in-time feature store ensures that when you train a model on historical data, it sees exactly what the production system would have seen at that historical moment. Without this, models silently train on future information and fail in production.</p><p><strong>How it works:</strong></p><ul><li>Every feature is stored with a <strong>valid_from</strong> and <strong>valid_to</strong> timestamp.</li><li>For a prediction on March 15, 2025, the feature store retrieves only feature values where valid_from &lt;= 2025-03-15 &lt; valid_to.</li><li>Unstructured-derived features are recomputed for each historical snapshot.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L42jFyAIFspUia7wJipj9w.png" /></figure><p>A model predicting on March 10th uses -0.3. A model predicting on March 16th uses -0.7. No leakage. In longitudinal prediction tasks, eliminating this kind of leakage can meaningfully close the gap between offline evaluation metrics and real-world model performance — the delta is often larger than teams expect.</p><p><strong>Why this matters for unstructured data specifically:</strong> Unstructured features are computed on the fly from raw text. Without point-in-time control, you might accidentally use a note written <em>after</em> a crisis to predict the crisis. The model looks excellent offline and fails in production.</p><p><strong>Cross-Cutting Layer: Temporal Knowledge Graph (For Explainability, Retrieval, and Causal Context)</strong></p><p>It is a shared layer used across feature building, retrieval, and prediction.</p><p>Vector databases can handle similarity and metadata filtering, including time ranges. The limitation is not filtering, but <strong>causal and relational reasoning across time</strong>.</p><p>What the TKG stores:</p><ul><li><strong>Entity nodes:</strong> Patient, Clinician, Medication, Symptom, LifeEvent</li><li><strong>Temporal edges:</strong> (Patient) --[had_symptom from 2025-03-01 to 2025-03-15]--&gt; (Insomnia)</li><li><strong>Causal edges:</strong> (JobLoss on 2025-02-01) --[triggered]--&gt; (Insomnia onset)</li></ul><p>Queries the TKG enables:</p><ul><li><em>“What symptoms preceded the March 20th crisis event?”</em></li><li><em>“Which patients with similar symptom trajectories had good outcomes with intervention X?”</em></li><li><em>“Show me all evidence considered for the March 20th prediction.”</em></li></ul><p>It acts as the system’s memory layer, supporting both model inputs and explainability.</p><p><strong>Integration with prediction:</strong> The agent retrieves relevant subgraphs from the TKG and injects them as context into the LLM or gradient-boosted model.</p><h4><strong>Three Approaches to Integrate Structured + Unstructured</strong></h4><p>Depending on your scale, latency needs, and budget, here are three integration strategies. These aren’t strictly either/or choices — most mature systems layer them, starting with feature concatenation as the baseline and adding retrieval or agentic reasoning selectively for high-stakes or low-throughput prediction paths.</p><p><strong>Approach 1: Feature Concatenation</strong></p><p>Treat unstructured-derived features (sentiment velocity, ideation density, mismatch score) as additional columns in your existing structured feature table. Train any standard model (XGBoost, Logistic Regression, Random Forest) on the concatenated feature set.</p><p><em>Use when:</em> You need high interpretability (regulated healthcare), have limited GPU budget, or need prediction latency under 100ms.</p><p><em>Trade-offs:</em> Loses some narrative nuance. The model sees aggregated features, not the raw text.</p><h4>Approach 2: Two-Tower Retrieval + Ranking</h4><ul><li><strong>Tower A (Structured):</strong> XGBoost or TabNet on structured features produces a baseline risk score.</li><li><strong>Tower B (Unstructured):</strong> A retrieval model queries the TKG for relevant past episodes and scores similarity. Note: ColBERT (a late-interaction retrieval model) and Contriever (a contrastive dense retriever) operate differently under the hood, and the right choice depends on your index size and latency tolerance.</li><li><strong>Fusion layer:</strong> Weighted combination or small neural network merging both scores.</li></ul><p><em>Use when:</em> You have a large corpus of historical cases to retrieve from, want example-based explanations (“similar to Patient X’s episode in March”), and can tolerate 200–500ms latency.</p><p><em>Trade-offs:</em> Requires maintaining a high-quality retrieval index. Fusion layer tuning is non-trivial.</p><h4>Approach 3: Agentic Reasoning Loop</h4><p>Used for high-stakes, low-volume predictions requiring explainability.</p><ol><li>Agent receives a prediction request.</li><li>Queries the TKG for the patient’s recent temporal trajectory.</li><li>Retrieves 3–5 similar historical cases from episodic memory.</li><li>Generates a candidate risk score with reasoning steps.</li><li>Validates reasoning against structured ground truth where available.</li><li>Revises or confirms the score.</li><li>Stores the prediction + reasoning trace for future learning.</li></ol><p><em>Use when:</em> You need human-level explanations for high-stakes decisions, have a small number of predictions per day (hundreds, not millions), and can tolerate 2–5 second latency.</p><p><em>Trade-offs:</em> Expensive (LLM calls per prediction). Harder to validate. Best for augmenting human clinicians, not replacing them.</p><p>Mature production systems evolve from 1 → 2 → selective use of 3.</p><h4>Putting It All Together: A Concrete Prediction Example</h4><p><strong>Patient context on March 15, 2025:</strong></p><ul><li><strong>Structured:</strong> PHQ-9 = 6 (PHQ is a depression intensity score, mild depression), no missed appointments, medication adherent.</li><li><strong>Unstructured (session note from March 14):</strong> <em>“Patient reports feeling ‘completely fine,’ but mentions ‘I’ve stopped answering my phone’ and ‘my partner doesn’t understand why I’m so tired all the time.’”</em></li></ul><p><strong>Pipeline execution:</strong></p><ol><li><strong>Temporal ingestion:</strong> Extracts social_withdrawal event with event_time = March 10 (inferred from narrative).</li><li><strong>Longitudinal builder:</strong> Computes sentiment velocity = -0.6 (sharp decline over 5 days), ideation density = low, mismatch score = HIGH (PHQ-9 says mild, but narrative suggests functional decline).</li><li><strong>Feature store:</strong> Ensures only data before March 15 is used.</li><li><strong>TKG retrieval:</strong> Finds 2 similar past cases where PHQ-9 under-reported risk.</li><li><strong>Prediction agent:</strong> Outputs <strong>High Risk</strong> with explanation: <em>“PHQ-9 is stable, but narrative shows rapid decline in social engagement and energy. Two similar historical cases preceded crisis within 7 days.”</em></li></ol><p>Without the unstructured layer, this patient would have been flagged low-risk. With it, they receive an intervention.</p><h4>Limitations and Failure Modes</h4><p>This pipeline is not foolproof.</p><ul><li>Temporal extraction can be wrong or ambiguous</li><li>Narrative signals can be noisy or misleading</li><li>Mismatch scores can produce false positives</li><li>Knowledge graphs require ongoing curation</li><li>Latency and cost increase with sophistication</li></ul><p>If you do not explicitly account for these, you will overestimate model reliability.</p><h3><strong>Conclusion: The Path to Credible Predictive AI</strong></h3><p>By focusing on temporal ingestion, longitudinal structuring, point-in-time correctness, and polyglot storage, we can create data layers that do not just store unstructured data, but <strong>operationalize</strong> it for intelligent decision-making.</p><p>If you are building these systems, the competitive advantage no longer lies in just the model. It lies in the architecture that feeds it.</p><p>If you are wrestling with point-in-time stores for unstructured data in regulated domains, or trying to get temporal leakage under control in a longitudinal prediction system, that’s a good conversation to have. DM me.</p><h3>What’s Next? (Beyond Data Engineering)</h3><p>This article focused on the foundational data layer. But challenges are beyond data engineering in Predictive AI Agent orchestration, inference and governance. We can explore further these tiers of challenges:</p><ol><li><strong>Model Governance for Agentic Systems:</strong> How do you audit why an agent retrieved a specific piece of unstructured data to justify a prediction?</li><li><strong>Real-Time vs. Batch Processing:</strong> The architectural trade-offs of moving these pipelines from offline batch processing to real-time streaming inference.</li><li><strong>Feedback Loops:</strong> How to build the data infrastructure to allow the agent to learn from its outcomes (correct and incorrect predictions) without causing model drift.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08bf0896d3a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differential Privacy: The Overlooked Gem Powering Safer AI]]></title>
            <link>https://medium.com/@shoa.devs/differential-privacy-the-overlooked-gem-powering-safer-ai-c29099ba2510?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/c29099ba2510</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[differential-privacy]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Sun, 24 Aug 2025 19:25:25 GMT</pubDate>
            <atom:updated>2025-08-24T19:37:07.434Z</atom:updated>
            <content:encoded><![CDATA[<p>The honeymoon phase is over.</p><p>We’ve spent the last two years frantically integrating ChatGPT APIs into everything that moves, celebrating every “hello world” AI demo, and marveling at the magic of large language models. Now let’s finally get our engineering hats on and build AI agents that are airtight-foolproof against prying eyes while still delivering the wow factor.</p><p>I’ve been digging into the privacy-preserving techniques for months now that separate toy projects from serious, production-ready systems, and one powerful yet often overlooked hero is <strong>Differential Privacy (DP).</strong></p><p><strong>Imagine the data hungry AI models training and working without ever needing to see the actual data.</strong> It’s a game-changer for making AI agents secure, especially during common steps like fine-tuning models or using Reinforcement Learning from Human Feedback (RLHF). In this blog, we’ll start with the <strong>foundations of DP</strong>, then blend in the latest advancements like combining it with <strong>synthetic data generation.</strong></p><h3>What is Differential Privacy?</h3><p>Imagine you live in an apartment building and a researcher asks, “What’s the average salary of everyone in this building?” You want to help get a statistical answer, but you may not want anyone to know <em>your specific</em> salary.</p><p><strong>Differential Privacy</strong> is a mathematical framework to solve this! It <em>masks individual contributions</em> while preserving the overall statistical patterns and useful insights of dataset as a <em>whole</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JzCB7M2dAA6qWk5X.png" /></figure><p><strong>How does it work? </strong>By carefully adding a controlled amount of <strong>“noise”</strong> (randomness) to the data or the process of querying it. It’s like the researcher asking everyone to write their salary on a piece of paper, put it in a bowl, but then also adding a few random fake numbers (while trying to retain the average) into the mix before calculating the average. The overall average is still statistically accurate, but you can’t be sure if any single number is real or just noise.</p><p>So, noise essentially means corrupting the data a bit. But, this “noise” isn’t just random numbers; it’s added in a specific, mathematically precise way to provide the strongest guarantees. There are two common types of noise used in differential privacy to achieve a balance of maintaining accuracy and privacy —</p><p>The most popular is <strong>Laplace noise. </strong>This is the classic and most common mechanism for DP. Imagine a bell curve, but centered on the true answer. Laplace noise randomly draws numbers from this curve, meaning results are more likely to be near the truth but have a known probability of being farther away. It’s perfect for protecting numerical data like counts and averages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*0yJ4hDBTzjX2n4ED.png" /></figure><p>Another is <strong>Gaussian noise, </strong>similar to Laplace, but it uses the more familiar “normal” bell curve. It’s a bit trickier to work with mathematically for pure DP but is very common in practices like DP-SGD <em>(will be discussed soon)</em> because of how it interacts with complex model training.</p><p>The type and amount of noise are calibrated based on the sensitivity of the query — essentially, how much a single person’s data could possibly change the result. A question like “how many people live in the building?” has low sensitivity (a single person changes the count by 1), while “what is the total salary?” has high sensitivity (one billionaire could drastically change the result).</p><p>To make sure this noise actually delivers reliable protection, and to prevent too much information from leaking out over time — we turn to a key idea that ties it all together: <strong>the privacy budget.</strong></p><p>It’s a mathematical guarantee that quantifies exactly how much privacy you’re “spending” with each operation. This budget is measured by a parameter called epsilon (ε):</p><ul><li><strong>Lower epsilon</strong> = stronger privacy protection but potentially less accurate results</li><li><strong>Higher epsilon</strong> = weaker privacy protection but better model performance</li></ul><p>It’s like having a finite number of privacy coins to spend. Every time your model learns something from the data, it costs privacy coins.</p><p><strong><em>Why is there a limit to the privacy budget?</em> </strong>Because if you ask an infinite number of questions, even the tiny amounts of information leaked each time could, in theory, be combined to reconstruct the original data — a process called <strong>privacy loss composition.</strong></p><p>The privacy budget is this mathematical safety net. It quantifies and limits the <em>total cumulative privacy loss</em> across all operations. You have to limit the spending of this budget, because continuing would break the mathematical promise of DP. This forces us to be intentional and efficient with how we learn from sensitive data, which is exactly how we build truly trustworthy systems.</p><p>There are certainly some trade-off considerations with its implementation. The noise that provides privacy also slightly reduces the accuracy or utility of the model. It also ramps up computation costs, as you need extra steps to inject and manage that noise. Plus, setting the right privacy budget is tricky; too tight, and your system becomes useless; too loose, and privacy weakens. Getting this balance right is the key challenge.</p><h3>Real World Applications: From ETL to LLMs</h3><p>Differential Privacy has been already powering privacy-preserving systems across industries.</p><p><strong>ETL and Data Pipelines: </strong>In ETL pipelines (that’s Extract, Transform, Load — the process of pulling data from sources, cleaning it, and storing it), DP can mask sensitive info during data movement, preventing leaks in big data warehouses. Apple uses it to collect usage analytics while protecting user privacy.</p><p><strong>Traditional Machine Learning</strong>: In machine learning (ML), it’s used to train models on user data without exposing individuals, like in recommendation systems. Banks employ DP-trained models to detect fraud patterns without exposing individual transaction details.</p><p><strong>Large Language Models and GenAI: </strong>DP helps during training to avoid memorizing personal details from chat logs or user prompts. For instance, companies like Apple use DP in features like Siri to analyze usage patterns without invading privacy.</p><p>Many people prefer Apple over Samsung, mostly not for the features (where Samsung often matches or beats them), but for Apple’s reputation and its strong stance on user privacy. This trust isn’t accidental — it’s built by embedding technologies like Differential Privacy into the core of their services, from Siri’s intelligence to usage analytics.</p><p>But this responsibility shouldn’t fall to just a few tech giants. Regulatory bodies are making it a standard requirement. <strong>The European Data Protection Board (EDPB)</strong>, which oversees <strong>GDPR</strong> enforcement, explicitly emphasizes DP as a key tool for high-risk data processing, such as in AI systems handling sensitive information like health records or financial data. They see it as essential for mitigating risks like data regurgitation (where models spit out memorized private info) or inference attacks (guessing hidden details). In high-stakes scenarios, skipping DP is not optional; it’s fundamental for building responsible AI.</p><h3>Evolution to DP-SGD</h3><p>Now, let’s build on that foundation and talk about a powerhouse evolution: <strong>DP-SGD, or Differential Privacy Stochastic Gradient Descent. </strong>If regular DP is like adding static noise to a photo to blur faces, DP-SGD is like dynamically blurring a video frame by frame during filming.</p><p>Here’s how it works in simple terms: In normal ML training, we use an algorithm called Stochastic Gradient Descent (SGD) to tweak model parameters based on data examples, minimizing errors step by step. DP-SGD adds two key twists:</p><ol><li><strong>Clipping</strong>: It “<strong>clips</strong>” the gradients (the direction of tweaks) to limit how much any single data point influences the model, preventing one person’s info from dominating.</li><li><strong>Noise Injection: </strong>it injects random noise into those gradients before applying them. This happens iteratively during training, ensuring the final model doesn’t leak individual data points.</li></ol><p>By embedding these steps directly into the training loop, DP-SGD provides a seamless and robust method for building models with strong, mathematical privacy guarantees. The final model doesn’t just have noise added as a post-processing step; it’s <em>fundamentally</em> architected not to leak or memorize individual data points.</p><h4>The Quantifiable Leap from Foundational DP</h4><p>DP-SGD is a transformative approach that makes DP practical for modern AI. Foundational DP mechanisms, like adding Laplace noise to query outputs, often struggle with the complex, high-dimensional, and iterative nature of model training. DP-SGD directly addresses this by integrating privacy into the core optimization process.</p><p>The key improvements are substantial:</p><ul><li><strong>Stronger Privacy for Complex Workflows</strong>: DP-SGD provides a provable privacy guarantee for the entire model training process, not just a single output. This is crucial for ML, where thousands of queries (gradient steps) are made against the data.</li><li><strong>Superior Accuracy-Privacy Trade-off:</strong> By applying noise specifically to gradients (the <em>direction</em> of learning) rather than to the data itself, DP-SGD achieves a much more favorable utility-privacy balance. <em>Research shows that models fine-tuned with DP-SGD on large pre-trained foundations can outperform strong non-private baselines while maintaining formal guarantees, a feat traditional DP often couldn’t achieve.</em></li><li><strong>Computational Feasibility for Large Models: </strong>While slower than standard training, advancements like parameter-efficient fine-tuning (e.g., using LoRA) with DP-SGD have dramatically reduced the overhead. Techniques that freeze most of a model’s parameters can reduce the number of trainable parameters from billions to millions, which significantly cuts the gradient norm and thus the amount of noise required. This can lead to a 10x reduction in the privacy budget (ε) for the same accuracy, or significantly better accuracy for the same privacy budget.</li></ul><h4>How It’s Perfect for the “LLM World”</h4><ul><li><strong>Fine-Tuning LLMs</strong>: Fine-tuning a foundational model (like Llama or Mistral) on proprietary data is a primary risk for memorization. DP-SGD is your shield. Crucially, research has found a “sweet spot” using parameter-efficient methods like LoRA (Low-Rank Adaptation). By privately fine-tuning only a small subset of parameters (e.g., 20 million instead of 8 billion), the noise magnitude is drastically reduced. This allows for generating high-quality, privacy-preserving synthetic data or creating private models that retain much more utility than those trained with full-parameter DP-SGD.</li><li><strong>Privacy-Preserving Human Feedback (RLHF):</strong> AI agents use Reinforcement Learning from Human Feedback (RLHF) to align with human preferences. DP-SGD can be applied during the preference or feedback learning phase.</li><li><strong>Secure Continuous and Multi-Agent Learning:</strong> For AI agents that learn continuously from user interactions, DP-SGD guarantees that no individual user’s data is memorizable. Furthermore, in systems where multiple agents share learned knowledge, DP-SGD ensures that the shared model updates or synthetic data are free from information about any individual training example.</li></ul><h4>Acknowledging the Persistent Challenges</h4><p>Despite its power, challenges in deploying DP-SGD persist, though the landscape is rapidly improving.</p><ul><li><strong>Computational Overhead:</strong> The per-sample gradient clipping requirement can still slow training down compared to non-private baselines, with slowdowns in the 2–10x range being common, though this is highly dependent on model size and implementation.</li></ul><blockquote>However, new libraries and frameworks like DynamoEnhance and DP-ZeRO are leveraging model sharding and distributed training across multiple GPUs to make private training of massive (&gt;7B parameter) models feasible for the first time, overcoming the memory constraints of previous frameworks like Opacus.</blockquote><ul><li><strong>Hyperparameter Tuning:</strong> Finding the optimal sweet spot for the clipping norm and noise multiplier requires experimentation. Nevertheless, new algorithms like <strong>Auto DP-SGD</strong> are emerging to automate this.</li></ul><h3>Breakthrough: Using DP for Synthetic Data Generation</h3><p>Now here is an exciting breakthrough technique research by Google — <strong>CTCL (ConTrollability and CLustering), </strong>that combines two fuels of privacy — <strong>Synthetic data and Differential Privacy </strong>— in AI together. It’s the formula to lightweight privacy with maximum impact:</p><p>Imagine you’re building a healthcare AI agent that needs to understand medical conversations, but you can’t risk exposing patient data during training. Traditional approaches might require you to fine-tune massive billion-parameter models with differential privacy, burning through computational resources and privacy budgets.</p><p>CTCL flips this approach on its head. Instead of directly fine-tuning giant models, CTCL uses a lightweight 140-million parameter model to generate unlimited synthetic data that captures the patterns of your private domain without paying additional privacy costs.</p><p>Rather than trying to make large language models private, <em>CTCL creates a small, privacy-preserving generator that can produce synthetic data matching your private domain’s characteristics.</em></p><h4>CTCL Architecture</h4><p>The architecture involves two complementary models working in harmony:</p><p><strong>CTCL-Topic</strong>: Think of this as a universal theme detector. It analyzes documents and groups them into topics, with each topic represented by 10 descriptive keywords. This model is trained once on public data (like Wikipedia) and can then understand the topic distribution of any private dataset.</p><p><strong>CTCL-Generator</strong>: This is the synthetic data factory. It’s a conditional language model that takes keyword descriptions as input and generates documents matching those specifications. The magic happens when you fine-tune this generator with differential privacy on your private data.</p><h3>The Three-Step CTCL Methodology</h3><p><strong>Step 1: Build the Foundation Models</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CVMkV1CstnoNFYkN.png" /></figure><ul><li><strong>Train CTCL-Topic</strong> using 6 million Wikipedia documents or any domain relevant public corpora like PubMed, creating clusters of related content with 10 keywords each</li><li><strong>Train CTCL-Generator</strong> using 430 million description-document pairs, teaching it to create content based on keyword inputs. For healthcare AI Agent example, these public foundations ensure the models start with general medical knowledge (e.g., from encyclopedic entries on diseases) without touching any PHI, setting the stage for controlled synthesis.</li></ul><p><strong>Step 2: Learn Your Private Domain</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kIo1zsZPZ-OFMlto.png" /></figure><ul><li>Create a topic histogram showing the distribution of themes in your private data (e.g., “30% cardiology reports focusing on heart conditions, 20% oncology notes on cancer treatments, 50% general checkup summaries”)</li><li>Transform your private dataset into keyword-document pairs using CTCL-Topic. Let’s say your private PHI (Personal Health Information) dataset — such as redacted electronic health records — into keyword-document pairs using CTCL-Topic, like “chest pain, EKG, stent” without exposing raw details.</li><li>Fine-tune CTCL-Generator with differential privacy on these pairs. In practice, this means your model learns subtle patterns, like phrasing in medical narratives, directly from PHI data but with noise to obscure individuals</li></ul><p><strong>Step 3: Generate Unlimited Synthetic Data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SYJv69rIT14CUD0T.png" /></figure><ul><li>Sample topics according to your private domain’s histogram to maintain realistic proportions.</li><li>Feed corresponding keywords to your DP-fine-tuned CTCL-Generator, prompting it to produce de-identified synthetic documents — like fictional patient notes with plausible symptoms and treatments but no real PHI.</li><li>Generate as much synthetic data as needed without additional privacy costs (thanks to DP’s post-processing property), scaling up to millions of examples for robust model training. Sticking to our healthcare AI agent analogy, this could yield diverse datasets for fine-tuning LLMs on tasks like symptom prediction, all while ensuring the outputs are inherently private and audit-ready.</li></ul><h4>Application in AI Agents</h4><p>Just integrate the synthetic data into your agent’s training pipeline. It plugs perfectly into <strong>LLM fine-tuning:</strong> Use the synthetic data for DP-SGD training, <strong>boosting privacy in RLHF loops</strong>. Extend it to larger models by swapping in bigger generators, or apply to non-text like images with similar clustering. Add monitoring for utility drop.</p><h4>CTCL vs. Traditional DP-SGD Fine-Tuning</h4><p>While traditional DP-SGD directly fine-tunes large LLMs on private data with noise added to gradients, CTCL shifts the heavy lifting to synthetic data generation, leading to notable advantages.</p><p>Instead of wrestling with billion-parameter models under noisy gradients for weeks, CTCL’s lightweight 140M-parameter generator finishes training in just a few hours on standard hardware, making it far more accessible for smaller teams. It doesn’t just save compute — it slashes resource needs by more than <strong>10x compared to DP-SGD on &gt;7B params LLMs</strong>. Just as importantly, CTCL holds onto <strong>90–95% of the model’s original accuracy</strong>, even under strict privacy budgets where <strong>traditional DP-SGD often drops to</strong> <strong>70–85%</strong>. And because it works through synthetic data, it naturally adds a layer of protection, showing <strong>stronger resistance against attacks</strong> that try to guess whether someone’s personal data was used in training.</p><h3>Conclusion</h3><p>Research in differential privacy keeps moving fast, and what once felt like a heavy tradeoff between privacy, accuracy, and compute cost is shrinking by the day. Techniques like CTCL show that you don’t have to choose between airtight privacy and strong utility anymore; you can train smarter, cheaper, faster, and safer.</p><p>This shift matters because Differential Privacy is not a computational burden holding you back; it’s the secret weapon freeing you to build personalized, high-impact AI without worrying about sensitive/user data. Different workflows already have their own best-fit DP techniques — <strong>KL-regularization</strong> for reinforcement learning, <strong>BLT-DP-FTRL</strong> for federated learning, synthetic data generation through <strong>CTCL</strong> for LLM fine-tuning. So, just pick the one that suits your AI Agent’s needs, and bake it in from day one, especially if your AI or ETL pipelines interact with sensitive data.</p><blockquote>I just share from my own learnings and experiences. If there is any correction required, please leave a comment I would be glad to address it!</blockquote><blockquote>Happy Reading!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c29099ba2510" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Privacy in AI: Not Optional, But Survival! (And Here’s the guide to thrive)]]></title>
            <link>https://medium.com/@shoa.devs/privacy-in-ai-not-optional-but-survival-and-heres-the-guide-to-thrive-3ecd5229219a?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ecd5229219a</guid>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[data-privacy]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Tue, 12 Aug 2025 17:11:08 GMT</pubDate>
            <atom:updated>2025-08-12T17:11:08.126Z</atom:updated>
            <content:encoded><![CDATA[<p>Your AI’s biggest feature? Trust. Lose it, and nothing else matters.</p><p>I’ve worked as an ML Engineer in critical industries, and that experience taught me one simple fact: trust and safety around user data are not optional. Every AI agent that touches sensitive or personal data needs to operate with trust built in to its DNA.</p><p>In 2025, AI’s explosive growth is crashing headfirst into stricter regulations and a surge in privacy breaches. Just last month, thousands of ChatGPT conversations — loaded with sensitive info on mental health, abuse, jobs, and finances — leaked publicly through Google search.</p><p>Breaches aren’t just embarrassing; they cost you big — legally, financially, and in trust. New laws like the California AI Transparency Act and the EU AI Act demand full transparency and accountability from AI builders.</p><h3>Real Privacy Leaks &amp; Regulatory Lashback</h3><p>IBM’s latest numbers are brutal: 97% of organizations hit by AI-related breaches had zero proper AI access controls. The 2025 <em>Cost of a Data Breach Report</em> shows these incidents involving AI cost organizations an average of $670,000 more than those without AI involvement — on top of the trust, reputation, and compliance hits.</p><p>And it’s not just privacy. AI agents can be manipulated like over-trusting interns:</p><ul><li>December 2023 saw a Chevrolet dealership’s AI bot tricked into selling a $76,000 Tahoe for $1.</li><li>January 2025 brought a DeepSeek cloud misconfiguration exposing over a million chat logs and API keys.</li></ul><p>Regulations are also not sitting back. The EU AI Act, enforceable since August 2024 with full compliance required by 2026, targets bias, discrimination, and data misuse with strict transparency, bias detection, and human oversight rules. Miss these, and you risk fines up to 4% of your global revenue. Meanwhile, California’s AI Transparency Act, effective January 2026, forces generative AI developers to openly disclose their training datasets and cleansing methods. Italy’s data protection authority has already fined OpenAI for GDPR breaches, signaling a growing global crackdown coming ahead.</p><p>Critical industries like healthcare and finance face the highest stakes. Their customers entrust AI with life-impacting data, making privacy breaches not just costly but potentially devastating.</p><p>We can’t just fear these rules, we have to outpace them! The goal is to innovate and build AI that earns trust, survives scrutiny, and drives real transformation without becoming tomorrow’s headline.</p><h3>Key Principles of Privacy By Design</h3><p>It’s crucial to revise the core foundations of privacy in technical systems before diving into its implementation into AI workflows and agents. These are the 7 essential principles —</p><ul><li><strong>Proactive, Not Reactive; Preventative, Not Remedial</strong></li><li><strong>Privacy as the Default Setting</strong></li><li><strong>Privacy Embedded Into Design</strong></li><li><strong>Full Functionality — Positive-Sum, Not Zero-Sum</strong></li><li><strong>End-to-End Security, Full Lifecycle Protection</strong></li><li><strong>Visibility and Transparency</strong></li><li><strong>Respect for User Privacy</strong></li></ul><p>These timeless principles are also the foundations of regulations like — GDPR and EU AI Act.</p><h3>Now How to Make Our AI Agents Trustworthy?</h3><p>This landscape is still evolving rapidly, but some core architectural patterns are emerging. Let’s walk through each component and explore how you can bake trustworthiness into every layer. Think of this as your friendly roadmap to building agents that users will actually want to interact with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z2U_fdfkp9LyDViU" /></figure><h4>1. Perception/Input Module</h4><p>Think of this as the agent’s eyes and ears (and one day, maybe touch, smell, and taste). It processes prompts, context, speech-to-text, or OCR from images — turning human-readable input into machine-readable form. This is the agent’s frontline defense, where the stakes are high.</p><p><strong>What’s the risks we have here?</strong></p><ul><li>Collection of sensitive data</li><li>Over-collection of user data</li><li>Not sufficient visibility on how data is to be used.</li><li>Prompt Injections and all other possible attacks</li></ul><p>These involve the following principles — Privacy as the Default Setting, Visibility and Transparency, Privacy Embedded into Design</p><p><strong>How you can reduce risks:</strong></p><ul><li><strong>Data minimization at source + context-aware collection</strong> — only take what’s needed and adjust dynamically based on context, sensitivity, and compliance rules.</li><li><strong>Real-time anonymization/pseudonymization</strong> — remove or mask identifiers before storing or processing. <em>k-anonymity</em> means grouping data so any individual can’t be singled out; <em>tokenization</em> swaps sensitive values for safe placeholders.</li><li><strong>Adaptive consent management</strong> — instead of static “agree/disagree” forms, adapt in real time to user preferences, data sensitivity and changing regulatory demands.</li><li><strong>Strong authentication &amp; encryption</strong> — lock down both the input channel and stored data.</li><li><strong>Continuous monitoring</strong> — track anomalies, detect injection attempts, and watch for unusual data flows.</li><li><strong>Version control for prompts/configurations</strong> — so you can roll back safely if something goes wrong.</li></ul><h4>2. Agent Core (“Brain”)</h4><p>This is the brain, for most of the agents these days — “the LLMs”. It’s where reasoning, decision-making, and sometimes planning or routing happen. This is also where privacy can break in subtle, hard-to-detect ways — the blackbox layer.</p><p><strong>Risks:</strong></p><ul><li>Wrongful inference of sensitive data from seemingly harmless inputs</li><li>Retaining memory beyond compliance limits or user rights</li><li>Lack of visibility in autonomous decision-making</li><li>Biases that lead to unfair or discriminatory outcomes</li><li>Cross-context data leakage between unrelated sessions or tasks</li></ul><p><strong>Relevant Principles:</strong> <em>Privacy Embedded into Design, Visibility &amp; Transparency, Respect for User Privacy</em></p><p><strong>How you can safeguard the brain:</strong></p><ul><li><strong>Anonymize high-risk data at every step</strong> — a baseline you should apply in each step.</li><li><strong>Federated learning with secure multi-party computation</strong> — trains models without moving raw data to one central place. Secure multi-party computation means multiple parties can compute together without revealing their data to each other.</li><li><strong>Differential privacy</strong> — adding small, controlled “noise” to data or training so individual info can’t be traced back.</li><li><strong>Privacy-preserving LLM architectures</strong> — designs that inherently limit data leakage.</li><li><strong>Knowledge unlearning</strong> — intentionally removing specific data the model has memorized (for legal or ethical reasons).</li><li><strong>Local/ Offsite adapter tuning</strong> — personalize models locally without sharing raw data upstream.</li><li><strong>Explainability and audit trails</strong> — so you can show why a decision was made.</li><li><strong>Human-in-the-loop oversight</strong> — have a person review high-risk decisions or step in if there’s a flagged issue.</li><li><strong>Privacy budgets</strong> — track how much “private” information the model has seen to avoid overexposure.</li><li><strong>Provenance tracking</strong> — keep records of where your training data came from.</li><li><strong>Dataset declarations</strong> — in some jurisdictions (like California) you must disclose what datasets were used.</li><li><strong>Pipeline hardening</strong> — monitor against data poisoning (feeding bad data), membership inference (guessing if someone’s data was used), and inversion attacks (reconstructing training data).</li><li><strong>On-prem or open-source models for sensitive industries</strong> — avoid sending critical data to third-party APIs.</li><li><strong>Synthetic data</strong> for training.</li></ul><h4>3. Memory / State Management</h4><p>This is the external memory of your AI agent — the cache or storage that holds context, tracks conversations, or passes state between components or even different agents.</p><p>The risks are familiar: if this storage leaks, wrongful access, data lingering longer than the subject’s rights allowance, unsecured memory layer prone to attacks. The tiny twist with AI is that we’re now also dealing embeddings and complex stores like Graph DBs are gaining prominence.</p><p>It is straightforward like the existing systems, just with new technologies for storage and retrieval because we query based semantic similarities and have different types of storage being prominent like Graph DBs.</p><p><strong>Solutions are straightforward (some also discussed above) —</strong></p><ul><li><strong>Strong Role-Based Access Control (RBAC)</strong></li><li><strong>Strong encryption at rest (AES-256) and in transit (TLS 1.3+).</strong></li><li><strong>Data expiration policies</strong> tied to subject rights and compliance rules.</li><li>Automatic<strong> sensitive data redaction and anonymization </strong>before storage.</li><li><strong>Continuous monitoring</strong></li><li><strong>Differential Privacy</strong> for query and retrieval operations</li></ul><blockquote>I am going to stop mentioning “continuous monitoring” and “sensitive data redaction” and “anonymization” in coming modules. Just do it in each step.</blockquote><h4>4. Tools &amp; Action Module</h4><p>This is the “hands” of the AI agent, i.e., calling external APIs, triggering third-party tools, or running local processes. At this stage, privacy risks extend beyond your own system and into the services you connect with. So, robust safeguards are non-negotiable:</p><ul><li><strong>Authenticate all requests</strong> — no anonymous calls.</li><li><strong>Encrypt in transit</strong> — keep it unreadable along the way.</li><li><strong>Audit third-party services</strong> if you’re working with critical or regulated data.</li></ul><h4>5. Learning &amp; Self-Reflection: The Improvement Engine</h4><p>This is where the agent refines itself over time. If you’ve seen the buzz around reinforcement learning (like DeepSeek’s LLM R1), this is that territory.</p><p>It shares similar risks as fine-tuning and uses similar technical safeguards as well, just take into consideration the reward or feedback dataset.</p><h4>6. Interface: The User Touchpoint</h4><p>The interface is where users judge whether your AI agent respects their autonomy. This is where trust can be won or lost.</p><p><strong>Tips:</strong></p><ul><li><strong>Privacy-by-default settings</strong> — start secure, let users loosen if they want.</li><li><strong>Granular but easy consent controls</strong> — let users decide without confusing them.</li><li><strong>Strong authentication</strong> — prevent unauthorized access.</li><li><strong>Plain-language explanations</strong> — no legalese walls of text.</li></ul><h4>7. AI Output</h4><p>While the interface is about <em>how</em> users interact with an agent, the <strong>output </strong>is about <em>what</em> the agent actually produces — its words, images, code, or decisions. This layer deserves its own spotlight because the concerns here are fundamentally different from UI concerns. Even if your data collection is airtight, the model can still leak info or generate unsafe responses.</p><p><strong>Risks:</strong></p><ul><li><strong>Leakage of sensitive or personally identifiable information</strong> in generated content</li><li><strong>Hallucinated facts</strong> presented as truth, leading to trust erosion eventually</li><li><strong>Cross-context information bleed</strong> from previous sessions or other users</li><li><strong>Unsafe or biased recommendations</strong> embedded in responses</li></ul><p><strong>Safeguards:</strong></p><ul><li><strong>Dynamic output filtering</strong> — catch and block personal info before it’s shown.</li><li><strong>Context-aware guardrails</strong> — adapt safety rules to the task at hand.</li><li><strong>Hybrid filtering</strong> — mix rules, ML classifiers, and semantic checks for sensitive patterns.</li><li><strong>NER (Named Entity Recognition) for privacy</strong> — flag names, IDs, and other personal details.</li></ul><p>AI agents can’t be treated as “just another new technology” anymore, bolted together piece by piece like a scrappy prototype. They must be designed as complete, cohesive systems. <strong>Evaluation, governance, and explainability</strong> aren’t separate add-ons — they’re foundational layers that should be baked into every step.</p><ul><li><strong>Evaluation</strong> — measure performance and safety continuously, not just after launch.</li><li><strong>Governance &amp; monitoring</strong> — set rules, watch for drift, and adapt over time, log and maintain audit trails and data flow diagrams. <em>(Needless to say at this point, but never log sensitive data)</em></li><li><strong>Explainability</strong> — make it possible for humans to understand why the agent made a decision.</li></ul><h3>Conclusion</h3><p>Phew! This was quite information heavy and technical perhaps. There will be future blogs to discuss some aspects and techniques in depth.</p><p>Privacy is hard to just bake in, you gotta optimize your ingredients, maintaining utility, costs, latency and privacy. Its a tough battle. But if you want your AI agents to handle some of the most sensitive, personal, and valuable data in the world, and provide critical value, the responsibility that comes with that capability is enormous. Organizations that master this will win user trust, stay compliant, and avoid costly disasters. The competition isn’t just over features — it’s over who can be trusted.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ecd5229219a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KV Cache Secrets: Star Attention 101]]></title>
            <link>https://medium.com/@shoa.devs/kv-cache-secrets-star-attention-101-b95a81b827ee?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b95a81b827ee</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Sat, 31 May 2025 12:54:19 GMT</pubDate>
            <atom:updated>2025-05-31T12:55:21.419Z</atom:updated>
            <content:encoded><![CDATA[<p>In previous posts, we explored the headaches of scaling LLMs — how attention computation gets increasingly expensive as sequences grow longer, and how that infamous quadratic scaling becomes a serious bottleneck. We’ve gone through many attention optimization mechanisms already like Flash Attention, Radix, and Chunk Attention.</p><p>And if you haven’t read my deep dive on KV cache and how it turbocharges inference — hit pause and check that out first: <a href="https://medium.com/@shoa.devs/kv-cache-secrets-boost-llm-inference-efficiency-ae6c53968857">KV Cache Secrets: Boost LLM Inference Efficiency</a>.</p><p>Now, if you’re thinking “Aren’t we done with attention optimizations already?” — not quite. Because there is another new player on the block: <strong>Star Attention</strong>.</p><p>Let’s get into why we need yet another attention mechanism.</p><h3>Why we need another Attention Mechanism?</h3><p>Let’s quickly recap why, despite all the fancy tricks, earlier discussed attention mechanisms still leave us wanting more:</p><p><strong>FlashAttention (and FlashAttention-2):</strong></p><ul><li><strong>Hardware Dependency:</strong> Delivers impressive speedups on NVIDIA H100 and A100 GPUs, but performance diminishes on older hardware like V100.</li><li><strong>Implementation Complexity:</strong> Relies on custom CUDA kernels optimized for specific architectures, posing challenges for portability and broader adoption.</li></ul><p><strong>Radix Attention:</strong></p><ul><li><strong>KV Cache Reuse:</strong> Excels in scenarios requiring reuse of key-value caches, such as multi-turn conversations.</li><li><strong>Memory Constraints:</strong> Limited by GPU memory capacity, and current strategies for cache management need further refinement.</li></ul><p><strong>Chunk Attention:</strong></p><ul><li><strong>Long-Range Dependencies:</strong> Effective when inputs can be segmented into independent chunks, but struggles with capturing long-range dependencies due to restricted token interactions across distant positions.</li></ul><p><strong>Blockwise or Segmented Attention Approaches:</strong></p><ul><li><strong>Adaptability Issues:</strong> These methods often require fine-tuning or additional components, making them less adaptable to existing inference pipelines without retraining.</li></ul><p><strong>Ring Attention</strong></p><p>Since it’s not been discussed before, here’s a quick overview before drawing any comparisons. You can read more in depth about it in this wonderful article over <a href="https://medium.com/@iamtanujsharma/breaking-the-boundaries-understanding-context-window-limitations-and-the-idea-of-ring-attention-170e522d44b2">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u0K-wkAj7-ue1qPD.png" /></figure><blockquote>Works by dividing the input sequence into blocks and distributing them across devices arranged in a ring topology.</blockquote><blockquote>Each device computes self-attention and feedforward layers for its block locally.</blockquote><blockquote>Key-value pairs are passed between devices in a circular (ring) manner, overlapping computation and data transfer to reduce communication overhead.</blockquote><blockquote>Final attention scores are updated using received key-value pairs to account for cross-block dependencies, and outputs are combined for the final result.</blockquote><p><strong>Key Features</strong></p><ul><li>Enables processing of near-infinite context lengths by distributing computation across devices, overcoming single-device memory limitations.</li><li>Maintains quadratic complexity (O(N²)) within blocks but scales across devices, making it suitable for sequences far beyond traditional Transformer limits (e.g., GPT-3.5: 16,000 tokens, GPT-4: 32,000 tokens).</li><li>Overlaps computation and communication, reducing the overhead associated with long-sequence processing.</li></ul><p>As mentioned above, best for<strong> near-infinite context </strong>processing <em>BUTTT</em> still involves <strong>quadratic complexities</strong>.</p><p><strong>Enter Star Attention 🌟:</strong></p><ul><li><strong>Distributed Computing Efficiency:</strong> Introduces a two-phase block-sparse approximation that enhances computational efficiency by sharding attention across multiple hosts, reducing communication overhead.</li><li><strong>Inference Speed and Accuracy:</strong> Achieves up to <strong>11x faster inference</strong> while maintaining <strong>95–100% accuracy</strong>, making it ideal for long-context tasks in <strong>scalable</strong> and <strong>distributed</strong> computing environments (and in this world of “Agents,” that seems quite useful).</li></ul><p><em>Besides, Star Attention is also compatible with Flash Attention and Ring Attention for additional speedup and enhanced scalabiliy.</em></p><h3>Star Attention’s Two-Phased Approach</h3><p>Star Attention optimizes long-sequence processing in Large Language Models (LLMs) by dividing the inference process into two distinct phases: <strong>Context Encoding</strong> and <strong>Query Generation</strong>. This structure enhances efficiency without compromising accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*Re7BNmJnq_OolaZz.png" /></figure><h4>Phase 1: Context Encoding</h4><p>In this phase, the input context is segmented and processed to prepare for efficient token generation:</p><ul><li>The long input is chopped into contiguous blocks.</li><li>Every block (except the first) starts with a copy of the initial block, also called the “anchor block.” Think of this anchor as the central brain that helps each piece of the puzzle stay connected to the big picture.</li><li>These blocks are then shipped off to separate compute hosts. Each host focuses only on its assigned block and the anchor. That means less memory, faster attention, and a scalable process.</li><li>As each host works through its block, it generates Key/Value (KV) pairs. These aren’t shared again — just a compressed summary is passed forward. That saves bandwidth and keeps things lean.</li><li>By processing blocks locally and avoiding full sequence attention, Star Attention reduces computational complexity from quadratic (O(N²)) to linear (O(N)) with respect to context length, as confirmed by the paper.</li></ul><h4>Phase 2: Query Generation</h4><p>Now comes the part of actual generation.</p><ul><li>The query is replicated and broadcasted to all compute hosts.</li><li>Each host uses its cached KV pairs to do its part of the thinking.</li><li>Then, a single “query host” steps in to unify the results using a distributed softmax. It doesn’t ask for everything — just a lightweight vector and scalar from each host.</li><li>With this efficient exchange, it computes the global attention and pops out the next token.</li></ul><p>Despite all the parallelization and compression under the hood, the model still sees the full picture and responds intelligently.</p><p>Let me break it down with also with an analogy (because honestly, the technical lingo can be a bit much — I’ve been there):</p><p><em>Imagine you’re reading a huge book, but instead of reading it all at once, you split it into chapters. Star Attention does this by dividing the text into smaller chunks called blocks. Each block gets a special helper called the “anchor block,” which is like the book’s table of contents — it helps each chunk stay connected to the whole story. These blocks are sent to different computers, which work on their chunk independently, saving time and memory. Each computer creates a summary (called Key/Value pairs) of its chunk, but only keeps the important parts, making the process super efficient.</em></p><p><em>Now, when you ask the AI a query, it sends that question to all the computers holding the chunks. Each computer checks its chunk’s summary to see what’s relevant. Then, one main computer, like a team leader, collects short updates (just a small piece of data) from each computer and combines them to give you the answer. This teamwork makes the AI answer quickly, even for very long texts.</em></p><p>By splitting inference into these two phases, Star Attention finds that sweet spot between staying smart and running fast. No retraining, no compromise — just efficient long-context inference.</p><h3>Star Attention Real World LLM Inference</h3><p>Here’s a simplified and guided walkthrough to help you get started with Star Attention for scalable LLM inference.</p><p>The GitHub repo <a href="https://github.com/NVIDIA/Star-Attention">NVIDIA/Star-Attention</a> provides the building blocks for implementing this attention mechanism, which isn’t available by default in common inference engines like vLLM or TGI. If you’re like me and found it tricky translating these research ideas into something that actually runs, this is for you.</p><h4>Step 1: Clone the Repo &amp; Set Up Environment</h4><p>First, clone the repo and set up a Python 3.10 environment to avoid compatibility issues.</p><p>Do a git clone of repository and follow the following steps to activate a conda environment with python3.10 version (to avoid any conflicts to run this)</p><pre># Clone and install<br>git clone https://github.com/NVIDIA/Star-Attention<br>cd Star-Attention</pre><p>Activate your preferred Python environment, then install dependencies:</p><pre>pip install -r requirements.txt</pre><h4>Step 2: Download the Model</h4><p>Use the download_hf_model.py script to download the desired model from Hugging Face. For example, to download the LLaMA 370B model.</p><blockquote>A heads up this is a non-quantized 70B LLAMA model so 3–4 A100s or H100s GPUs should be required for efficient inference. Prepare that environment, if you want to start with a small experiment just go with a 8B model perhaps.</blockquote><pre>python download_hf_model.py \<br>  -n meta-llama/Meta-Llama-3-70B \<br>  -d ./hf_models \<br>  -t &lt;your_huggingface_token&gt;</pre><ul><li>-n: Specifies the model name on Hugging Face.</li><li>-d: Specifies the directory to download the model into.</li><li>-t: Your Hugging Face access token.</li></ul><p>This will download the model to ./hf_models/meta-llama/Meta-Llama-3-70B</p><h4>Step 3: Build a Simple Inference API</h4><p>Below is a basic FastAPI server that handles inference requests using Star Attention:</p><pre>from fastapi import FastAPI, HTTPException<br>from pydantic import BaseModel, Field<br>from typing import List, Optional<br>import json<br>import os<br>import subprocess<br>import uuid<br><br>app = FastAPI()<br><br># Define the input schema using Pydantic<br>class InferenceInput(BaseModel):<br>    input_context: str = Field(..., description=&quot;Long context portion of the input sample&quot;)<br>    input_query: str = Field(..., description=&quot;Query portion of the input sample&quot;)<br>    index: Optional[str] = Field(None, description=&quot;Sample index&quot;)<br>    output: Optional[str] = Field(None, description=&quot;Expected output response&quot;)<br><br>@app.post(&quot;/inference/&quot;)<br>async def run_inference(inputs: List[InferenceInput]):<br>    # Generate unique filenames for input and output<br>    input_filename = f&quot;input_{uuid.uuid4().hex}.jsonl&quot;<br>    output_filename = f&quot;output_{uuid.uuid4().hex}.jsonl&quot;<br><br>    try:<br>        # Write the input data to a JSONL file<br>        with open(input_filename, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>            for item in inputs:<br>                f.write(json.dumps(item.dict()) + &quot;\n&quot;)<br><br>        # Run the inference script<br>        subprocess.run([<br>            &quot;python&quot;, &quot;run_star_attn_inference.py&quot;,<br>            &quot;--model_path&quot;, &quot;/models/llama70b&quot;,<br>            &quot;--attn_type&quot;, &quot;star&quot;,<br>            &quot;--block_size&quot;, &quot;1024&quot;,<br>            &quot;--tokens_to_generate&quot;, &quot;150&quot;,<br>            &quot;--stop_words&quot;, &quot;&lt;/s&gt;&quot;,<br>            &quot;--input_path&quot;, input_filename,<br>            &quot;--output_path&quot;, output_filename<br>        ], check=True)<br><br>        # Read and return the output<br>        results = []<br>        with open(output_filename, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>            for line in f:<br>                results.append(json.loads(line))<br><br>        return {&quot;results&quot;: results}<br><br>    except subprocess.CalledProcessError as e:<br>        raise HTTPException(status_code=500, detail=f&quot;Inference script failed: {e}&quot;)<br>    except Exception as e:<br>        raise HTTPException(status_code=500, detail=str(e))<br>    finally:<br>        # Clean up temporary files<br>        if os.path.exists(input_filename):<br>            os.remove(input_filename)<br>        if os.path.exists(output_filename):<br>            os.remove(output_filename)</pre><h4>Step 4: Containerize with Docker</h4><p>To scale this setup, containerize the service —</p><pre>FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04<br><br># Set environment variables<br>ENV DEBIAN_FRONTEND=noninteractive<br><br># Install system dependencies<br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    python3.10 python3-pip git wget &amp;&amp; \<br>    rm -rf /var/lib/apt/lists/*<br><br># Set Python aliases<br>RUN ln -s /usr/bin/python3.10 /usr/bin/python &amp;&amp; \<br>    ln -s /usr/bin/pip3 /usr/bin/pip<br><br># Set working directory<br>WORKDIR /app<br><br># Copy application code<br>COPY . /app<br><br># Install Python dependencies<br>RUN pip install --upgrade pip &amp;&amp; \<br>    pip install -r requirements.txt<br><br># Expose port<br>EXPOSE 8000<br><br># Command to run the application<br>CMD [&quot;uvicorn&quot;, &quot;app:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8000&quot;]</pre><h4>Step 5: Test the Endpoint</h4><p>You can test the deployed endpoint using curl:</p><pre>curl -X POST &quot;http://localhost:8000/inference/&quot; \<br>     -H &quot;Content-Type: application/json&quot; \<br>     -d &#39;[<br>           {<br>             &quot;input_context&quot;: &quot;Once upon a time...&quot;,<br>             &quot;input_query&quot;: &quot;What happened next?&quot;<br>           },<br>           {<br>             &quot;input_context&quot;: &quot;In a galaxy far, far away...&quot;,<br>             &quot;input_query&quot;: &quot;Who was the main character?&quot;<br>           }<br>         ]&#39;<br></pre><p>The repo also includes extended scripts integrating Star Attention with Flash and Ring Attention: <a href="https://github.com/NVIDIA/Star-Attention/tree/main/star_attention">Star Attention Extensions</a></p><p>With this setup, you’re now ready to deploy smarter, faster, long-context inference using Star Attention. The hard part in a cutting edge tech is to connect the dots from research to production — hopefully this saves you that learning curve.</p><h3>Balancing Speed and Accuracy in Star Attention</h3><p>Star Attention divides long sequences into smaller blocks for scalable inference. But how you configure those blocks — especially the size — directly affects both speed and accuracy.</p><p><strong>Block Size: What It Impacts</strong></p><ul><li>Larger blocks capture more long-range context, which helps with accuracy.</li><li>But they also increase computational cost and slow down inference.</li><li>Smaller blocks reduce processing time and memory use, but may slightly compromise accuracy.</li></ul><p><strong>Rule of Thumb (Research Suggested):</strong> Set block size to roughly 25% of the total sequence length. For ultra-long sequences (128K+ tokens), fixing it at 32K is a good speed-first choice with minimal accuracy loss.</p><h3>Fine-Tuning Anchor Blocks</h3><p>Anchor blocks act as the shared brain that ties all the other blocks together. Tuning them is key to keeping local context aligned with the bigger picture.</p><p><strong>Anchor Block Size</strong></p><ul><li>Matching the anchor block size to the regular block size tends to maintain accuracy best.</li><li>Shrinking the anchor can make things faster but risks losing critical global context.</li></ul><p><strong>Anchor Block Placement and Content</strong></p><ul><li>Typically, the first block is duplicated as the anchor for all others.</li><li>Make sure this block contains a strong summary or core of the overall sequence. It’s the reference point every other block uses to stay aligned.</li></ul><p>Tweak these based on what matters more to your use case — speed or precision. Star Attention gives you the knobs. You decide how to tune them.</p><h3>Wrapping It Up: Star Attention in the Real World</h3><p>Star Attention makes “long context” a cost‑linear problem. By chopping up the context into blocks and using anchor blocks to keep everything connected, it speeds up inference without losing much accuracy. It’s a solid choice for scaling up models without overloading your hardware.</p><p>But, like any tool, it’s not without its quirks. If your tasks demand a deep, nuanced understanding across the entire sequence, Star Attention might miss some details. Also, dialing in the right block sizes and anchor content takes some trial and error to hit best trade-off between speed and accuracy.</p><p>And here’s a real-world snag: no major inference engine supports Star Attention out of the box yet. TensorRT-LLM and others mostly stick to well-established methods like Flash Attention or paged attention because they’re tested and stable. Since Star Attention is comparatively new — introduced only in late 2024 — it’ll take some time for these engines to adopt and integrate it. That said, TensorRT might explore it in the future as Star Attention gains more attention (pun intended).</p><p><em>Happy scaling, and may your sequences grow long while your invoices stay short.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b95a81b827ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KV Cache Secrets: Boost LLM Inference Efficiency]]></title>
            <link>https://medium.com/@shoa.devs/kv-cache-secrets-boost-llm-inference-efficiency-ae6c53968857?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae6c53968857</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Thu, 26 Dec 2024 22:13:37 GMT</pubDate>
            <atom:updated>2024-12-26T22:13:37.129Z</atom:updated>
            <content:encoded><![CDATA[<p>Deploying a Large Language Model isn’t just about generating responses. It requires behind-the-scenes engineering, especially for self-managed deployments. And scaling your LLM to many users comes with the challenge of managing concurrent queries, dodging crippling latencies, and keeping the seamless user experience intact — all without burning a hole in your wallet from the skyrocketing computational costs.</p><p>Now, that sounds intense, right? Hence, you need an optimized and smooth LLM inference process. And here’s the kicker: sometimes, all it takes is mastering KV Cache management to make a world of difference. Yes, it’s that powerful. So let’s cache some of its optimization tricks into your memory!</p><blockquote>During my research, I uncovered a ton of techniques around optimizations, far too many for a single blog. So, this will be the first in a series where I’ll share my findings.</blockquote><blockquote><strong>What to Expect:</strong></blockquote><blockquote><strong>This Blog:</strong> Focuses on common attention mechanisms for memory management in inference stage where no retraining or fine-tuning LLM is required.</blockquote><blockquote><strong>Future Blogs:</strong> More enhanced attention mechanisms, compression, eviction strategies, and workflow efficiencies for LLM applications using KV Cache.</blockquote><h3>Background on Attention Mechanism</h3><p>You’ve likely heard, “Attention is all you need” for LLMs to come to existence. Before diving into KV Cache, let’s quickly revisit what is Attention and why do we exactly need it? If you’re already an expert on this, feel free to skip ahead to the next section.</p><p>Attention is the backbone of LLMs, enabling them to dynamically process context and capture dependencies between tokens, regardless of their position in a sequence. This ability is what empowers LLMs to generate coherent and contextually accurate outputs.</p><p>In simple terms, attention helps the model grasp the meaning of words by analyzing their context — considering nearby words and sentence structure.</p><p>It does so by creating three matrices for every input token: <strong>Queries (Q)</strong>, <strong>Keys (K)</strong>, and <strong>Values (V)</strong>.</p><ul><li><strong>Keys (K) : </strong>Keys represent the context of each token. They act as a unique “fingerprint” for every token in the input sequence. Each key encodes information about its token’s role or importance in relation to the entire sequence.</li><li><strong>Values (V): </strong>Values represent the actual information that the model might need to retrieve when focusing on a token. While <strong>keys</strong> indicate <em>what a token can provide</em>, <strong>values</strong> contain the <em>content that will be delivered</em> if that token is deemed relevant. For example, in the sentence “The cat sat on the mat”, the value for “cat” would include detailed information that the model uses to construct meaningful relationships with other tokens, like “sat” or “mat.”</li><li><strong>Queries (Q): </strong>Determine which <strong>keys</strong> (and their corresponding values) are most relevant for a given token. They act as “search requests” generated for each token in the sequence. A <strong>query</strong> specifies <em>what this token is looking for</em> in other tokens to build context. For example, when processing the word “sat,” the query might be searching for the subject (like “cat”) and the location (like “mat”). The <strong>query</strong> interacts with the <strong>keys</strong> of all tokens in the sequence to determine how strongly “sat” should pay attention to each of them.</li></ul><p><strong>How Queries, Keys, and Values Work Together?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pXhafYrPaIkcnEAy" /></figure><p>The self-attention mechanism uses these three matrices as follows:</p><ol><li>The <strong>query</strong> of the current token is compared to the <strong>keys</strong> of all tokens in the sequence using a dot product, which produces a score indicating the relevance of each token.</li><li>These scores are then normalized to calculate attention weights.</li><li>The attention weights are applied to the corresponding <strong>values</strong>, and the weighted sum of these <strong>values</strong> is returned as the output for the token.</li></ol><p>In simple terms:</p><ul><li><strong>Queries</strong> ask, “What do I need to know?”</li><li><strong>Keys</strong> answer, “Here’s what I can tell you.”</li><li><strong>Values</strong> deliver, “Here’s the actual information you requested.”</li></ul><p>This mechanism allows the model to dynamically adjust its focus and interpret relationships between tokens, making it capable of handling context-dependent tasks like generating coherent text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*ebq03VfDv-n04p_K" /></figure><h4><strong>Inference is split into two phases:</strong></h4><ol><li><strong>Pre-fill Phase</strong>: This is when the model processes the input sequence. During this phase, the attention mechanism generates keys and values for all tokens in the input and stores them in the <strong>KV Cache</strong>. This is the key process that saturates the GPU utilization.</li><li><strong>Decoder Phase</strong>: When generating tokens one at a time (auto-regressively), the model uses the cached <em>keys</em> and <em>values</em> from the <strong>pre-fill phase</strong> instead of recomputing them. This means the model only needs to compute the <em>query</em> for the new token and compare it against the stored <em>keys</em> to fetch the relevant <em>values</em>. This phase is memory-intensive, with latency stemming from data transfer (keys, values, activations, and weights) rather than computation Efficient memory management and optimizations happen at this stage.</li></ol><p>We can deduce KV Cache’s significant from the above details, but to iterate — <strong><em>KV Cache </em></strong><em>stores</em><strong><em> </em></strong><em>intermediate Key and Value representations used in the self-attention mechanism of decoder-only transformers, which is crucial to avoid re-computation during the decoder phase.</em></p><h3>KV Cache Considerations</h3><p>The Self-Attention Mechanism has quadratic complexity, but KV Cache offers powerful ways to precompute input sequences and speed up inference — it almost feels magical. But, all good things come with some trade-offs:</p><ul><li><strong>Memory vs. Compute:</strong> KV Cache grows linearly with batch size and sequence length, leading to significant memory overhead for long conversations, which can be challenging with extended context windows. So, there’s a decision in making memory vs. compute more efficient.</li><li><strong>Disk Overhead in Traditional Systems:</strong> KV Cache is often stored on external disks to address GPU memory limitations and reduce costs. However, traditional prefix caching mechanisms can suffer from high latency due to I/O bottlenecks when transferring KV Cache from disk to GPU memory.</li><li><strong>Network Overhead in Distributed Systems:</strong> Fetching large KV Cache tensors for long contexts over a network may lead to delays, hindering real-time inference.</li></ul><p>These considerations give rise to the need of improved attention mechanisms…</p><h3>Efficient Attention Mechanisms</h3><h4>Paged Attention</h4><ul><li>Paged Attention is inspired by <strong>CPU page memory mechanisms</strong>.</li><li>It maps the KV Cache from contiguous storage to <strong>discontinuous GPU memory using a mapping table</strong></li><li>Employs <strong>custom CUDA kernels</strong> to generate the next token during inference efficiently.</li><li><strong>Custom CUDA kernels</strong> streamline these memory lookups, ensuring the scattered KV Cache is fetched efficiently without unnecessary latency or overhead. These kernels also handle computations for generating next tokens in parallel across thousands of GPU threads.</li><li>Supports <strong>just-in-time buffer allocation</strong> instead of ahead-of-time when starting a new generation, i.e, the framework doesn’t need to allocate contiguous buffer of maximum context length size. The scheduler can decide the storage requirement for generation on the fly without any degradation. This significantly improves upon memory wastage from ahead-of-time allocation schemes.</li><li>Adopted by inference engines like <strong>vLLM</strong> and <strong>TensorRT</strong></li><li>Compatible with usage of other techniques as well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q5KShXObf_bN2gJQ.gif" /><figcaption>vLLM Using PagedAttention</figcaption></figure><p><strong>Considerations</strong>:</p><ul><li>While memory utilization is significantly improved, the approach doesn’t guarantee perfect utilization, especially in edge cases of mixed lengths of input sequences.</li><li>The last allocated page often contains unused space because the exact memory required for each generation may not perfectly align with page sizes, which may cause a minimal wastage. (Memory wastage is limited to under 4% (as noted in the <a href="https://blog.vllm.ai/2023/06/20/vllm.html">vLLM blog</a>))</li></ul><p>Paged Attention represents a significant step forward in optimizing KV Cache usage, making it a compelling choice for modern LLM inference systems aiming for high efficiency and scalability.</p><h4>Disk Attention</h4><ul><li>The Disk Attention Mechanism manages the KV Cache by <strong>offloading it to disk</strong> rather than being limited to GPU/CPU memory, enabling distributed deployment across multiple servers.</li><li>By leveraging disk-based storage, it allows for handling much larger context windows and sequences.</li><li>It is ideal for large-scale providers who need to serve high concurrency while maintaining cost efficiency.</li></ul><p><strong>Considerations</strong></p><ul><li>Possible network latency from fetching data from remote disk</li><li>Disk I/O speeds are typically much slower than memory access speeds, which can limit throughput in scenarios requiring frequent KV Cache updates or retrievals.</li></ul><p>To mitigate these latencies, multi-tier caching system (e.g., using a small portion of GPU memory for hot cache) can be implemented.</p><h4>Sliding Window Attention</h4><ul><li>Optimizes memory and computational efficiency by <strong>focusing attention on a fixed window of tokens</strong> while retaining some access to earlier context.</li><li>Enables <strong>near-infinite context</strong> handling</li><li>Focusing attention on a <strong>fixed window of recent tokens</strong> makes it ideal for streaming or real-time applications.</li><li>It <strong>offloads all units on CPU memory</strong> and dynamically retains the <strong>frequently used units on GPU memory</strong>, significantly reducing GPU memory usage</li><li><strong>Reduces complexity to O(nw)</strong>, where <em>w</em> is the window size, making it scalable for long-context or continuous input streams.</li></ul><p><strong>Partial retention of earlier context</strong>:</p><ul><li>Values from outside the active window are offloaded to CPU memory.</li><li>A <strong>representation of earlier keys</strong> is retained to allow selective activation of previous memory blocks when needed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/0*45Tm85-T9kUcHEV3.png" /></figure><p><strong>Considerations</strong>:</p><ul><li><strong>Limited Global Context</strong>: Only retains a partial representation of the global context outside the active window, which may affect tasks requiring full-sequence context awareness.</li><li><strong>CPU Memory Overhead</strong>: Offloading KV Cache to CPU memory increases reliance on slower memory, potentially impacting latency when accessing earlier blocks.</li><li><strong>Selective Activation Overhead</strong>: Activating earlier memory blocks adds computational and memory transfer delays, especially in high-throughput scenarios.</li></ul><p>Well-suited for <strong>real-time and streaming LLM systems</strong>, such as chatbots, live transcription, and conversational AI, where maintaining a moving window of recent context is sufficient.</p><h4>Flash Attention</h4><ul><li>Flash Attention is an optimized <strong>exact</strong> attention mechanism that is <strong>fast, memory-efficient, and IO-aware</strong>, leveraging the GPU memory hierarchy.</li></ul><p><strong>Exact</strong>: Achieves the same output as traditional attention mechanisms while significantly reducing memory overhead.</p><p><strong>Fast and Memory-Efficient: </strong>Breaks input sequences to smaller chunks of the sequence sequentially, avoiding the need to store the entire attention matrix in memory.</p><p><strong>IO-Awareness</strong>: It considers the memory hierarchy of GPUs rather than treating hardware as a black box.</p><p><strong>On-the-Fly Computation</strong>: Computes attention scores as needed, rather than precomputing and storing them.</p><p><strong>Optimized CUDA Kernels</strong>: It uses highly optimized CUDA kernels to fuse multiple operations (e.g., softmax and matrix multiplication) into a single GPU kernel call. This reduces memory bandwidth usage and improves computational efficiency.</p><p><strong>Blockwise Softmax: </strong>It processes chunks of the sequence using a blockwise softmax, which ensures numerical stability and avoids overflow/underflow issues.</p><p><strong>Tiling</strong>: It uses “tiling” to fully compute and write out a small part of the final matrix at once, rather than doing part of the computation on the whole matrix in steps, writing out the intermediate values in between.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fsdltwCabTFx5EaH.png" /><figcaption><a href="https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad">Image Citation</a></figcaption></figure><ul><li>Used by inference engines like <strong>vLLM</strong> and <strong>TensorRT</strong> LLM in combination with paged attention.</li></ul><p><strong>Considerations</strong>:</p><ul><li><strong>Sequential Processing</strong>: Sequential chunk processing may introduce slight latency compared to fully parallel operations.</li></ul><p>It is ideal for scenarios requiring exact attention with minimal memory usage and maximum speed. It’s especially well-suited for applications running on high-performance GPUs.</p><p>(For a deeper dive in Flash Attention — <a href="https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad">Step by Step Flash Attention</a>. There have also been further modifications in Flash Attention, you can checkout here — <a href="https://arxiv.org/abs/2407.08608">Flash Attention 3</a>)</p><h4>Chunk Attention</h4><ul><li>Chunk Attention optimizes the use of KV Cache for dialogue-based interactions by <strong>identifying reusable portions of cached data</strong>.</li><li><strong>Leverages shared tokens or prompts</strong>, example long, consistent system prompts or shared context across requests</li><li>Maintains a <strong>dictionary tree of all historical dialogues</strong> to identify the longest common prefix for a new request.</li><li><strong>Reuses KV Cache</strong> for this prefix to reduce redundant computations in pre-fill stage.</li><li><strong>Reduces the Time to First Token (TTFT)</strong>, especially for requests with consistent, lengthy prompts.</li><li>Compatible with other techniques like Paged Attention</li></ul><p><strong>Considerations</strong>:</p><ul><li><strong>Dependency on Consistency</strong>: Effectiveness relies on the presence of consistent tokens or prompts across dialogues. Highly varied inputs may limit its benefits.</li><li><strong>Overhead for Dictionary Tree Maintenance</strong>: Maintaining and searching the dictionary tree adds a small computational overhead, though this is outweighed by the gains in most cases.</li></ul><h4>Radix Attention</h4><p>Radix Attention is a highly efficient mechanism for <strong>automated KV cache reuse </strong>across multiple LLM generation calls.</p><p><strong>Automatic KV Cache Reuse</strong>: It allows different prompts with the same prefix to share the intermediate KV cache automatically, preventing redundant memory usage and computation. This is done seamlessly, unlike some systems where manual configurations are needed for cache reuse.</p><p><strong>Radix Tree for Space and Time Efficient Prefix Matching</strong></p><ul><li>The <strong>KV cache is stored in a radix tree</strong>, which helps manage the mapping between sequences of tokens (keys) and their corresponding KV cache tensors (values).</li><li>The radix tree allows for <strong>efficient prefix search, insertion, and eviction in O(k) where k is the length of tokens</strong>.</li><li>Radix trees are <strong>more space-efficient</strong> than balanced trees (like binary trees or tries) because they allow sequences of elements to be stored on edges, reducing the number of nodes and comparisons. This is beneficial for handling long common prefixes in token sequences, which are common in language models.</li><li>Requires perform fewer comparisons compared to balance trees, especially for long common prefixes, improving lookup efficiency.</li></ul><p><strong>GPU Memory Optimization</strong>:</p><ul><li>It combines existing techniques like <strong>continuous batching and paged Attention Mechanism</strong> as well to optimize GPU memory utilization.</li><li>A <strong>Least Recently Used (LRU)</strong> eviction policy is used to maintain cache efficiency. (More Details on Eviction Policies for KV Cache in later series).</li></ul><p><strong>Multi-Modal Support</strong>: Radix Attention can be extended to handle multi-modal models, including image tokens, making it versatile for various applications.</p><ul><li>Radix Attention and Chunk Attention both optimize KV cache reuse for memory efficiency, but Radix Attention uses a radix tree for efficient token-to-KV cache mapping, enabling it to handle large-scale, varied prompts with specific prefix matches. In contrast, Chunk Attention uses a dictionary tree to manage shared prefixes across dialogues, focusing on segmenting and reusing consistent dialogue patterns. While Radix Attention is more flexible and suited for diverse input scenarios, Chunk Attention excels in environments with long, consistent system prompts.</li><li><strong>SG Lang</strong> inference uses Radix Attention</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkIn2i9bIznNPo8ziuBVmA.png" /></figure><p><strong>Considerations</strong>:</p><ul><li><strong>Eviction Limitations</strong>: While the LRU policy helps manage memory, there could be instances where older tokens are evicted prematurely, leading to a potential reduction in cache hit rates for subsequent requests.</li><li><strong>Overhead for Tree Maintenance</strong>: While the radix tree structure is space-efficient, there is some small maintenance overhead for managing the tree on the CPU. This overhead is minimal compared to the benefits of cache reuse.</li><li><strong>Limited Performance Improvement in Edge Cases</strong>: Radix Attention is optimized for handling common token prefixes, but performance might degrade in cases with varied input sequences.</li></ul><h3>Conclusion</h3><p>As LLMs evolve with increasingly longer context capacities, making inference memory and compute efficient for extended contexts has become crucial. So, the techniques discussed above have been our heroes. There are many amazing inference engines which already utilize some of the approaches discussed above. But now that you have this knowledge, you’re empowered to go beyond — customizing and combining these methods to suit your application’s unique needs. And this is just the beginning! In future, I’ll dive into even more advanced techniques which you can experiment with to push the boundaries of your application’s or agent’s efficiency.</p><h3>References</h3><ul><li><a href="https://arxiv.org/abs/2407.18003">Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption</a></li><li><a href="https://medium.com/@joaolages/kv-caching-explained-276520203249">https://medium.com/@joaolages/kv-caching-explained-276520203249</a></li><li><a href="https://developer.nvidia.com/blog/5x-faster-time-to-first-token-with-nvidia-tensorrt-llm-kv-cache-early-reuse/">https://developer.nvidia.com/blog/5x-faster-time-to-first-token-with-nvidia-tensorrt-llm-kv-cache-early-reuse/</a></li><li><a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/">https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/</a></li><li><a href="https://lmsys.org/blog/2024-01-17-sglang/">https://lmsys.org/blog/2024-01-17-sglang/</a></li><li><a href="https://arxiv.org/abs/2205.14135">Flash Attention Research Paper</a></li></ul><blockquote>I hope you found today’s content insightful and resourceful. If you have any questions, suggestions, or ideas to improve the blog, I’d love to hear from you! Feel free to drop a comment below or connect with me on my socials:</blockquote><blockquote>Linkedin: <a href="https://www.linkedin.com/in/shoa-aamir/">www.linkedin.com/in/shoa-aamir/</a></blockquote><blockquote>Twitter/ X: <a href="https://x.com/ShoaAamir">x.com/ShoaAamir</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae6c53968857" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding LLM Quantization]]></title>
            <link>https://medium.com/@shoa.devs/understanding-llm-quantization-39d6330885dc?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/39d6330885dc</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm-optimization]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Thu, 26 Sep 2024 19:27:30 GMT</pubDate>
            <atom:updated>2024-10-20T00:25:45.606Z</atom:updated>
            <content:encoded><![CDATA[<p>With the surge in applications using Large Language Models (LLMs), optimizing their performance has become more important than ever — and that can’t happen without looking into LLM quantization. Hence, I started to explore it as well for my work, and the deeper I went, the more I uncovered a range of quantization types and schemes, each with its own quirks. To save you the hassle of diving through it all yourself, I’ve gathered my notes and put them into this blog. I hope you find it insightful and beneficial!</p><h3>What is Quantization?</h3><p>A <strong>Large Language Model (LLM)</strong> is fundamentally characterized by a collection of weights and activations. Typically, these weights are represented in formats like <strong>float32</strong> or <strong>Bfloat16</strong> to optimize computation and maintain precision.</p><blockquote><strong>Note on Bfloat16: </strong>It is<strong> </strong>brain floating point developed by google which retains the dynamic range of <strong>float32</strong> but with reduced precision of <strong>float16</strong>. It is used during both training and inference to reduce memory consumption and computation time, particularly without the same loss in dynamic range that occurs with other 16-bit formats.</blockquote><p><strong>Quantization</strong> of an LLM refers to the process of compressing the number of bits used to represent its weights. This compression involves mapping larger floating-point weights to a smaller floating range or a discrete set through the process of scaling. The primary goal of this quantization is to reduce computational requirements, although it does come with a trade-off in accuracy.</p><blockquote>Following is an image depicting the quantization of value of <strong>Pi</strong> from <strong>Float 32</strong> to <strong>Float 16</strong> and the consequent reduced memory requirement:-</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*r76pPxLZ7rSl6DXK.png" /><figcaption><a href="https://medium.com/@mahmoud.bidry11/demystifying-quantization-a-clear-guide-to-understanding-the-concept-and-methods-in-large-language-aa407903da1d#:~:text=GPTQ%20converts%20the%20model&#39;s%20parameters,quantized%20to%204%2Dbit%20precision.">Image Source</a></figcaption></figure><h3>Types of Quantizations</h3><h4>Post Training Quantization (PTQ)</h4><ul><li>Quantization of weights done on pretrained LLMs.</li><li>Lower implementation complexity.</li><li>Commonly utilized for efficient deployment post-training.</li><li>Can lead to accuracy loss, particularly in sensitive models, due to reduced weight precision.</li></ul><p><strong>Types of PTQ</strong></p><ul><li><strong>Dynamic Quantization:</strong> Weights are quantized dynamically during inference, while activations remain unchanged. This approach helps avoid increased noise and accuracy degradation, as activation values fluctuate based on input.</li><li><strong>Static Quantization: </strong>Both weights and activations are quantized to lower precision before inference.</li><li><strong>Per-Tensor/Per-Channel Quantization: </strong>Quantization hapens uniformly per whole tensor or channel.</li></ul><h4>Quantization Aware Training (QAT)</h4><ul><li>In this method, gradients are calculated with high precision, while forward passes (activations and weights) utilize simulated lower precision, enabling the model to adapt and learn to mitigate quantization errors.</li><li>Weight conversions process integrated during training of LLM.</li><li>Generally causes lesser accuracy degradation than Post Training Quantization</li><li>Involves higher implementation complexity and longer training time due to the need for quantization awareness during training.</li><li>Highly used QAT technique is <strong>QLoRA.</strong></li></ul><blockquote><strong>Brief Note on QLoRA</strong>: Q stands for the Quantization whereas <strong>LoRA</strong> is a fine-tuning technique that freezes most of the pre-trained model’s weights and only learns low-rank updates for a subset of the model’s parameters, typically focusing on attention layers. This reduces the number of trainable parameters and memory usage during fine-tuning.</blockquote><ul><li>Used when accuracy and consistency is critical in the application.</li></ul><h3>Post-Training Quantization Techniques</h3><p>Here are some of the techniques used currently to implement <strong>PTQ</strong>.</p><h4>NF4</h4><ul><li>Stands for 4 bit Normal Float</li><li>Compresses to 4 bit precision</li><li>Significantly degrades accuracy but improves performance.</li></ul><h4>GPTQ</h4><ul><li>Quantizes each row of the weight matrix independently to minimize error.</li><li>Initially quantized to 4-bit integers, then dequantized to <strong>fp16</strong> in a fused kernel during inference.</li><li>Focuses on quantization for GPU inference and performance.</li><li>Implementation using the libraries <a href="https://github.com/PanQiWei/AutoGPTQ">AutoGPTQ</a>, <a href="https://github.com/turboderp/exllama">ExLlama</a> and <a href="https://github.com/qwopqwop200/GPTQ-for-LLaMa">GPTQ-for-LLaMa</a></li></ul><h4>GGUF (GPT-Generated Unified Format) / GGML (GPT-Generated Model Language)</h4><ul><li>Works by assigning varying bit-widths to different weight groups based on their gradient magnitudes.</li><li>It features a unique binary format for LLMs optimized for quick loading and saving of models, making it highly efficient for inference.</li><li>GGML has transitioned to GGUF for improved extensibility and compatibility.</li><li>GGUF eliminates breaking changes, it eases transitions to newer versions and supports a wide range of models and architectures.</li><li>Compatible with GPU as well as CPU</li><li>Works closely with <a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> for quantization.</li></ul><h4>AWQ (Activation-Aware Weight Quantization)</h4><ul><li>Focuses solely on dynamic quantization of weights.</li><li>designed to be aware of both weights and activations, leading to minimal accuracy degradation.</li><li>Does not quantize all weights; instead, it preserves a small percentage of critical weights essential for LLM performance.</li><li>Significant speed up and accuracy maintenance compared to GPTQ</li><li>Several libraries for quantizing models with the AWQ algorithm, such as <a href="https://github.com/mit-han-lab/llm-awq">llm-awq</a>, <a href="https://github.com/casper-hansen/AutoAWQ">autoawq</a> or <a href="https://huggingface.co/docs/optimum/main/en/intel/optimization_inc">optimum-intel</a>.</li></ul><h3>Common Quantization Schemes</h3><p>There are many quantization schemes to consider. Once one understands the basics of quantization and numerical representations, it’ll make more sense. I’ve laid out the key schemes here.</p><h4>Floating Point Quantization</h4><ol><li><strong>FP32 (32-bit floating point)</strong>: Standard full precision floating point representation.</li><li><strong>FP16 (16-bit floating point)</strong>: Half precision, commonly used to speed up training and inference, especially on GPUs.</li><li><strong>FP8 (8-bit floating point):</strong> Floating-point representation using 8 bits</li><li><strong>bfloat16 (Brain Float 16)</strong>: As mentioned above, 16-bit float with a larger range, used to improve model accuracy while reducing precision requirements.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*DM91TKHpJ015fQxE.png" /></figure><h4>Integer Quantization</h4><ul><li>When you see quantizations like <strong>INT8, INT4,</strong> etc., these refer to integer-based quantizations.</li><li>Weights are quantized to represent values in discrete fixed-point formats.</li><li>There is a higher risk of accuracy degradation due to the scaling of floating-point numbers to discrete values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*jTtn5nSppTynwowd.png" /><figcaption><a href="https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization">Image Source</a></figcaption></figure><h4>Quantized Variants of GGUF/GGML Quantization</h4><p>In GGUF quantized LLMs, you may encounter various quantization formats such as Q8, Q5, Q4, etc. These are primarily integer-based quantizations but can have various variants that might cause confusion. Below are the types:</p><p><strong>Type-0 (eg: Q4_0, Q5_0)</strong></p><ul><li>Weights are quantized uniformly without any advanced scaling or optimization techniques.</li><li>Weights <em>w</em> are obtained from quants <em>q</em> using <em>w = d * q</em>, where <em>d</em> is the block scale</li><li>Simple to implement</li><li>May suffer from higher accuracy loss as it ignores weight distribution.</li></ul><p><strong>Type-1 (eg: Q4_1 , Q5_1)</strong></p><ul><li>Each channel (or group of weights) is scaled individually during quantization.</li><li>Weights are given by <em>w = d * q + m</em>, where <em>m</em> is the block minimum.</li><li>Allows for finer-grained quantization, reducing the impact of weight distribution on performance.</li><li>Improved accuracy over<strong> Type-0.</strong></li></ul><p><strong>K-Quant Models</strong></p><ul><li>A more advanced technique where weights are quantized in groups with dedicated scaling factors for each group, adapting the quantization to different layers or regions of the model.</li><li>This method offers higher accuracy than <strong>Type-0</strong> and <strong>Type-1</strong> due to dynamic scaling</li><li>The format may include a suffix (l, m, or s) such as <strong>q4_k_l, q4_k_m, or q4_k_s</strong>, indicating model sizes of Large, Medium, and Small, respectively.</li></ul><h4>Mixed Precision Quantization</h4><ul><li><strong>TF32 (TensorFloat-32):</strong> Used primarily in NVIDIA Ampere GPUs, TensorFloat-32 uses a mixture of 19-bit for mantissa and 8-bit for exponent to balance performance and accuracy in matrix multiplications.</li><li><strong>Mixed Precision (FP16/INT8): </strong>Combines different bit-widths for different layers or operations, for example, FP16 for weights and INT8 for activations.</li></ul><p>These quantization schemes are employed depending on the quantization techniques, hardware support, model architecture, and the trade-offs between speed, accuracy, and memory usage</p><h3>Memory Requirement of Quantized Models</h3><p>When it comes to quantization, the goal is all about optimizing how we use hardware and memory resources, so knowing how to calculate those needs is essential.</p><p>There is a very simple formula for calculating an approximate of GPU memory usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/0*AXaGKOvBcgIh3phO.png" /><figcaption><a href="https://www.substratus.ai/blog/calculating-gpu-memory-for-llm">Image source</a></figcaption></figure><p>So, if we have a LLAMA 3.1 70B parameters LLM, the original GPU memory requirement would be —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*iJ_9wCtTVTLQB_CIyJfltw.png" /><figcaption><a href="https://www.substratus.ai/blog/calculating-gpu-memory-for-llm">Image source</a></figcaption></figure><p>The original GPU memory requirement is quite hefty — you’d be looking at needing 2 or 3 A100 or H100 GPUs, each with 80 GB of memory.</p><p>Whereas, if we do a 4 bit quantization, if you go for 4-bit quantization, that requirement shrinks significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*TSCN14ATgkI7wLG-ePzO0Q.png" /></figure><p>Only 2 L4s of 24GB would be suffcient to deploy the 4-bit quantized model.</p><h3>Choosing the Right Quantization for LLM</h3><p>It’s all about finding the right fit for your specific needs. Different use cases might call for different quantization methods. For example, if you’re working on a project where speed is crucial, like real-time applications, you might lean towards lower precision formats, such as 4-bit quantization. This will help you achieve faster inference times and save on memory.</p><p>On the other hand, if your model demands high accuracy — say, in healthcare or finance — consider using quantization-aware training (QAT) or higher or mixed-precision quantization. These methods help maintain more precision while still providing the benefits of quantization.</p><p>In a nutshell, the best way to determine which quantization to use is to assess your model’s performance requirements, the hardware you’re working with, the costs, and how much accuracy you’re willing to trade off for speed and efficiency.</p><h3>Conclusion</h3><p>In the end, quantization is a powerful technique that can help you optimize your models for better performance and efficient resource consumption, especially with the growing use of large language models (LLMs). In essence, when comparing models with similar inference costs, larger quantized models can actually outperform their smaller, non-quantized counterparts. This advantage becomes even more pronounced with larger networks, which tend to show less quality loss when quantized.</p><p>By understanding the different quantization schemes and their respective use cases, you can make informed choices that best fit your projects.</p><p>So, whether you’re looking to speed things up or save some memory, there’s likely a quantization method out there that fits the bill. Dive in, experiment, and see what works best for you. Happy quantizing!</p><blockquote>The blog is based on my research on quantization, feel free to drop a comment if anyone spots any misinformation, I’d be happy to correct it!</blockquote><h3>More Resources</h3><ul><li><em>I loved this extensive visual representations of quantization, check it out for more depth into the topic</em> — <a href="https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization">Visual Guide to Quantization</a></li><li><a href="https://huggingface.co/docs/optimum/en/concept_guides/quantization">Hugging face’ guide to Quantization</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39d6330885dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uncovering the Anti-Corruption Layer]]></title>
            <link>https://medium.com/@shoa.devs/uncovering-the-anti-corruption-layer-3e7e06a74c94?source=rss-c9a282dbece6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e7e06a74c94</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[anti-corruption-layer]]></category>
            <category><![CDATA[system-design-patterns]]></category>
            <category><![CDATA[domain-driven-design]]></category>
            <dc:creator><![CDATA[Shoa Aamir]]></dc:creator>
            <pubDate>Fri, 10 Nov 2023 19:39:31 GMT</pubDate>
            <atom:updated>2023-11-10T19:39:31.273Z</atom:updated>
            <content:encoded><![CDATA[<p>In the intricate web of software development, seamless communication and integration between different systems is often a necessity. However, systems may get corrupted when they possess divergent semantics or data models. This is where the Anti-Corruption Layer (ACL) steps in, acting as a guardian at the boundaries of our systems, ensuring that data translation and validation occur smoothly. Let’s delve more into the layers of ACL and explore its technical implementation, use cases, and the considerations one should keep in mind before embracing this powerful design pattern.</p><h3>Table Of Contents</h3><ul><li><strong>Introduction to Anti-Corruption Layer</strong></li><li><strong>Use Cases of ACL Design Pattern</strong></li><li><strong>Challenges in ACL Implementation</strong></li><li><strong>Considerations Before Using ACL Pattern</strong></li><li><strong>Conclusion</strong></li></ul><h3>What is Anti-Corruption Layer exactly?</h3><p>At its core, an Anti-Corruption Layer (ACL) is a system design pattern used to maintain the integrity of business data and models by serving as a bridge between our system and external systems or legacy applications. Its purpose is to isolate the core system from corruption by adeptly translating data and requests and to ensure the decoupling of separate domains or business logics.</p><p>An ACL further consists of the <strong>Facade</strong> and <strong>Adapter</strong> design patterns.</p><h4><strong>Facade Design Pattern</strong></h4><p>The <strong>Façade Design Pattern</strong> acts as a simplified interface, providing a unified and straightforward entry point to a complex system. It abstracts the underlying complexities, offering a concise and user-friendly API. This abstraction not only simplifies client interactions but also decouples the client code from the subsystems, promoting a cleaner and more maintainable codebase.</p><h4>Adapter Design Pattern</h4><p>The <strong>Adapter Design Pattern </strong>serves as a bridge, allowing incompatible interfaces to work seamlessly together. It consists of an adapter class that acts as a translator, converting the interface of one class into a form that another class can understand.</p><p>Hence, an ACL employs an <strong>abstracted interface (Facade) </strong>to simplify interactions with complex subsystems and utilizes an <strong>Adapter</strong> to bridge the interface disparities between different components. Thus, contributing to the creation of a modular, interoperable, and maintainable software architecture.</p><h3>Use Cases of ACL Design</h3><h4>System Migration</h4><p>During phased system migrations, where new and legacy systems need to coexist, ACL can be the intermediary, facilitating smooth communication with the old system until the migration is complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IACxvgsj22ys4Yxnw31mzQ.png" /></figure><p>For instance, consider the evolution of an e-commerce website that initially implemented a simple monolithic service to reach its limited audience at the optimal cost of resources. Then, it transitions from a simpler monolithic service to a dynamic micro-services architecture to meet scaling demands. Amidst this metamorphosis, a challenge arises: how to seamlessly migrate while ensuring uninterrupted operations?</p><p><strong>Enter the Anti-Corruption Layer (ACL).</strong></p><p>During the phased migration, where modern microservices coexist with the legacy monolithic system, ACL acts as the orchestrator. Positioned in between those systems, it facilitates a smooth translation of data models, ensuring that the evolving micro-services can seamlessly understand the language of the legacy system. In essence, ACL becomes the hub, enabling the e-commerce giant to evolve without operational disruptions.</p><h4>Systems Integration</h4><p>When integrating with other systems, especially third-party entities, the semantic and structural differences in data communication can be vast. In such cases, the ACL can reside at the edge of our system, translating and validating incoming data to ensure compatibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*50ayq6conqmWKoAokyxgXw.png" /></figure><p>Building upon our e-commerce website scenario, let’s focus on the crucial aspect of checkouts and payments. To enhance efficiency, the website employs a local micro-service dedicated to managing payment-related data. However, for the actual payment processing, a third-party service like Stripe comes into play.</p><p>The semantics of our local payments micro-service and the external Stripe service may speak different languages. To bridge this semantic gap and ensure seamless integration, ACL takes its position at the edge of our system. Acting as a translator, it refines the requests sent to Stripe and harmonises the data flowing back.</p><p>We have covered multiple benefits of implementing the Anti-Corruption Layer above, but it also brings forth a set of challenges.</p><h3>Challenges in ACL Design Implementation</h3><ul><li><strong>Latency: </strong>Introducing an ACL may introduce latency due if additional service calls for data transformation and mapping are being made.</li><li><strong>Indirection: </strong>The layer adds an extra level of indirection while communicating, which might complicate the system architecture.</li><li><strong>Maintenance and Complexity: </strong>Managing the ACL introduces complexity and maintenance overhead.</li></ul><h3>Considerations before Using ACL Pattern</h3><ul><li><strong>Semantics and Domains:</strong> Evaluate the disparity in semantics and domains; if the complexity of an ACL outweighs adapting one system to another’s semantics, reconsider the implementation.</li><li><strong>Dependency Isolation:</strong> If systems undergo frequent changes, an ACL acts as a protective barrier, reducing the impact of these changes.</li><li><strong>Cost and Time Constraints:</strong> Assess project constraints; if time and resources are limited, simpler integration methods might be more practical.</li><li><strong>High-Performance Requirements:</strong> In high-performance environments, consider direct integration to minimise latency.</li><li><strong>Interim Solutions:</strong> If integration is temporary or one system is slated for phasing out soon, investing in an ACL may not be justified.</li></ul><h3>Conclusion</h3><p>In essence, ACL isn’t a mere necessity; it’s a strategic choice. It’s the assurance of data sanctity in the face of integration challenges. While challenges and considerations surround its implementation, the strategic use of ACL can be a powerful tool in maintaining the integrity of systems and data, fostering seamless communication in the dynamic landscape of software development.</p><h3>References</h3><ul><li><a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer">https://learn.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer</a></li><li><a href="https://www.linkedin.com/pulse/anti-corruption-layer-adapter-patterns-subbu-mahadevan/">https://www.linkedin.com/pulse/anti-corruption-layer-adapter-patterns-subbu-mahadevan/</a></li><li><a href="https://www.youtube.com/watch?v=Dok2Ikcjaro">https://www.youtube.com/watch?v=Dok2Ikcjaro</a></li></ul><p>I hope you took away some knowledge of significance from my article, if yes then feel free to drop a comment, clap👏🏻 and share it.❤</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e7e06a74c94" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>