<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roberto Sanchez on Medium]]></title>
        <description><![CDATA[Stories by Roberto Sanchez on Medium]]></description>
        <link>https://medium.com/@trev21mx?source=rss-f26deb13af12------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*HFNEh66Uas7rrSIgRzkBTg@2x.jpeg</url>
            <title>Stories by Roberto Sanchez on Medium</title>
            <link>https://medium.com/@trev21mx?source=rss-f26deb13af12------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:15:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@trev21mx/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop Guessing Your Prompts]]></title>
            <link>https://medium.com/@trev21mx/stop-guessing-your-prompts-1d14100fc66e?source=rss-f26deb13af12------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d14100fc66e</guid>
            <category><![CDATA[design-of-experiments]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Roberto Sanchez]]></dc:creator>
            <pubDate>Wed, 02 Sep 2026 19:09:50 GMT</pubDate>
            <atom:updated>2026-09-02T19:09:50.391Z</atom:updated>
            <content:encoded><![CDATA[<h4>How 100-Year-Old Statistics Shrank 1,024 Prompt Experiments to 16</h4><p>You’ve probably lived this one.</p><p>You’re tuning a system prompt — for an extraction pipeline, a support agent, a RAG assistant, something that matters. You’ve got a backlog of good-sounding ideas: give the model a senior-auditor persona, add three few-shot examples, put in “think step by step,” enforce a strict JSON schema, tack on a “never hallucinate” guardrail, drop the temperature from 0.7 to 0.1.</p><p>So you do what everyone does: stack all the changes into one prompt, run your 50-case benchmark, and watch the score jump from 0.62 to 0.71.</p><p>Nine percent. That feels like a win.</p><p>Except the next morning you stare at the diff and realize you have no idea which change did it. Was it the persona? The examples? Maybe the “never hallucinate” line quietly hurt accuracy while the few-shot examples carried the whole gain. Maybe the chain-of-thought instruction is just burning extra tokens and latency for nothing. Maybe the entire +9% was a fluke of sampling noise on a 50-item test set.</p><p>Without isolating your variables, you’re not engineering. You’re guessing with a scoreboard.</p><p><strong>Why we can’t just test everything</strong></p><p>The obvious fix is to test every combination. The obvious reason not to is arithmetic.</p><p>Ten binary prompt factors — on or off — means 2¹⁰ = 1,024 prompt variants. Multiply by a 50-item benchmark and a single round of “just be thorough” is 51,200 LLM calls. At typical API pricing that’s a few hundred dollars of compute for one iteration, plus the hours your team waits for it to finish. Do that per team, per prompt, per week, and it stops being an engineering practice and starts being a budget line nobody wants.</p><p>So teams compromise. They change one thing at a time, call it good when the score goes up, and move on. This works sometimes. But one-factor-at-a-time testing is blind to interactions — the “JSON format + persona” combo that works great together even though neither helps alone — and it takes weeks of manual fiddling before you trust the result.</p><p>There’s a much better way, and it’s older than machine learning.</p><p><strong>This problem was solved in the 1930s</strong></p><p>Decades before anyone was prompting a transformer, industrial engineers faced exactly this: how do you optimize a process with dozens of knobs without testing millions of combinations? Metallurgists, aerospace engineers, semiconductor fabs, clinical trials — everyone hit the same wall.</p><p>The answer came from statisticians like Ronald Fisher, George Box, and J. Stuart Hunter: <strong>fractional factorial designs</strong>. Instead of running every combination, you run a carefully chosen fraction of them, arranged in an orthogonal matrix where every factor is tested “on” and “off” equally often against every other factor. The math guarantees that each factor’s individual effect can be read out cleanly, despite testing a small fraction of the space.</p><p>For prompts, it looks like this:</p><p><em>Full Factorial (10 factors): 1,024 runs</em></p><p><em>Fractional Factorial 2^(10–6)_III: 16 runs (-98.4% of the search space)</em></p><p>To be precise about what that “98%” is and isn’t: it’s not that your API bill automatically drops 98%. Nobody was about to run all 1,024 combinations — it’s a complexity number. The exhaustive search space needed to cover every factor shrinks from 1,024 combinations to 16, and since each run costs the same as any other, your spend follows the run count. The headline benefit is that <em>*full, defensible coverage*</em> of your factors goes from a theoretical exercise to a one-day experiment — and each of those 16 runs is far more informative than a haphazard handful.</p><p>Sixteen runs instead of a thousand. Ten factors, each one measured for its true effect, each one with a standard error, a t-statistic, and a p-value — because the second half of the toolkit is <strong>ANOVA</strong> (Analysis of Variance), which decides whether an observed effect is real or just chance.</p><p>None of this is novel to statisticians. Applying it systematically to prompt engineering is, though. That’s the gap we built <a href="https://github.com/robsanmx/prompt_prism">prompt_prism</a> to fill.</p><h3><strong>Introducing prompt_prism</strong></h3><p>`prompt_prism` is a Python library that runs the whole DoE → execute → ANOVA → “here’s your winning prompt” workflow for you:</p><pre>pip install prompt-prism<br># or with visualization and LLM-judge metrics<br>pip install &quot;prompt-prism[viz,deepeval]&quot;</pre><p>Let me walk you through a real run: optimizing a product-attribute extraction prompt, the kind you’d use in an e-commerce catalog pipeline.</p><p><strong>Optimizing an extraction prompt in five minutes</strong></p><p><strong>1. List your candidate factors</strong></p><p>Instead of one big monolithic prompt string, you write down the candidate pieces you’re debating. Each one becomes a factor with an “off” version and an “on” version:</p><pre>from prompt_prism import Factor, FactorSet<br><br>factors = [<br>    Factor.binary(<br>        &quot;persona&quot;,<br>        level_0_content=&quot;&quot;,<br>        level_1_content=&quot;You are a senior catalog data auditor with 15 years experience.&quot;,<br>        description=&quot;Domain expert persona&quot;,<br>    ),<br>    Factor.binary(<br>        &quot;few_shot&quot;,<br>        level_0_content=&quot;&quot;,<br>        level_1_content=&quot;Example: &#39;Nike Air Max 90 Red&#39; -&gt; Brand: Nike, Color: Red&quot;,<br>        description=&quot;Few-shot exemplar&quot;,<br>    ),<br>    Factor.binary(<br>        &quot;cot&quot;,<br>        level_0_content=&quot;&quot;,<br>        level_1_content=&quot;Analyze the text step-by-step before producing the final attributes.&quot;,<br>        description=&quot;Chain-of-thought reasoning&quot;,<br>    ),<br>    Factor.binary(<br>        &quot;strict_json&quot;,<br>        level_0_content=&quot;Output: key=value&quot;,<br>        level_1_content=&quot;OUTPUT FORMAT: Return a valid JSON object matching schema.&quot;,<br>        description=&quot;Strict JSON schema formatting&quot;,<br>    ),<br>    Factor.binary(<br>        &quot;negative_guardrail&quot;,<br>        level_0_content=&quot;&quot;,<br>        level_1_content=&quot;CRITICAL: Never extract attributes not explicitly written in input.&quot;,<br>        description=&quot;Hallucination guardrail&quot;,<br>    ),<br>]</pre><p><strong>2. Pick a design</strong></p><p>The library ships with a catalog of 28 standard orthogonal designs, and it’ll recommend one for you. For five factors, the 2^(5–1)_V design runs <strong>16 runs</strong> at Resolution V, which means every main effect and every two-way interaction is unaliased — you can even answer “do JSON and persona <em>interact</em>?” not just “does each one work?”</p><pre>from prompt_prism import Experiment, ExactMatch, F1Score<br><br>exp = Experiment.from_factors(<br>    factors=factors,<br>    design=&quot;2(5-1)V&quot;,   # 16 runs instead of 32<br>    system_prompt=&quot;Extract product attributes from listings.&quot;,<br>    data_template=&quot;Title: {{ title }}\nDescription: {{ description }}&quot;,<br>    metrics=[ExactMatch(), F1Score()],<br>    target_metric=&quot;f1_score&quot;,<br>    title=&quot;Catalog Extraction DoE&quot;,<br>)</pre><p><strong>3. Run it</strong></p><p>Bring your benchmark dataset (we’re using ~10 items with golden targets here for brevity — for real decisions you want more) and any LLM callable. OpenAI, Anthropic, Gemini, Vertex, Ollama, a random internal model behind a function — it just needs to be `fn(prompt) -&gt; str`:</p><pre>dataset = [<br>    {<br>        &quot;id&quot;: 1,<br>        &quot;title&quot;: &quot;Nike Air Max 90 Red&quot;,<br>        &quot;description&quot;: &quot;Men sneakers size 42&quot;,<br>        &quot;target&quot;: &quot;Brand: Nike, Color: Red&quot;,<br>    },<br>    # ... your other test items<br>]<br><br>def my_llm_client(prompt: str) -&gt; str:<br>    # however you call your model<br>    return client.chat.completions.create(...)<br><br># threaded execution + built-in SQLite response caching<br>results = exp.run(dataset=dataset, client=my_llm_client, max_workers=4)</pre><p>Two things are quietly happening here that save real money: execution is parallelized, and every (run, item, prompt) response is cached in SQLite, so if you rerun or extend the experiment you never pay the API twice for the same call.</p><p><strong>4. Let ANOVA decide</strong></p><pre>report = exp.analyze(block_by=&quot;sample_id&quot;)<br>print(report.to_markdown())</pre><p>The block_by argument matters more than it looks. We’re using a Randomized Complete Block Design: each test item is a “block,” so the model of analysis can attribute score differences to prompt factors rather than to item difficulty. That hard sample that confuses the model no longer drags one arm of the experiment down and makes a random factor look great.</p><p><strong>What the results actually said</strong></p><p>Here’s the pareto chart of effects that came out of that run:</p><pre>=== Pareto Chart of Standardized Effects (f1_score) ===<br>Factor ID | Name                 | Effect Δ | t-value | Chart<br>----------+----------------------+----------+---------+-----------------------------------------<br>   D      | strict_json          |  +0.3520 |   14.21 | ██████████████████████████████ [*** SIG ***]<br>   B      | few_shot             |  +0.2480 |   10.02 | █████████████████████          [*** SIG ***]<br>   E      | negative_guardrail   |  -0.1210 |   -4.88 | ██████████                     [*** SIG ***]<br>   A      | persona              |  +0.0150 |    0.60 | █                              [NOT SIG]<br>   C      | cot                  |  -0.0080 |   -0.32 | ░                              [NOT SIG]</pre><p>Three takeaways, in the order I’d tell them to my team:</p><p><strong>The JSON format instruction was doing most of the work.</strong> +35.2% F1, t = 14.21, p &lt; 0.001. That’s not a borderline result. Forcing a valid JSON schema killed a whole class of parsing failures that free-form “key=value” output kept producing.</p><p><strong>One factor we </strong><em>loved</em><strong> was actively hurting us.</strong> The “never extract attributes not in the input” guardrail cost us 12.1% F1 (t = −4.88, p = 0.002). The model read the instruction as “be very careful,” so it started refusing valid attributes whenever it wasn’t fully sure. We had shipped that line for months, convinced it was doing us a favor. Nobody would have caught it in one-at-a-time testing, because the few-shot examples in the same prompt were masking the damage.</p><p><strong>The persona and the chain-of-thought did nothing.</strong> Neither was significant (p &gt; 0.5), so the statistical verdict is: drop them. That wasn’t just a cleanliness call — it removed roughly 180 tokens from every request, which cut our inference latency about a third and trimmed the ongoing token bill on top of that.</p><p>Then the last step, which is the part I actually care about:</p><pre>optimal_template = exp.get_optimal_prompt_template()<br>print(optimal_template.render(title=&quot;MacBook Pro M3&quot;, description=&quot;16GB RAM 512GB SSD&quot;))</pre><p>The library assembles the statistically optimal configuration for you: keep the significant boosters, remove the significant hurts, drop the neutral bloat. You review the rendered prompt, you merge it, you’re done. The whole loop took about as long as it takes to read this article.</p><p><strong>What about qualities you can’t exact-match?</strong></p><p>Extraction tasks have golden answers, so string-based metrics (exact match, F1, schema validity) work fine. But hallucination, faithfulness, answer relevancy, tone — those need judgment.</p><p>`prompt_prism` wraps DeepEval’s LLM-as-a-Judge metrics as first-class evaluation metrics, so you can run the same DoE over subjective qualities:</p><pre>from prompt_prism import deepeval_metric, JudgeCache<br><br># SQLite cache so you never pay for the same judge call twice<br>judge_cache = JudgeCache(db_path=&quot;judge_cache.db&quot;)<br><br># reference-based G-Eval against golden targets<br>geval_accuracy = deepeval_metric(<br>    &quot;g_eval&quot;,<br>    name=&quot;factual_accuracy&quot;,<br>    criteria=&quot;Determine if the generated answer factually matches the golden target.&quot;,<br>    cache=judge_cache,<br>)<br><br># reference-free faithfulness &amp; hallucination (great for RAG)<br>faithfulness = deepeval_metric(&quot;faithfulness&quot;, threshold=0.7, cache=judge_cache)<br>hallucination = deepeval_metric(&quot;hallucination&quot;, threshold=0.8, cache=judge_cache)<br><br>exp = Experiment.from_factors(<br>    factors=factors,<br>    design=&quot;2(5-1)V&quot;,<br>    metrics=[geval_accuracy, faithfulness, hallucination],<br>    target_metric=&quot;factual_accuracy&quot;,<br>)</pre><p>There’s one honest caveat here: judge metrics are noisier than exact-match metrics, so the ANOVA is fighting two sources of variance, not one. The judge cache makes reruns cheap, which is exactly what you want when the evaluation itself is noisy — replicate, confirm, trust the stable effects.</p><p><strong>Autonomous self-healing: Plugging PromptPrism into agentic harnesses</strong></p><p>If you are building autonomous agents — using LangGraph, CrewAI, AutoGen, or custom tool-calling loops — you know the dirty secret of agent evaluation: <strong>prompts drift, and prompt self-improvement is usually a disaster.</strong></p><p>The common pattern today is to give an agent a “reflection” loop: when a tool call fails or an edge case breaks in production, a supervisor LLM rewrites the system prompt.</p><p>The problem? <strong>Unconstrained LLM reflection causes prompt degradation.</strong></p><p>- Prompts rapidly accumulate emotional caps-lock rules (<strong>”CRITICAL: NEVER DO X AGAIN”</strong>).</p><p>- Prompts bloat from 400 tokens to 3,000 tokens of contradictory instructions, spiking latency and API costs.</p><p>- The agent overfits to one noisy failure sample and silently regresses on five other capabilities.</p><p>Instead of letting an agent blindly rewrite its prompt, you can plug `prompt_prism` directly into your agent harness as a <strong>statistical self-healing engine</strong>:</p><pre><br>                          ┌────────────────────────────┐<br>                          │   Production Error Log /   │<br>                          │   Agent Failure Cluster    │<br>                          └─────────────┬──────────────┘<br>                                        │<br>                                        ▼<br>                          ┌────────────────────────────┐<br>                          │  Agent Proposer:           │<br>                          │  Synthesize 5 Fix Factors  │<br>                          └─────────────┬──────────────┘<br>                                        │<br>                                        ▼<br>                          ┌────────────────────────────┐<br>                          │  PromptPrism 2^(5-1)_V     │<br>                          │  Orthogonal Tournament     │<br>                          │  (16 Runs, Parallelized)   │<br>                          └─────────────┬──────────────┘<br>                                        │<br>                                        ▼<br>                          ┌────────────────────────────┐<br>                          │  RCBD ANOVA Analysis       │<br>                          │  Filter: p &lt; 0.05, Δ &gt; 0   │<br>                          └─────────────┬──────────────┘<br>                                        │<br>                         ┌──────────────┴──────────────┐<br>                         ▼                             ▼<br>             🟢 Lock in Significant Fixes    🔴 Purge Harmful &amp; Bloat<br>             (Proven positive Δ)             (p &gt;= 0.05 or Δ &lt; 0)<br>                         │                             │<br>                         └──────────────┬──────────────┘<br>                                        │<br>                                        ▼<br>                          ┌────────────────────────────┐<br>                          │  Auto-Deploy Winning       │<br>                          │  Prompt + ANOVA Audit Trail│<br>                          └────────────────────────────┘</pre><p>Here is what that looks like in code inside an agent harness:</p><pre>rom prompt_prism import Factor, Experiment, JSONValidation, ExactMatch<br><br>def self_heal_agent_prompt(failure_cluster, current_system_prompt):<br>    # 1. The supervisor LLM proposes candidate modular fixes (not a full rewrite)<br>    candidate_factors = [<br>        Factor.binary(<br>            &quot;retry_schema_hint&quot;,<br>            level_0_content=&quot;&quot;,<br>            level_1_content=&quot;If tool arguments fail validation, inspect the schema signature before retrying.&quot;,<br>            description=&quot;Schema correction hint&quot;,<br>        ),<br>        Factor.binary(<br>            &quot;scratchpad_planning&quot;,<br>            level_0_content=&quot;&quot;,<br>            level_1_content=&quot;Write your tool execution plan inside &lt;plan&gt; tags before invoking any tool.&quot;,<br>            description=&quot;Scratchpad pre-planning&quot;,<br>        ),<br>        Factor.binary(<br>            &quot;dynamic_exemplar&quot;,<br>            level_0_content=&quot;&quot;,<br>            level_1_content=&quot;Example: call_api(query=&#39;...&#39;) -&gt; valid response.&quot;,<br>            description=&quot;Dynamic failure exemplar&quot;,<br>        ),<br>        Factor.binary(<br>            &quot;negative_constraint&quot;,<br>            level_0_content=&quot;&quot;,<br>            level_1_content=&quot;Never invoke tools with empty or placeholder arguments.&quot;,<br>            description=&quot;Argument constraint&quot;,<br>        ),<br>        Factor.binary(<br>            &quot;strict_json_mode&quot;,<br>            level_0_content=&quot;Output: plain text&quot;,<br>            level_1_content=&quot;OUTPUT: Strict JSON object matching ToolCall schema&quot;,<br>            description=&quot;Strict tool formatting&quot;,<br>        ),<br>    ]<br><br>    # 2. Setup a 16-run orthogonal tournament across the failure cases<br>    exp = Experiment.from_factors(<br>        factors=candidate_factors,<br>        design=&quot;2(5-1)V&quot;,<br>        system_prompt=current_system_prompt,<br>        data_template=&quot;Task: {{ task }}\nPrevious Error: {{ error }}&quot;,<br>        metrics=[JSONValidation(), ExactMatch()],<br>        target_metric=&quot;json_validation&quot;,<br>    )<br><br>    # 3. Execute concurrently with SQLite caching<br>    exp.run(dataset=failure_cluster, client=agent_llm_client, max_workers=4)<br><br>    # 4. Statistical gatekeeping via ANOVA with RCBD blocking<br>    report = exp.analyze(block_by=&quot;sample_id&quot;)<br><br>    # 5. Extract only mathematically proven fixes &amp; generate audit trail<br>    optimal_template = exp.get_optimal_prompt_template()<br>    <br>    # Check if we have statistically significant improvements<br>    significant_boosters = [<br>        f for f in report.factor_effects <br>        if f.p_value &lt; 0.05 and f.main_effect &gt; 0<br>    ]<br>    <br>    if significant_boosters:<br>        # Hot-patch agent runtime or generate an auditable PR with the ANOVA report<br>        deploy_agent_prompt(<br>            new_prompt=optimal_template.render(),<br>            changelog_md=report.to_markdown()<br>        )</pre><p>Why this changes agent development:</p><p>1. <strong>Immunity to Prompt Bloat</strong>: Neutral factors ($p \ge 0.05$) are stripped automatically. Your prompt only grows if an addition <em>*proves*</em> its worth.</p><p>2. <strong>Zero Hallucinated Fixes</strong>: The supervisor LLM might <em>*think*</em> adding a three-paragraph explanation will fix tool calling, but ANOVA tests it across the whole failure cluster and drops it if $p \ge 0.05$.</p><p>3. <strong>Continuous, Auditable Self-Improvement</strong>: Every prompt iteration is backed by a mathematical artifact — an ANOVA table, $F$-statistic, and Pareto chart — instead of vibes.</p><p><strong>When <em>not</em></strong><em> </em><strong>to do this</strong></p><p>For balance: a fractional design is not free of assumptions.</p><p>- <strong>Screening designs (Resolution III) alias some effects.</strong> A 2^(10–6)_III design gives you clean main effects but main effects can be aliased with two-way interactions. If you suspect big interactions, use a higher-resolution design (like the Resolution V above) or follow up the screen with a confirmation run — the library’s report suggests unaliased follow-up designs for exactly this.</p><p>- <strong>You still need a decent benchmark.</strong> ANOVA can tell you an effect is real, but a real 2% effect isn’t always worth shipping. The statistics sharpen your decisions; they don’t make them trivial.</p><p>- <strong>Overkill for one factor.</strong> If you’re genuinely choosing between two prompts, just A/B them. DoE earns its keep when you have ~4 or more candidate factors in play, which is most real prompts.</p><p><strong>The difference in one table</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*tqSQHQxl-wzmmxqXlfnhmg.png" /></figure><p>prompt_prism is MIT-licensed and open source:</p><p>- <strong>GitHub</strong>: <a href="https://github.com/robsanmx/prompt_prism">https://github.com/robsanmx/prompt_prism</a></p><p>- There’s an agent skill in the repo too, if you want your coding agent to run the whole experiment loop and iterate on prompts autonomously.</p><p>The next time a prompt “improved by 9%,” you deserve to know which line of the prompt earned it. The math to find out is a hundred years old — it just hadn’t been pointed at your API bill until now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d14100fc66e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Cosine Similarity]]></title>
            <link>https://medium.com/@trev21mx/beyond-cosine-similarity-56bc8c57dc9a?source=rss-f26deb13af12------2</link>
            <guid isPermaLink="false">https://medium.com/p/56bc8c57dc9a</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fuzzy-search]]></category>
            <dc:creator><![CDATA[Roberto Sanchez]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 12:22:17 GMT</pubDate>
            <atom:updated>2026-02-04T16:11:26.615Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Sometimes, the simplest solution isn’t just elegant — it’s the only one that actually works.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O43AajN1B9i42OVsLMYX6g.png" /></figure><p>I’ve been working in AI for years — building RAG pipelines, fine-tuning embeddings, training models, orchestrating agents, and chasing the “next big thing.” And somewhere along the way, I started to feel… lost.</p><p>Not in the code — but in the <em>purpose</em>.<br> We were building systems so complex, so layered, so <em>agentic</em>, that they started to feel like magic — but magic that breaks when the lights go out.</p><p>And then, one day, I stumbled on something quiet.<br> Something simple.<br> Something ancient.</p><p>And it changed everything.</p><h3>The Problem: We’re Missing the “Why” Behind the “What”</h3><p>Let’s be honest: most semantic search today is still just cosine similarity on steroids.</p><p>You embed your query and documents. You compute the angle between them. You rank by closeness.<br> <strong>Done</strong>.</p><p>But here’s the catch: this only tells you how similar two things <em>look</em> — not whether they <em>mean</em> the same thing.</p><p>Take this example:</p><blockquote><em>Query: </em>“She was happy to leave.”<em><br> Document: </em>“She was sad to leave.”</blockquote><p>Same structure. Same words. Almost identical embeddings.<br> Cosine similarity says: <em>“Wow, these are super similar!”</em></p><p>But we <em>know</em> — deep down — that they mean the <em>opposite</em>.</p><p>And that’s where the real problem lies.</p><p>We’re not just searching for similarity.<br> We’re searching for meaning.<br> And meaning isn’t just “close.”<br> It’s <em>right</em>.</p><h3>Enter Pythagorean Fuzzy Sets: Where Math Meets Meaning</h3><p>So I started asking:<br> <em>What if we didn’t just measure similarity — but also measured conflict and uncertainty?</em></p><p>That’s when I discovered Pythagorean Fuzzy Sets (PyFS).</p><p>It’s not some flashy new AI model.<br> It’s an extension of fuzzy logic — a 100-year-old mathematical idea — that says:</p><blockquote><em>“What if, instead of just saying </em>‘yes’<em> or </em>‘no’<em>, we also say </em>‘I’m not sure’<em> — and even </em>‘this might be wrong’<em>?”</em></blockquote><p>In PyFS, every comparison has three values:</p><ul><li>μ (Membership): How much the document <em>satisfies</em> the query. (Like cosine similarity.)</li><li>ν (Non-membership): How much it <em>disagrees</em> or <em>conflicts</em> with the query.</li><li>π (Hesitation/uncertainty): How unsure we are — calculated as:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*OT-WLRtrzXXEGQbC7pjKrQ.png" /></figure><p><strong>Boom</strong>.</p><p>Now, instead of a single number (cosine), we get a full picture of semantic alignment — including the moments of doubt, the signs of contradiction, and the edge cases where things seem similar but aren’t.</p><h3>Why <em>π</em> Matters in Real Search</h3><ul><li>High <em>π</em> → The result is uncertain. Use it as a signal to:</li><li>Flag for human review</li><li>Reduce ranking weight</li><li>Request clarification (e.g., “<strong><em>Did you mean X or Y?</em></strong>”)</li><li>Low <em>π</em> → The system is confident. High trust for ranking.</li><li>Balanced <em>μ</em> and <em>ν</em> → Look for true contradictions — not just noise.</li></ul><p>This transforms search from a “what’s closest?” game into a “what’s meaningful and reliable?” system.</p><h3>Detecting What Cosine Misses</h3><p>Let’s say you’re searching for “best practices in AI ethics.”<br> Cosine similarity might return a document that uses the same words: “AI,” “ethics,” “bias,” “transparency.”</p><p>But what if that document argues that AI bias is inevitable — a direct contradiction to the ethical stance you’re looking for?</p><p>Cosine similarity sees only alignment.<br> PyFS sees conflict.</p><h3>The Real Power: ν Is Not Just “1 — μ”</h3><p>Most people assume ν=1−μ. That’s <em>naive</em>.</p><p>In <strong>PyFS</strong>, <em>ν</em> is independent — it can be calculated from different data sources and different logic.</p><p>And that’s where the magic happens.</p><h3>How to Compute ν: 3 Powerful Strategies</h3><p>Here’s how to build ν as a meaningful, independent signal of semantic conflict — not just noise.</p><h3>Negative Cosine Similarity — Detecting Semantic Opposition</h3><p>Use the negative cosine of the same vectors used for <em>μ</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*gbWx2BS49vPnEq-eKYGARA.png" /></figure><p><strong>Example</strong></p><ul><li>Query: “The sky is blue”</li><li>Document: “The sky is green”</li><li>Cosine similarity: 0.1 → μ=0.1</li><li>Negative cosine: 0.9 → ν=0.9</li></ul><p>This catches semantic opposition — even when vectors are close.</p><p><em>Use Case: Detecting emotional tone conflict or deliberate contradiction.</em></p><h3>2. Jaccard Distance on Keywords — Catching Topic Mismatch</h3><p>Even if embeddings are similar, the actual terms might be unrelated.</p><p>Compute the Jaccard distance between keyword sets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*898kSnDYIiPzMy4ygJK-Lw.png" /></figure><p><strong>Example:</strong></p><ul><li>Doc A: [“climate”, “change”, “global”]</li><li>Doc B: [“economy”, “growth”, “investment”]</li><li>Jaccard = 0 → ν=1</li></ul><p>Even if their embeddings are <em>slightly</em> aligned (e.g., due to shared “global” or “change”), this <em>ν</em> score reveals topic mismatch.</p><p><em>Use Case: Preventing “hallucinatory” relevance in hybrid search systems.</em></p><h3>3. Contradiction via Knowledge Graphs — Factual Incompatibility</h3><p>Use external knowledge (WordNet, ConceptNet, medical ontologies) to detect factual contradictions.</p><ul><li>“The sky is green” vs. “The sky is blue” → known contradiction → ν=1</li><li>“Water boils at 50°C” vs. “Water boils at 100°C” → contradiction detected</li></ul><p><em>Use Case: High-stakes domains — healthcare, law, scientific research — where truth matters more than surface similarity.</em></p><h3>Key Takeaway</h3><blockquote>π<em> isn’t just a side calculation — it’s the semantic conscience of the model.<br> It answers: </em>“How much should I trust this similarity?”<em><br> And the math? It’s not arbitrary — it’s geometrically grounded in the unit sphere, ensuring logical consistency and rich expressiveness.</em></blockquote><h3>Why This Matters: The Power of “I Don’t Know”</h3><p>Here’s what’s wild:<br> The system now penalizes ambiguity.</p><p>If μ and ν are both around 0.6, then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*NG0pEA734UOqe6WVwH1nVg.png" /></figure><p><strong>High uncertainty</strong></p><p>. The system says: <em>“Hmm… I’m not confident.”</em></p><p>But if μ is 0.9 and ν is 0.1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*K0-yTrJD4S01UewEgDshpg.png" /></figure><p><strong><em>Still some doubt</em></strong>, but much clearer.</p><p>And if μ² + ν² &gt; 1?<br> Then <strong>π</strong> becomes <em>imaginary</em>.</p><p>Yes — <em>imaginary</em>.<br> Not an error. A signal.</p><p>It means: <em>“This data is fundamentally inconsistent. There’s a real conflict.”</em></p><p>That’s not a bug. That’s deep insight.</p><p>It’s like the math is saying: <em>“You’re asking for something impossible. The system is screaming.”</em></p><h3>The Real Game-Changer: No Agent. No Overengineering.</h3><p>Now, here’s where I get emotional.</p><p>One of the things I love most about mathematics — and especially about working in AI — is this truth:</p><p><em>Sometimes, the simplest solution is the better one.</em></p><p>We’ve been tricked into thinking that every problem needs an agentic RAG. A chain-of-thought. A multi-agent system. A “smart” pipeline that can <em>reason</em>, <em>plan</em>, <em>self-correct</em>.</p><p>But what if the answer isn’t <em>more complexity</em> — but better math?</p><p>What if we stopped trying to build AI that <em>feels</em> like a human — and started building AI that <em>understands</em> like one?</p><p>PyFS doesn’t need agents.<br> It doesn’t need memory.<br> It doesn’t need a fancy orchestrator.</p><p>It just needs two vectors, a formula, and a little courage to think differently.</p><p>And that’s the beauty I’ve come to see — not just in this math, but in the whole field.</p><p>We have powerful tools — embeddings, attention, transformers — but we’ve forgotten how to <em>use</em> them simply.</p><p>We’re so busy chasing the next breakthrough that we miss the elegance in what’s already working.</p><h3>The Future Isn’t More Agents — It’s More Clarity</h3><p>I’ve seen too many teams spend months building RAG systems that are expensive, fragile, and hard to maintain — only to realize the real issue wasn’t <em>data</em>, or <em>retrieval</em>, or <em>context</em> — it was uncertainty.</p><p>And it wasn’t solved with more agents.</p><p>It was solved with a better way of thinking about similarity.</p><p>PyFS doesn’t replace embeddings.<br> It <em>enhances</em> them.<br> It gives them depth.</p><p>It turns a flat score into a semantic profile — one that can detect contradictions, flag ambiguity, and even say <em>“I don’t know”</em> with confidence.</p><p>And that — that’s not just better engineering.<br> That’s <em>better intelligence</em>.</p><h3>Final Thought: Simplicity Is Wisdom</h3><p>As an AI engineer, I’ve learned this:<br> The most powerful systems aren’t always the most complex.</p><p>Sometimes, they’re the ones that stop to ask: <em>“What if I just measured doubt?”</em></p><p>Because in the end, truth isn’t always clear.<br> And the best AI isn’t the one that pretends to know everything — <br> It’s the one that knows when it doesn’t.</p><p>And in that space — between certainty and uncertainty — <br> That’s where real meaning lives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56bc8c57dc9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>