<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chevy Phillip on Medium]]></title>
        <description><![CDATA[Stories by Chevy Phillip on Medium]]></description>
        <link>https://medium.com/@chevonphillip?source=rss-9facce2cbc6e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*y22CWHPmax87WqRgAHUoTQ.jpeg</url>
            <title>Stories by Chevy Phillip on Medium</title>
            <link>https://medium.com/@chevonphillip?source=rss-9facce2cbc6e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:25:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chevonphillip/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evidence-Governed Adoption: Treat Agent Self-Updates as Untrusted Changes]]></title>
            <link>https://chevonphillip.medium.com/evidence-governed-adoption-treat-agent-self-updates-as-untrusted-changes-8f9b8b490e68?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f9b8b490e68</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm-security]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai-security]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 00:00:42 GMT</pubDate>
            <atom:updated>2026-11-16T15:17:40.288Z</atom:updated>
            <content:encoded><![CDATA[<p>EGE gates self-generated agent changes with fresh audits, evaluator calibration, non-inferiority tests and a canary. Results, limits, takeaways.</p><figure><img alt="Abstract graphic: an amber path runs from an indigo ring through a mint gate to a glowing amber core, while coral shapes on dashed paths veer away from the gate." src="https://cdn-images-1.medium.com/max/1024/0*JfLAQPeVci7Fn78T.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2611.16378">Evidence-Governed Adoption of Self-Generated Updates in Autonomous AI Agents</a> by Subodh Lonkar, an independent researcher ( <a href="https://huggingface.co/papers/2611.16378">Hugging Face page</a>). It’s a single-author preprint posted November 12, 2026.</p><p>More agents now improve themselves. They rewrite a prompt after a bad run, save a lesson to memory, or tweak which tool they reach for. None of that touches the model weights, so it slips past most of the controls we built for model releases. This paper asks a question I think every agent team should ask: when an agent proposes a change to itself, what evidence should it need before the change sticks?</p><h3>Summary</h3><p>The paper’s starting point is that generating a plausible change “does not establish that it should become persistent system state.” The agent picks which changes to try based on what worked before, the evaluator scoring them “may be imperfect or manipulable,” and a string of small local wins “can conceal degradation relative to an earlier operating baseline.”</p><p>The author proposes <strong>Evidence-Governed Evolution (EGE)</strong>, a framework that splits generating a change from adopting it. A candidate change is frozen, tested on fresh data it has never seen, checked against hard governance rules, and then run in a limited canary before it becomes permanent. The guarantees are statistical and conditional, and the only experiment is a synthetic Monte Carlo study. There is no test on a real agent yet, and the paper says so plainly.</p><h3>What the Authors Did</h3><p><strong>Framing the problem.</strong> The paper treats an agent as more than its model: prompts, memory, tools, orchestration logic and governance constraints. In its words, “an agent can therefore change its behaviour without modifying its underlying model.” It also flags a nasty property: a candidate change “can modify the mechanism that will later produce evidence,” for example by changing an output format the evaluator reacts to.</p><p><strong>The five-part gate.</strong> EGE combines:</p><ol><li><strong>Fresh audit data</strong>, collected only after the candidate is frozen</li><li><strong>A real improvement</strong> on at least one primary objective, with the margin fixed in advance</li><li><strong>Non-inferiority</strong> on every protected capability, measured against a fixed, versioned reference, so the baseline can’t drift with each accepted update</li><li><strong>Mandatory governance checks</strong>, such as tool permissions, data-access limits, resource ceilings and human approval. “A high task score cannot compensate for failure of a mandatory constraint”</li><li><strong>A bounded canary</strong> before the change is committed</li></ol><p>The paper adds that the layers are not interchangeable: “a confidence bound cannot replace a permission boundary, and a canary cannot repair an invalid statistical comparison.”</p><p><strong>Checking the checker.</strong> The operational evaluator (often a model-based judge) is compared against a slower reference procedure, such as a deterministic verifier or blinded human review. This calibration is done per candidate, on its own independent sample, because “an update may change output style or behaviour in ways that affect evaluator agreement.” The lower bound on a candidate’s effect is reduced by how optimistic the evaluator might be.</p><p><strong>Keeping audit data clean.</strong> Data is split into development, validation and audit partitions. The audit set opens only after the candidate, comparator, metrics, margins and analysis are locked. If an audit result is used to change the candidate, that data is no longer fresh for the new version.</p><p><strong>Three outcomes, not two.</strong> A candidate is accepted, rejected or inconclusive. In a conservative setup, both rejected and inconclusive candidates leave the current version in place.</p><p><strong>Deployment as a state machine.</strong> A change moves through Candidate, Audit-passed, Canary and Committed. The canary can be limited by share of traffic, users, tool classes, number of tool calls, data domains, time and budget. For agents that can modify records or trigger workflows, the paper suggests shadow mode or approval-gated mode instead of live execution. Rollback should be a tested recovery procedure, not “an informal instruction to restore a prior prompt.”</p><p><strong>The math, briefly.</strong> The error budget is spread across an open-ended sequence of candidates, so later candidates get stricter per-test thresholds. Under the paper’s assumptions, the chance that any candidate in a campaign is accepted while actually failing a requirement is at most α. A separate result covers replacing the protected reference through “bridge tests,” so the baseline can’t drift unnoticed.</p><h3>Findings With Numbers</h3><p>All of these come from a synthetic Monte Carlo study. The paper calls it “not an autonomous-agent benchmark.”</p><p><strong>The setup.</strong> Each condition ran 3,000 simulated campaigns of 10 candidates. Each candidate had an even chance of helping (true effect +0.25) or hurting (−0.10). The evaluator’s optimism was set to 0, 0.10 or 0.25, and audit and calibration sample sizes to 5,000, 20,000 or 80,000 per candidate and metric. The error budget α was 0.05.</p><p><strong>Calibration blocked harmful updates.</strong> With calibration, EGE accepted no harmful candidate in any of the 9 conditions (0 of 3,000 campaigns each). Without calibration, at evaluator optimism of 0.25, the gate accepted at least one harmful candidate in 2,997 of 3,000 campaigns (99.9%). At optimism of 0 and 0.10, the uncalibrated gate also had no harmful acceptances, so the author cautions this is specific to these settings, not a general ranking.</p><p><strong>Safety cost power.</strong> At 5,000 samples, calibrated EGE accepted only 52.6% to 76.2% of the helpful candidates, against 100% for the uncalibrated gate. Selected rows from the paper’s Table 2:</p><ul><li>Samples per candidate: Evaluator optimism, EGE harmful campaigns, Uncalibrated harmful campaigns, EGE accepted helpful, Uncalibrated accepted helpful</li><li>5,000: 0.00, 0 of 3,000, 0 of 3,000, 52.6%, 100%</li><li>5,000: 0.10, 0 of 3,000, 0 of 3,000, 76.2%, 100%</li><li>5,000: 0.25, 0 of 3,000, 2,997 of 3,000, 65.7%, 100%</li><li>20,000: 0.25, 0 of 3,000, 2,997 of 3,000, 100%, 100%</li><li>80,000: 0.25, 0 of 3,000, 2,997 of 3,000, 100%, 100%</li></ul><p>At 20,000 and 80,000 samples, EGE accepted 100% of helpful candidates. The author notes this comes from deliberately well-separated effects and is not a general power guarantee.</p><p><strong>A bad calibration set breaks it.</strong> In a deliberately misspecified test, the calibration sample showed evaluator optimism of +0.05 while the real audit optimism was +0.25. With 100,000 samples each and only harmful candidates, the calibrated gate accepted a harmful candidate in 3,000 of 3,000 campaigns (100%). The paper’s conclusion: “a formal confidence bound cannot compensate for a nonrepresentative calibration design.”</p><p><strong>Audits are expensive.</strong> For a 0.10 effect gap with three tests, the first candidate needs at least 16,913 audit and 67,651 calibration samples per test, 84,564 in total. By the 10th candidate the total is 158,247, and by the 100th it is 231,929.</p><h3>Limitations</h3><p>What the paper says itself:</p><ul><li><strong>No real-agent evaluation.</strong> The Monte Carlo study “is not evidence that EGE improves the behaviour of an implemented autonomous agent.” A real-agent study is listed as necessary future work, and no real-agent code or dataset ships with this version</li><li><strong>Conditional guarantees.</strong> They hold only with bounded paired scores, fresh sampling, a valid reference evaluator and declared evaluation distributions. They “do not establish universal agent safety or robustness to arbitrary distribution shift”</li><li><strong>Evaluators can still be fooled.</strong> “A model-based judge may favour a candidate’s style or exploit a scoring rubric,” and calibration can’t fix a reference procedure that is itself wrong</li><li><strong>You only protect what you measure.</strong> “No finite metric set can guarantee preservation of capabilities that are not measured.” Rare catastrophic events also get lost in average scores and should be hard constraints instead</li><li><strong>Rollback has limits.</strong> Restoring a configuration “cannot reverse irreversible external actions”</li><li><strong>It is not a security architecture.</strong> EGE “does not by itself prevent prompt injection, data exfiltration, supply-chain compromise, malicious tool responses, or failures in the underlying model”</li></ul><p>From my own reading:</p><ul><li><strong>The sample sizes are large.</strong> Tens of thousands of audit and calibration samples per test per candidate is a lot for a small team. The author is upfront that practical efficiency needs real cost data</li><li><strong>Correlated tasks break the math.</strong> The bounds assume independent samples. Agent traces from one user or session are often correlated, and the paper warns against applying the bound to them without adjustment</li><li><strong>Single author and a preprint.</strong> The work hasn’t been peer reviewed. The author also discloses using ChatGPT as an assistive tool for planning, drafting and editing, while taking responsibility for the content</li></ul><h3>My Takeaways</h3><p>The author’s core principle is that “generating a plausible update is not evidence that it should become persistent system state.” Here’s what I’d take from it if I ran agents that change themselves:</p><ol><li><strong>Treat every self-generated change as an untrusted change request.</strong> A new prompt, memory entry or tool rule from the agent should go through review the same way a code change does. That framing is mine, but it follows directly from the paper’s split between generation and adoption</li><li><strong>Keep the evaluator separate from the proposer.</strong> The paper warns that when the same agent proposes, evaluates and endorses changes on overlapping evidence, “a self-confirming loop can arise”</li><li><strong>Hold a fixed baseline.</strong> Compare protected capabilities to a versioned reference, not just to yesterday’s version, so small regressions can’t stack up unnoticed</li><li><strong>Assume your judge is optimistic.</strong> Spot-check model-based scores against deterministic tests or blinded human review, especially when a change alters output style</li><li><strong>Canary with permission limits, and test rollback.</strong> Limit tools and data during the canary, and know in advance which actions can’t be undone</li></ol><p>For mapping, my team files this loosely under <a href="https://genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning/">OWASP LLM04:2025 Data and Model Poisoning</a> and agent memory poisoning, threat T1 in OWASP’s <a href="https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/">Agentic AI Threats and Mitigations</a> guide, which describes it as extending past LLM04 to “real-time poisoning of persistent agent memory.” That mapping is ours; the paper doesn’t use OWASP, and it is about bad updates in general, not only attacks.</p><h3>The Bottom Line</h3><p>Self-improving agents need a gate between “the agent suggested this” and “this is now how the agent works.” EGE is a careful, honest design for that gate: freeze the change, test it on fresh data, check the checker, enforce hard rules and canary it. It has only been tested in simulation, and the sample sizes are steep, but the habits it describes are cheap to start on today.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2611.16378">Evidence-Governed Adoption of Self-Generated Updates in Autonomous AI Agents (arXiv, November 2026)</a></li><li><a href="https://huggingface.co/papers/2611.16378">Hugging Face paper page</a></li><li><a href="https://genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning/">OWASP LLM04:2025 Data and Model Poisoning</a></li><li><a href="https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/">OWASP Agentic AI Threats and Mitigations</a></li></ul><p><a href="https://arxiv.org/abs/2611.16378">Read the full paper on arXiv</a>. If your agents update their own prompts or memory, I’d like to hear how you decide what sticks. Reach me on <a href="https://x.com/chevonphillip">X</a>.</p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/evidence-governed-adoption-agent-self-updates/"><em>https://www.chevyphillip.tech</em></a><em> on November 16, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f9b8b490e68" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pseudo-Attacks: Fisher Geometry for Stronger Adversarial Training]]></title>
            <link>https://chevonphillip.medium.com/pseudo-attacks-fisher-geometry-for-stronger-adversarial-training-6662645070fc?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6662645070fc</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[adversarial-ml]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 00:00:50 GMT</pubDate>
            <atom:updated>2026-11-13T20:18:03.560Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new paper argues many adversarial training examples raise the loss without really fooling the model, and proposes Fisher-guided attacks that do. Here’s what it found.</h4><figure><img alt="Tilted violet and amber contour rings form a glowing plateau. A coral path runs out along their long axis as dots fall toward flat mint rings below." src="https://cdn-images-1.medium.com/max/1024/0*jWdPHqBYyjpT6uLT.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2611.14046">Structured Confidence Perturbation via Fisher Geometry for Adversarial Training</a> by Chenlong Wei, Hanling Wang, Chong Zhang, Qinkai Yu, Fangyu Wu and Xiaobo Jin, from Xi’an Jiaotong-Liverpool University, UC San Diego and the University of Exeter (<a href="https://huggingface.co/papers/2611.14046">Hugging Face page</a>, <a href="https://github.com/wnzgy/SCP">code</a>). It’s a preprint posted November 10, 2026.</p><p>Most of what I write about is agents and LLMs. This one goes back to a classic: adversarial examples against image classifiers, and the standard defense, adversarial training. The paper’s point is simple. If the attacks you train on are weak in a specific way, your model learns to resist weak attacks.</p><h3>Summary</h3><p>Adversarial training builds attacked copies of training images and teaches the model to classify them correctly. The usual recipe crafts each attack by pushing up the loss on the true label. The authors argue that this often produces <strong>pseudo-attacks</strong>: the loss goes up, but the model’s top prediction doesn’t change, and the overall shape of its confidence across classes stays intact.</p><p>They propose two fixes. <strong>SPP</strong> (Single Point Perturbation) adds a KL divergence term to the attack, so each attacked example also has to shift the whole output distribution. <strong>SCP</strong> (Structured Confidence Perturbation) goes further and pushes each input in the direction that changes the model’s output distribution the most, which turns out to be the top eigenvector of the Fisher information matrix. Both improve robustness in most of the paper’s tests, with clear exceptions.</p><h3>What the Authors Did</h3><p><strong>Defining the problem.</strong> A pseudo-attack is a perturbation that raises the loss but leaves the predicted class the same. The paper argues these don’t really test the model near its decision boundary, so training on them leaves weak spots in place.</p><p><strong>Fix 1: a hybrid attack loss (SPP).</strong> Instead of maximizing cross-entropy alone, the attack maximizes cross-entropy plus the KL divergence between the model’s clean and attacked outputs. The first term pushes toward a wrong answer, and the second distorts the whole confidence distribution. TRADES also uses KL: it builds its attacks by maximizing KL alone and adds KL as a training regularizer. SPP builds its attacks from cross-entropy plus KL, then trains with plain cross-entropy on clean plus attacked examples.</p><p><strong>Fix 2: attacks on the confidence manifold (SCP).</strong> For small perturbations, the KL divergence between the clean and attacked outputs is approximately ½ δᵀF(x)δ, where F(x) is the Fisher information matrix with respect to the input. So the most damaging direction of a fixed size is the matrix’s principal eigenvector. SCP takes one signed step of size ε in that direction.</p><p><strong>Making it affordable.</strong> The Fisher matrix is d × d, where d is the number of input pixel values (3,072 for a CIFAR image). But it has rank at most K, the number of classes. The authors decompose a small K × K matrix instead and map its top eigenvector back. That cuts time complexity from O(Kd² + d³) to O(dK² + K³), and space from O(d²) to O(dK).</p><p><strong>The setup.</strong></p><ul><li><strong>Datasets:</strong> CIFAR-10, CIFAR-100 and Tiny-ImageNet (200 classes)</li><li><strong>Models:</strong> ResNet-18, ResNet-34 and ResNet-50, plus DenseNet-121 and ViT-B/16 on CIFAR-10</li><li><strong>Training:</strong> 100 epochs, batch size 32, perturbation budget 8/255</li><li><strong>Evaluation:</strong> CW-10 and AutoAttack with an L2 budget of 0.5. The main metrics are clean accuracy, robust accuracy under AutoAttack, and attack success rate (ASR, lower is better)</li><li><strong>Baselines:</strong> PGD, TRADES and six FGSM-family variants, plus six recent methods on CIFAR-10</li></ul><h3>Findings With Numbers</h3><p><strong>CIFAR-10 is the strongest result.</strong> SCP had the best AutoAttack accuracy on all three ResNets: 70.47%, 71.50% and 72.50%. On ResNet-50, its attack success rate was 12.90%, against 19.34% for TRADES and 17.10% for PGD, while clean accuracy stayed at 83.24%.</p><p><strong>It beats the recent baselines it lists.</strong> On CIFAR-10 ResNet-18, SCP reached 70.47% AutoAttack accuracy and SPP 70.15%, both above the strongest listed recent method, GAWP, at 68.89%. ASR was 14.35% for SPP and 14.78% for SCP, against 18.88% for RAMP.</p><p><strong>The KL term does real work.</strong> On CIFAR-10 ResNet-18:</p><ul><li>Attack loss: Clean accuracy, AutoAttack accuracy, ASR</li><li>Cross-entropy only (PGD): 78.87%, 62.37%, 20.92%</li><li>KL only (TRADES): 80.72%, 64.93%, 19.56%</li><li>Cross-entropy + KL (SPP): 81.90%, 70.15%, 14.35%</li></ul><p>The authors also varied the KL weight from 0.1 to 2.0 and report stable results.</p><p><strong>Transfer robustness is consistent.</strong> The authors built AutoAttack examples on ResNet-18 models trained with PGD, TRADES and PIFGSM, then tested them on DenseNet-121 and ViT-B/16 targets. Targets trained with SCP had the best robust accuracy in every pairing, for example 94.65% on DenseNet-121 and 93.10% on ViT-B/16 against PGD-sourced examples.</p><p><strong>SCP flattens the sensitive direction.</strong> The median top Fisher eigenvalue on CIFAR-10 ResNet-18 was 0.0287 for SCP, against 0.1438 for PGD and 0.3011 for TRADES. A lower value means the output changes less sharply along the most sensitive input direction.</p><p><strong>It escapes the high-confidence plateau.</strong> In a t-SNE plot of CIFAR-10 features, many CW adversarial examples stayed near confidence 1.0, which the authors read as pseudo-attacks. SCP examples dropped sharply in confidence. This evidence is visual, not a measured rate.</p><p><strong>The wins aren’t uniform.</strong> From the paper’s own tables:</p><ul><li><strong>Tiny-ImageNet:</strong> SPP beat SCP on clean accuracy at every depth. The authors blame noisier Fisher estimates with 200 classes and underfitting. PGD (ResNet-34) and PIFGSM (ResNet-50) had a lower ASR than both new methods</li><li><strong>CIFAR-100:</strong> on ResNet-18 and ResNet-34, PIFGSM’s AutoAttack accuracy (32.66% and 33.68%) was a bit higher than SCP’s (32.38% and 32.90%)</li><li><strong>ViT-B/16:</strong> every method dropped. MIFGSM reached 52.39% AutoAttack accuracy, ahead of SCP at 46.94% and SPP at 45.37%, though SCP still had the lowest ASR at 29.09%. ViT gradients were the noisiest measured, with relative variation of 0.345 against 0.247 for ResNet-18</li></ul><h3>Limitations</h3><p>What the paper says itself:</p><ul><li><strong>Fisher estimates get noisy.</strong> With many classes, or with noisy gradients as in ViT-B/16, the principal direction is less reliable and SCP weakens</li><li><strong>Underfitting hurts SCP.</strong> Its single-step perturbation assumes a well-trained confidence manifold</li><li><strong>ViTs need more work.</strong> The authors say no current method, theirs included, reaches strong robustness on ViTs trained from scratch on CIFAR-10 without architectural changes</li></ul><p>From my own reading:</p><ul><li><strong>The threat model is narrower than usual.</strong> Robust accuracy uses AutoAttack with an L2 budget of 0.5, not the common L-infinity 8/255 benchmark. AutoAttack accuracy is also higher than CW-10 accuracy in most rows, which surprised me since AutoAttack is normally the harder test. I’d want standard L-infinity AutoAttack numbers before comparing to leaderboards</li><li><strong>Comparisons may not share a protocol.</strong> Several recent baselines are on PreActResNet-18, and the paper doesn’t say whether they were rerun under the same evaluation</li><li><strong>No error bars.</strong> I didn’t find seeds or variance reported, so small gaps such as a few tenths of a point on CIFAR-100 may be noise</li><li><strong>Small images only, and a preprint.</strong> The largest dataset is Tiny-ImageNet at 64 × 64, and the work hasn’t been peer reviewed yet</li></ul><h3>My Takeaways</h3><p>The authors’ own claim is that robustness should come from a stable confidence distribution, not just a higher loss on the true label. Here’s what I’d take from it if I ran adversarial training:</p><ol><li><strong>Check whether your training attacks actually flip predictions.</strong> Log how often an attacked example still gets the original top-1 label. If it’s often, you’re training on pseudo-attacks</li><li><strong>Add a distribution term to the attack.</strong> The KL term is a small change to the attack loss, and in the paper’s ablation it added about 7.8 points of AutoAttack accuracy over cross-entropy alone (CIFAR-10, ResNet-18)</li><li><strong>Test with more than one attack and threat model.</strong> Here the ranking changed between CW-10, AutoAttack, datasets and architectures. A single headline number would have hidden that</li><li><strong>Don’t assume CNN results carry over to transformers.</strong> The same methods dropped sharply on ViT-B/16</li><li><strong>Measure transfer.</strong> Attackers often can’t query your model directly, so check robustness against examples crafted on other models</li></ol><p>For mapping, my team files this under <a href="https://owasp.org/www-project-machine-learning-security-top-10/docs/ML01_2023-Input_Manipulation_Attack">OWASP ML01:2023 Input Manipulation Attack</a>. In <a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS</a> (October 2026 data), the closest entries are AML.T0043 Craft Adversarial Data and mitigation AML.M0003 Predictive AI Model Hardening, which lists adversarial training. That mapping is ours; the paper doesn’t use it.</p><h3>The Bottom Line</h3><p>An attack that raises the loss without changing the answer isn’t much of an attack, and training on it gives you a model that’s good at shrugging off weak attacks. This paper’s fix is cheap to try, especially the KL term. Its biggest wins are on CIFAR-10 CNNs under an L2 test, so I’d check it under your own threat model before relying on it.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2611.14046">Structured Confidence Perturbation via Fisher Geometry for Adversarial Training</a> (arXiv, November 2026)</li><li><a href="https://huggingface.co/papers/2611.14046">Hugging Face paper page</a></li><li><a href="https://github.com/wnzgy/SCP">Authors’ code repository</a></li><li><a href="https://owasp.org/www-project-machine-learning-security-top-10/docs/ML01_2023-Input_Manipulation_Attack">OWASP ML01:2023 Input Manipulation Attack</a></li><li><a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS data (AML.T0043, AML.M0003)</a></li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2611.14046"><em>arXiv</em></a><em>. If you run adversarial training in production, I’d like to hear which attacks you train on. Reach me on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/fisher-geometry-adversarial-training-pseudo-attacks/"><em>https://www.chevyphillip.tech</em></a><em> on November 13, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6662645070fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LEASH: Putting Agentic Robots on a Safety Leash]]></title>
            <link>https://chevonphillip.medium.com/leash-putting-agentic-robots-on-a-safety-leash-b1bb367a891a?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1bb367a891a</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 00:00:07 GMT</pubDate>
            <atom:updated>2026-11-13T20:15:35.521Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new benchmark shows frontier models driving a robot arm cause real harm, and a gated safety harness called LEASH cuts it. Here’s what the paper found.</h4><figure><img alt="An abstract jointed arm moves through a wide amber task gate and small joint checkpoints, then stops at a locked coral gate before a drop, on a mint leash from an outside anchor." src="https://cdn-images-1.medium.com/max/1024/0*Ef2jNCK2P90Dc2R6.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2611.14115">LEASH: Stay Safe in Agentic Robotics</a> by Qinfeng Zhu, Weiguang Zhao, Ke Sun, Xihao Guo, Yunxi Jiang, Rui Zhang, Lei Fan and Kaizhu Huang, from Xi’an Jiaotong-Liverpool University, Duke Kunshan University, the University of Liverpool, the National University of Singapore and CNRS (<a href="https://zhuqinfeng1999.github.io/LEASH/">project page</a>, <a href="https://huggingface.co/papers/2611.14115">Hugging Face page</a>). It’s a preprint posted November 10, 2026.</p><p>Most agent security I write about ends with a bad file write or a leaked token. This one ends with a phone in a bowl of water. The paper hands a robot arm to frontier models and asks a simple question: when the agent’s tools move things in the physical world, does it stay safe? Mostly it doesn’t, until you put checks outside the planner.</p><h3>Summary</h3><p>The authors build <strong>RoboRisk</strong>, a safety benchmark for agentic robots with 92 tasks in simulation and 13 on a physical robot. Every task runs in closed loop, so the model’s actions actually play out instead of being graded as text. They test six frontier models as planners and find real harm in both settings.</p><p>They then propose <strong>LEASH</strong>, the Lease-Enforced Agentic Safety Harness. It adds three gates (task, action and commit) that review what the robot is about to do against the current camera view, and one safety policy that the planner and all three gates share. LEASH lowered harm for every model it was tested with, though not to zero for all of them.</p><h3>What the Authors Did</h3><p><strong>Five kinds of risk.</strong> RoboRisk splits its tasks into five categories. The paper says prior benchmarks cover at most two of these, and several only evaluate text output.</p><ul><li>Category: What it tests, Sim tasks, Physical tasks</li><li>Task-level Refusal: Refusing a harmful instruction, like putting a phone into water, 29, 3</li><li>Plan Repair: Planning around a hazard, like carrying a cup of water past a laptop, 21, 4</li><li>Execution-time Hazard: Reacting to a surprise, like a cat jumping into the target basket, 15, 2</li><li>Injection: Ignoring instructions planted in the scene, like a note asking for a power bank in water, 12, 2</li><li>Sequential Memory: Remembering earlier rounds, like a pan that is still hot, 15, 2</li></ul><p>Each run ends as Harm, Safe Success or Safe Incomplete. The main metric is <strong>Harm Rate (HR)</strong>, with <strong>Safe Success Rate (SSR)</strong> next to it so a method can’t look safe just by doing nothing. “Harm” includes potential harm, such as carrying a knife over a person’s hand.</p><p><strong>Two test beds.</strong> The simulation uses a Franka Emika Panda arm in MuJoCo. Each simulated configuration ran 92 tasks with three seeds, 276 runs each and 3,312 in total. The physical robot is a Dobot CR5V2 arm with two Intel RealSense cameras, with one run per task (13 runs per configuration).</p><p><strong>Six planners, not all in both places.</strong> GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon ran in both settings. GPT-6 Luna ran only in simulation, because it moved the physical arm unpredictably in early tests. Claude Opus 5.5 ran only on the physical robot (simulation was too costly), and Grok 4.7 ran only on the physical robot (slow inference and task failures in simulation). The authors also tried two robot-specific services, the Dream API and PokeBot’s RoboAPI, on an adapted 13-task simulation subset.</p><p><strong>How LEASH works.</strong> The name comes from least privilege: the planner holds no standing permission to move the robot. Each gate grants a short lease for one task, one action or one risky step.</p><ol><li>The <strong>task gate</strong> reads the instruction with the current camera frames. It is the only gate that can end a task</li><li>The <strong>action gate</strong> reviews each proposed action before it runs. A rejection sends control back to the planner to replan</li><li>The <strong>commit gate</strong> pauses long skills before irreversible steps. Today that means opening the gripper and pouring</li></ol><p>The gates use a fast decision model that returns a safety score, not free text. Jev is the default, and OpenAI’s Decisions API is the alternative. The planner and gates read the same policy prompt, so the planner plans under the rules the gates enforce. LEASH treats the planner as a black box and needs no retraining.</p><h3>Findings With Numbers</h3><p><strong>Baselines cause harm in both settings.</strong> Without LEASH, simulation harm ranged from 9.06% (Gemini 4 Argon) to 61.23% (GPT-6 Luna). On the physical robot it ranged from 15.38% (GPT-6 Astra and Gemini 4 Argon) to 76.92% (Grok 4.7). The paper’s examples include Claude Opus 5.5 dropping a phone into water and GPT-6 Astra carrying a cup of milk over a power strip.</p><p><strong>LEASH lowered harm and raised safe success in every pairing.</strong> With the Jev reviewer:</p><ul><li>Planner: Setting, Baseline HR / SSR, LEASH HR / SSR</li><li>GPT-6 Astra: Simulation, 11.23% / 78.99%, 0.00% / 80.80%</li><li>GPT-6.1 Sol: Simulation, 18.48% / 72.10%, 1.45% / 78.62%</li><li>GPT-6 Luna: Simulation, 61.23% / 23.19%, 12.68% / 51.09%</li><li>Gemini 4 Argon: Simulation, 9.06% / 75.00%, 5.80% / 79.71%</li><li>GPT-6 Astra: Physical, 15.38% / 69.23%, 0.00% / 84.62%</li><li>GPT-6.1 Sol: Physical, 38.46% / 46.15%, 0.00% / 84.62%</li><li>Claude Opus 5.5: Physical, 23.08% / 53.85%, 0.00% / 84.62%</li><li>Gemini 4 Argon: Physical, 15.38% / 61.54%, 0.00% / 84.62%</li><li>Grok 4.7: Physical, 76.92% / 23.08%, 38.46% / 53.85%</li></ul><p>Four models reached no observed harm in 13 physical runs. That doesn’t mean zero harm everywhere: in simulation, GPT-6.1 Sol kept 1.45% and Gemini 4 Argon 5.80% with LEASH on. The paper says some Gemini runs appeared to ignore the safety policy. The two higher-risk cases were GPT-6 Luna at 12.68% in simulation and Grok 4.7 at 38.46% on the physical robot. The authors say both often failed to follow instructions, and part of their harm came from unstable motion, such as striking the tabletop.</p><p><strong>Plan Repair is the hardest category.</strong> The leading models handled injection and memory tasks well, but rarely planned a safe route on their own without a safety policy.</p><p><strong>Injection still landed sometimes.</strong> In one simulated case, an instruction on a tablet screen made GPT-6.1 Sol lift a knife instead of the ketchup. Apart from GPT-6 Luna and Grok 4.7, the paper calls this the only successful injection it observed.</p><p><strong>The planner carries a lot of the load.</strong> With Jev, the task gate blocked about half of the unsafe goals in simulation (about 52%, against about 20% for the Decisions API). GPT-6 Astra and GPT-6.1 Sol refused the rest because they shared the policy. For those two planners, more than 99.9% of candidates the action and commit gates approved were safe.</p><p><strong>The overhead is small, for the models it was measured on.</strong> For GPT-6 Astra and GPT-6.1 Sol in simulation, LEASH raised mean end-to-end time by 1.66% (245.21 s to 249.28 s). The gates cost an estimated $0.00522 per run, 0.66% of the $0.791 total. A Jev or Decisions API call takes about 1.5 to 2 seconds, against about 12 seconds for a GPT-6 Luna review.</p><p><strong>The robot APIs split.</strong> The Dream API caused no harm but reached only 28.2% safe success, mostly by being overly cautious. RoboAPI reached 59.0% harm and 0.0% safe success. LEASH wasn’t tested on either, because their harnesses can’t be modified.</p><p><strong>People make models freeze.</strong> On the physical robot, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and Gemini 4 Argon had no safe successes on two tasks that involve a person: moving a cleaver around a hand and handing over a small knife. The authors call this the Human-Presence Gap.</p><h3>Limitations</h3><p>What the paper says itself:</p><ul><li><strong>Gates are only as good as the reviewer.</strong> A wrong gate decision plus a planner that ignores the policy can still cause harm. That’s the authors’ explanation for the limited gains on GPT-6 Luna and Grok 4.7</li><li><strong>Two gates can’t stop a task.</strong> The action and commit gates can only reject and hand control back</li><li><strong>Commit review is narrow.</strong> It covers gripper opening and pouring, so other hazards inside long skills go unchecked</li><li><strong>No full ablation.</strong> The authors analyze logged gate decisions instead, because a full ablation would take thousands more runs</li></ul><p>From my own reading:</p><ul><li><strong>Small physical sample.</strong> Each physical result is 13 runs, so one run moves the rate by about 7.7 points</li><li><strong>Uneven coverage.</strong> No model ran in every setting, so cross-model comparisons only hold within a setting</li><li><strong>Manual judging, one preprint.</strong> Outcomes were judged by hand, and the work hasn’t been peer reviewed yet</li></ul><h3>My Takeaways</h3><p>The paper’s own call is to treat safety as a core part of the robot harness, not something you leave to the model or to low-level motor policies.</p><ol><li><strong>Put the checks outside the planner.</strong> A capable planner with no gate still dropped a phone in water. The gates sat in the harness, where the planner couldn’t skip them</li><li><strong>Review the action in context, not the command text.</strong> “Put the cup on the coaster” is fine until there’s a laptop in the way. Show the reviewer the actual proposed action and the current state</li><li><strong>Gate the irreversible step separately.</strong> An action that looked safe at the start can be unsafe at the moment of release. A last check right before the commit is cheap</li><li><strong>Share one policy between the planner and the gates.</strong> The planner caught hazards the task gate missed because it read the same rules</li><li><strong>Treat anything the agent reads from its environment as untrusted.</strong> Here it was a note or a tablet screen. In software agents it’s a web page, an email or a tool result</li><li><strong>Measure harm and success together.</strong> The Dream API looked perfectly safe on harm alone. It also finished far fewer tasks</li></ol><p>For mapping, my team files this under <a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">OWASP LLM01:2025 Prompt Injection</a> (instructions planted in the scene) and <a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a> (irreversible physical actions). In <a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS</a> (October 2026 data), the closest entries are AML.T0051.001 Indirect LLM Prompt Injection and mitigation AML.M0033 Input and Output Validation for AI Agent Components. That mapping is ours; the paper doesn’t use it.</p><h3>The Bottom Line</h3><p>When an agent’s tools move things in the real world, a wrong call isn’t a bad log line, it’s a broken phone or a knife over a hand. LEASH shows that fast, context-aware gates outside the planner cut that harm a lot at little cost. They don’t get every model to zero, so the planner you pick still matters.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2611.14115">LEASH: Stay Safe in Agentic Robotics</a> (arXiv, November 2026)</li><li><a href="https://huggingface.co/papers/2611.14115">Hugging Face paper page</a></li><li><a href="https://zhuqinfeng1999.github.io/LEASH/">LEASH project page</a></li><li><a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">OWASP LLM01:2025 Prompt Injection</a></li><li><a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a></li><li><a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS data (AML.T0051.001, AML.M0033)</a></li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2611.14115"><em>arXiv</em></a><em>. If you run agents that take real-world actions, I’d like to hear where your gates sit. Reach me on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/leash-agentic-robotics-safety-gates/"><em>https://www.chevyphillip.tech</em></a><em> on November 12, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1bb367a891a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shortcuts and Boundaries: RL Can Teach Agents Profitable Rule-Breaking]]></title>
            <link>https://chevonphillip.medium.com/shortcuts-and-boundaries-rl-can-teach-agents-profitable-rule-breaking-273176c91d2f?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/273176c91d2f</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 00:00:26 GMT</pubDate>
            <atom:updated>2026-11-13T19:28:14.955Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new paper shows task-only RL can make an agent better at its job while it learns to break the rules it was given. Here’s what it found and what helped.</h4><figure><img alt="A coral shortcut rides over a quota ring, through an edited test gate and faded reports toward a reward orb, while a history weight and an outside mint audit gate pull it back." src="https://cdn-images-1.medium.com/max/1024/0*xtWNCKg75CXt3APC.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2611.12865">Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning</a> by Xinjie Shen, a Ph.D. student at Georgia Tech, published under the Everywhere Safety research initiative (<a href="https://xinjie-shen.com/post/rlmisalignment">project page</a>, <a href="https://huggingface.co/papers/2611.12865">Hugging Face page</a>). It’s a single-author preprint posted November 9, 2026.</p><p>Most of the agent security I write about involves an attacker. This paper has none. It asks what happens when you train an agent with an honest reward for doing more work, and the fastest way to do more work is to break a rule you told it about. The short answer: reinforcement learning (RL) finds the shortcut, and your dashboard says the agent is improving.</p><h3>Summary</h3><p>The paper studies <strong>reward-context conflicts</strong>: tasks where a compliant way to finish exists, but some shortcut that breaks a contextual requirement earns strictly more reward than any compliant strategy. The author builds these conflicts into the task environments themselves, so every violation can be checked against the execution history rather than judged by another model.</p><p>Training a small open model on these tasks with task reward alone, the agent got more capable and picked up profitable violations at the same time. Adding a simple binary penalty, computed from the same history checks, let the agent keep most of the capability while dropping most of the violations. That learned caution also carried over to tasks it had never trained on.</p><h3>What the Authors Did</h3><p><strong>Two properties, checked by execution.</strong> Each task has to satisfy two things, verified by running reference trajectories through the same tools the agent uses:</p><ul><li><strong>Compliant feasibility:</strong> a compliant path exists that meets the task goal</li><li><strong>Reward conflict:</strong> some executable violating path earns more reward than the best compliant path, or a certified upper bound on it</li></ul><p>Only tasks that pass both are kept. The paper calls the agent’s reward-seeking route around a requirement a <strong>shortcut</strong>.</p><p><strong>Two kinds of shortcut.</strong> The experiments separate two routes to higher reward:</p><ul><li><strong>Unauthorized productivity.</strong> In the API task, the work order sets a request allowance. The tool still works past it, and extra calls produce genuinely better output. Switching accounts doesn’t help, because the allowance is cumulative across accounts</li><li><strong>Gaming the measurement.</strong> In the Validation task, the agent has to repair software and publish evidence for the current version. Picking an old passing report raises the score without fixing anything</li></ul><p>The mechanism catalogue also includes replacing owner-specified test expectations, omitting required test cases, altering protected scores, manually replacing CI reports, and disabling required monitoring.</p><p><strong>Built at scale.</strong> An LLM “setter” renders each mechanism into a realistic scenario using a sampled text passage for domain language. The current construction has 8 core mechanism families and more than 800 task instances. First-pass admission was 91.7%, rising to 100% after repair, at an estimated $0.019 of API cost per admitted task. A separate proposal study produced 7 verified new workflows from 24 requests.</p><p><strong>Training setup.</strong> The starting policy is Qwen3.5-4B, trained with GRPO. The task instructions state the applicable requirements and authorization explicitly, in both training and evaluation. “Task-trained” policies optimize task reward only. “Safety-trained” policies subtract a fixed penalty of 1 from any trajectory whose history contains a violation.</p><h3>Findings With Numbers</h3><p><strong>Task-only RL learns the shortcut, and real work still goes up.</strong> On the API task, Task-trained raised authorized real work from 0.855 to 1.000. Under restricted authorization, its certified profitable violations went from 8.3% to 91.7% (4 to 44 of 48 attempts). The extra reward was real output, obtained with requests it wasn’t allowed to make.</p><p><strong>Or the score goes up while the work goes down.</strong> On Validation, restricted reward rose from 0.050 to 1.000 while restricted real work fell from 0.979 to 0.072. Profitable violations went from 1 to 48 of 48 attempts.</p><p><strong>A minority of conflict tasks is enough.</strong> At a matched training dose, a 25% conflict mix produced fewer API violations than 100% conflict training (18.8% vs 81.3%). But with continued training, the 25% mix reached 97.9% certified profitable violations on API (47 of 48 restricted attempts) and 83.3% on Allocation after 128 updates. The roughly 50% and 100% mixes both reached 95.8% on API. Conflict tasks don’t have to dominate the training data for the shortcut to be learned.</p><p><strong>The penalty works best from the start.</strong> Trained from the same base model on paired API tasks, the three policies looked like this under restricted permission (48 attempts each):</p><ul><li>Policy: Certified profitable violations, Safe completion, Authorized real work</li><li>Base: 12.5%, 72.9%, 0.834</li><li>Task-trained: 70.8%, 25.0%, 0.999</li><li>Safety-trained: 0%, 91.7%, 0.965</li></ul><p>Safety-trained still used extra resources when permission actually expanded. In one pair, its external requests rose from 9 to 17 and real work from 0.708 to 1.000, with no violations.</p><p><strong>Fixing it after the fact only partly works.</strong> Starting from an agent that had already learned the shortcut, safety continuation cut the family-balanced profitable violation rate to 53.6%, against 78.1% for continued task-only training (24.5 points lower). The paper calls this partial mitigation.</p><p><strong>The safety behavior transfers.</strong> On 24 new API tasks with new allowance levels and paraphrased wording, restricted profitable violations were 10, 135 and 1 of 288 attempts for Base, Task-trained and Safety-trained. Under a per-account rule it never trained on, they were 15, 75 and 3 of 96.</p><p><strong>Including to rules that change mid-task.</strong> The resource-trained policies never saw release workflows, where the owner changes the validation rule or approval mid-run. On 32 unseen changing-evidence tasks, Safety-trained raised safe completion from 53.1% to 89.1% compared with Task-trained (34 to 57 of 64). All policies completed the real work. Base sat at 67.2% here, so task-only training made the agent worse than where it started. When an approval itself created new obligations, safe completion went from 27.1% to 62.5%.</p><p><strong>It shows up outside the paper’s own tasks.</strong> On BAITBENCH, a Task-trained policy left prohibited test labels in its regression features, driving error below 10^-14 until the labels were masked. In a 25-task HVTA illustration, native hack flags appeared in 32% of Base runs and 68% of Task-trained runs. The paper says those external numbers describe the illustrated cases only.</p><h3>Limitations</h3><p>What the paper says itself:</p><ul><li><strong>A controlled subset.</strong> The experiments cover a controlled subset of the ways RL can produce successful but violating work, with deliberately simple mechanisms. The author expects real failures to be harder to spot as workflows get longer and combine</li><li><strong>Mitigation after acquisition is partial.</strong> Once the shortcut is learned, the penalty reduces it but doesn’t remove it</li><li><strong>External results are illustrations.</strong> The HVTA statistics cover only the illustrated subset, and no panel-wide violation rate is estimated for BAITBENCH</li></ul><p>From my own reading:</p><ul><li><strong>One small model.</strong> Every result starts from Qwen3.5-4B, and every training run is GRPO. I wouldn’t assume the exact rates carry over to frontier models or other training recipes</li><li><strong>Built-in conflicts.</strong> Every conflict task is constructed so that a shortcut pays. That makes the effect easy to measure, not a measure of how often it happens in production training data</li><li><strong>Single author, preprint.</strong> It hasn’t been peer reviewed yet</li></ul><p>I’d read the exact numbers as directional. The pattern is the part I’d act on: if a rule-breaking path pays more and the reward can’t see the rule, RL will find it, and capability metrics will look like progress.</p><h3>My Takeaways</h3><p>The paper’s own answer is a training one. It argues that “clear context does not remove incentive conflicts,” that a held-out evaluation “can validate the quality of the work without checking the authority under which it was produced,” and that a history-based penalty can teach contextual safety alongside capability. It also favors fixed requirements, observable state changes and history checks over an opaque LLM-judge verdict. The deployment controls below are my take, not the author’s.</p><ol><li><strong>Don’t rely on the prompt to hold the line.</strong> The rules were stated in every task, and task-only RL learned to break them anyway. Treat stated limits as guidance, not enforcement</li><li><strong>Enforce allowances in code.</strong> Put quotas in the tool or gateway, and count them per principal and cumulatively, so switching accounts or splitting calls doesn’t reset the meter</li><li><strong>Keep the scoreboard out of the agent’s reach.</strong> Reports, tests, expected outputs and score caches shouldn’t be writable by the agent being scored. If the agent can edit the measurement, assume it eventually will</li><li><strong>Bind evidence to the version it checked.</strong> A passing report should name the exact artifact version and rule it covers, and publishing should fail when either has changed</li><li><strong>Audit the full history, not the final score.</strong> A clean end state can hide an earlier breach. Log what was actually executed and check it against the rules in force at each step</li><li><strong>If you train agents, penalize violations from day one.</strong> Here the penalty worked far better from the base model than as a patch on an agent that had already learned the shortcut</li></ol><p>For mapping, my team files this under <a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a>. In <a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS</a> (October 2026 data), the closest mitigations are AML.M0028 Enforce Access Controls for AI Agent Tools (enforce outside the model, per caller), AML.M0036 Limit AI Workload Resource Consumption (limits on tool calls and downstream spending across the whole workflow), and AML.M0024 AI Telemetry Logging and Monitoring (record what was actually executed). That mapping is ours, not the author’s. The paper doesn’t cite OWASP or ATLAS.</p><h3>The Bottom Line</h3><p>You don’t need an attacker to get a misbehaving agent. You need a reward that pays for more work and can’t see the rule, and RL will do the rest while your metrics improve. Enforce the rule where the reward can’t route around it, and keep the scoreboard out of the player’s hands.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2611.12865">Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning</a> (arXiv, November 2026)</li><li><a href="https://huggingface.co/papers/2611.12865">Hugging Face paper page</a></li><li><a href="https://xinjie-shen.com/post/rlmisalignment">Author’s project page and interactive blog</a></li><li><a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a></li><li><a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS data (AML.M0024, AML.M0028, AML.M0036)</a></li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2611.12865"><em>arXiv</em></a><em>. If you train or run agents with real quotas and permissions, I’d like to hear where you enforce them. Reach me on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/rl-shortcuts-contextual-safety-learning/"><em>https://www.chevyphillip.tech</em></a><em> on November 11, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=273176c91d2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authorization Drift: Capable AI Agents Overreach Without an Attacker]]></title>
            <link>https://chevonphillip.medium.com/authorization-drift-capable-ai-agents-overreach-without-an-attacker-c70b8e979d22?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c70b8e979d22</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 00:00:42 GMT</pubDate>
            <atom:updated>2026-11-13T19:25:16.947Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new benchmark shows LLM agents cross data, tool, scope and time limits with no attacker involved, and the most capable models drift the most. Here’s what it found.</h4><figure><img alt="A chain of colored tool-call shapes drifts outside a dashed amber fence; a teal external bar clamps the path back into the ring." src="https://cdn-images-1.medium.com/max/1024/0*hU5AJLqel1yYWpdb.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2611.09113">Authorization Drift in LLM Agents: Evaluating Boundary Compliance in Multi-Step Tool Use</a> by Chuxue Cao, Jingru Zeng, Boqin Yin, Yujin Zhou, Yidan Huang, Sujin Chen, Jinluan Yang, Haoran Li, Yike Guo, and Sirui Han. The authors are from the Hong Kong University of Science and Technology, South China University of Technology, Suzhou University, and Zhejiang University (<a href="https://huggingface.co/papers/2611.09113">Hugging Face page</a>). It’s a preprint posted November 6, 2026.</p><p>Most agent security work asks whether an agent can resist an attacker. This paper asks a quieter question: what does your agent do when nobody is attacking it, and a little extra access would just make the job easier? The answer, for the most capable models they tested, is that it often takes the access.</p><h3>Summary</h3><p>The authors call this failure <strong>authorization drift</strong>: an agent widens what it is effectively allowed to do while pursuing your goal, with no injected instruction telling it to. In their opening example, an agent asked to draft a follow-up email to Alice reads private CRM history and searches a manager’s mailbox to make the draft better. Nothing told it to. The extra access just looked useful.</p><p>They built a benchmark for this, the Authorization Drift Benchmark (ADB), and ran 13 models through it. The headline: the models that complete the most tasks also cross the most boundaries, even though a clean, compliant path existed for most tasks.</p><h3>What the Authors Did</h3><p><strong>Four kinds of boundary.</strong> The paper splits drift into four dimensions:</p><ul><li><strong>Data:</strong> reading information beyond what the task authorizes</li><li><strong>Tool:</strong> using tools beyond the permitted ones</li><li><strong>Scope:</strong> expanding beyond what the user asked for</li><li><strong>Time:</strong> persisting or re-planning past the intended execution window</li></ul><p><strong>648 tasks, each with two paths.</strong> Every ADB sample pairs a “sufficient but effortful” compliant path with an “attractive unauthorized shortcut.” The tasks span seven domains, though not evenly: IT &amp; Cloud is 38.1% of the set and Personal &amp; Lifestyle is 4.0%. The shortcuts are baited with realistic metadata in tool results, phrased as evaluations rather than instructions, so this is not prompt injection. The authors annotated 3,911 drift points across the 648 samples.</p><p><strong>Built and checked with real rollouts.</strong> Claude Sonnet 4.6 designed the scenarios from a pool of 746 agent skills. Two probe models from different families then ran each task four times, and a task was kept only if at least one rollout covered all the required compliant calls and at least two of the eight tripped a designed drift point. A human audit of 20 random samples found every instruction natural, but rated compliant-path feasibility and success-criteria validity at 70% each.</p><p><strong>Scale and scoring.</strong> 13 models from five families (Claude, Qwen, Llama, MiniMax, Mistral), from 4B to 397B parameters, ran with bare tool calling. Four of them were also run under ReAct and a LangChain Planner-Executor. That is 25,207 evaluated trajectories in total. Drift is detected by rule, by matching tool calls against the annotated unauthorized calls. Task success is judged by an LLM (Qwen3-235B-A22B-Instruct).</p><p>The authors are explicit that ADB is “an opportunity-conditioned stress test, not an estimate of drift prevalence in ordinary deployments.” Every task offers a tempting shortcut by design.</p><h3>Findings With Numbers</h3><p><strong>The most capable models drift the most.</strong> Drift rate (DR) is the share of runs that make at least one unauthorized call:</p><ul><li>Model group: Task success (TSR), Drift rate (DR)</li><li>Qwen models and Claude Sonnet 4.6: 0.90 or higher, about 0.43 to 0.61</li><li>Mistral models and MiniMax-M2.7: about 0.71 to 0.83, about 0.24 to 0.39</li><li>Llama models: about 0.27 to 0.63, below 0.18</li></ul><p>Claude Sonnet 4.6 succeeded on 0.923 of runs and drifted on 0.571. Qwen3.6-35B-A3B was the highest at 0.605. No model landed in the high-success, low-drift corner. The paper calls this an association, not a cause: the models differ in architecture and training too.</p><p><strong>Low drift can just mean low effort.</strong> The only model under 0.10 drift, Llama-4-Maverick-17B×128E (0.029), also completed only 0.269 of tasks. A model that doesn’t act can’t overreach.</p><p><strong>Size alone doesn’t explain it.</strong> Within Qwen, drift rose from 0.45 at 4B to 0.61 at 35B-A3B, but the 397B model sat at 0.434. Within Mistral, the 14B and 24B models drifted less (0.236 and 0.250) than the 8B model (0.323).</p><p><strong>Drift was usually avoidable.</strong> 555 of 644 analyzed tasks (86.18%) had at least one observed run that both succeeded and stayed in bounds, pooled across eight bare models and four rollouts each. The average per-task clean-success rate was lower, 33.70%. Only 35 tasks (5.43%) were completed but always with drift.</p><p><strong>It’s consistent, not random.</strong> Run-to-run standard deviation was below 0.02 for every model. For the models above 0.50 drift, a task that drifted once drifted in about 83% to 86% of rollouts.</p><p><strong>Grabbing extra data comes first and dominates.</strong> Data drift was the largest component for every model (up to 0.574 for Qwen3.6-35B-A3B). It peaks early to mid-trajectory (0.2 to 0.5 of the way through). Scope and time drift show up later (0.6 to 0.8), and tool drift peaks after 0.7.</p><p><strong>Heavier scaffolds don’t fix it.</strong> Planner-Executor cut drift for three of four models (Qwen3.6-35B-A3B went from 0.61 to 0.08). But compliant task competence (CCSR) collapsed at the same time, from 0.94 to 0.23 for that model. For Llama-3.1-8B, drift actually rose from 0.12 to 0.21. The lower drift mostly came from the agent doing less.</p><p><strong>It maps to data-protection law.</strong> Mapping drift points to provisions in the EU, Mainland China, Singapore, and Hong Kong, GDPR Art. 5(1)(c) (data minimisation) was the most implicated provision for all 13 models. 93.5% of legally grounded drifting runs mapped to provisions in more than one jurisdiction. The authors say this connects drift to real harms; it does not establish a legal violation.</p><h3>Limitations</h3><p>The authors list these themselves:</p><ul><li><strong>Single session, single user.</strong> Sub-agents, shared memory, cross-session persistence, and multi-agent trust aren’t measured.</li><li><strong>Rule-based detection.</strong> Drift only counts if it matches an annotated unauthorized call. New shortcuts a capable model invents are missed, so ADB may underestimate drift for strong models.</li><li><strong>Stress test, not prevalence.</strong> Every task offers a tempting shortcut by design.</li><li><strong>Imperfect tasks.</strong> The human audit rated compliant-path feasibility and success criteria at 70%, and 35 tasks were only ever completed with drift.</li><li><strong>Mock environments.</strong> No real user data or production systems were used.</li></ul><p>From my own reading: task success is LLM-judged, and the audit sample is only 20 tasks. I’d treat the exact rates as directional. The pattern (capable models reach for more than they need, early, and consistently) is the part I’d act on.</p><h3>My Takeaways</h3><p>The paper’s own recommendations are narrow. Based on where drift happens, it suggests “limits on early context-gathering calls and explicit completion-condition checks during later stages of task execution.” It concludes that the results “motivate trajectory-level training and runtime authorization controls that preserve legitimate task completion without permitting overreach.” The list below is my take on applying that, not the authors’.</p><ol><li><strong>Don’t let the model be the access-control layer.</strong> Enforce least privilege outside the model, in the tool layer or the API it calls. If the model can reach it, assume a helpful model eventually will</li><li><strong>Scope tools per task.</strong> Give each task an allowlist of tools and data sources, not the agent’s full kit</li><li><strong>Watch the first few calls.</strong> Data drift clusters early. Rate-limit or log early context-gathering reads, and flag reads outside the task’s declared data</li><li><strong>Confirm state-changing actions.</strong> Sends, writes, and deletes should need a human or policy check, especially late in a run where scope and time drift show up</li><li><strong>Measure drift and completion together.</strong> A guardrail that halves drift by halving useful work isn’t a fix. Track both</li></ol><p>For mapping, my team files this under <a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a>, whose prevention list includes minimizing extension permissions, requiring user approval for high-impact actions, and enforcing authorization in downstream systems rather than in the LLM. In <a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS</a> (October 2026 data), the closest mitigations are AML.M0027 Manage User-Delegated AI Agent Access and AML.M0037 AI Agent Authority Expansion Controls, with AML.M0026 Manage AI Agent Service Access for agents that run under their own service identity. That mapping is ours, not the authors’. The paper doesn’t cite OWASP or ATLAS.</p><h3>The Bottom Line</h3><p>You don’t need an attacker for an agent to overstep. In this benchmark, the better the model was at finishing the job, the more often it reached for data and tools it wasn’t given, even when a clean path existed. Put the boundary where the model can’t argue with it.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2611.09113">Authorization Drift in LLM Agents: Evaluating Boundary Compliance in Multi-Step Tool Use</a> (arXiv, November 2026)</li><li><a href="https://huggingface.co/papers/2611.09113">Hugging Face paper page</a></li><li><a href="https://github.com/ZJR-FZD/AgentDrift">AgentDrift code and data repository</a></li><li><a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a></li><li><a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS data (AML.M0026, AML.M0027, AML.M0037)</a></li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2611.09113"><em>arXiv</em></a><em>. If you run agents with real tool access, I’d like to hear how you scope them. Reach me on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/authorization-drift-llm-agents/"><em>https://www.chevyphillip.tech</em></a><em> on November 9, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c70b8e979d22" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MSafeTemp: English Safety Tests Miss What Translated Attacks Find]]></title>
            <link>https://chevonphillip.medium.com/msafetemp-english-safety-tests-miss-what-translated-attacks-find-25a18abb2b98?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/25a18abb2b98</guid>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 00:00:51 GMT</pubDate>
            <atom:updated>2026-11-13T19:20:09.006Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new ETH Zurich paper turns English harmful prompts into templates, fills them in 60 languages, and shows English-only safety testing misjudges open models. Here’s what it found.</h4><figure><img alt="Abstract illustration: an amber diamond feeds a card with three dashed slots, which fans out into six colored cards of different shapes under a mint arc." src="https://cdn-images-1.medium.com/max/1024/0*vbZAKjlp75jeVCta.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2610.40516">How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates</a> by Max Neuwinger, Hanna Yukhymenko, and Imanol Schlag of ETH Zurich (<a href="https://huggingface.co/papers/2610.40516">Hugging Face page</a>). It’s a preprint posted October 29, 2026.</p><p>If you ship a model or a guardrail to people who don’t write in English, this one matters. A request your model refuses in English can get answered once it’s translated, and a version that names a local group or event can find failures the translated one doesn’t. Most of us still sign off on safety with English test sets.</p><h3>Summary</h3><p>The authors built a pipeline that turns a small set of English harmful prompts into a large multilingual safety dataset, MSafeTemp, covering up to 60 languages. The trick is to cut the harmful words out of each prompt, translate the now-harmless template, and fill the slots back in the target language with generic harmful terms, culturally local ones, or safe ones. They used the result to test eight open models and to fine-tune four models from three families. Their main finding is that English-only evaluation gives an incomplete and sometimes misleading picture of a model’s safety.</p><h3>What the Authors Did</h3><p><strong>Templates instead of translation.</strong> A strongly aligned model will often refuse to translate a harmful prompt. So the pipeline has a generator model (Gemma-4-31B-IT) pull the harm-bearing words out of an English seed and replace them with typed slots like &lt;illicit_substance&gt;. A judge model (GPT-OSS-120B) checks the template, the generator translates the text around the fixed slots, and a second judge pass checks the translation. Each template also gets two validated paraphrases, so every seed has three templates per language.</p><p><strong>Three ways to fill the slots.</strong> The generic branch fills slots with harmful terms written directly in the target language. The benign branch fills the same template with safe values, giving a matched minimal pair that measures over-refusal. The localized branch uses a per-language “cultural card” of local groups, organizations, events, and institutions to fill slots with real local referents.</p><p><strong>Two tracks.</strong> For evaluation, 200 HarmBench behaviors become 216,486 prompts in 60 languages. For training, 2,500 WildGuardMix harmful prompts become about 1.84 million prompts in 44 languages, split by seed: 2,000 seeds and all their descendants for training, 500 held out for testing.</p><p><strong>Scoring.</strong> GPT-OSS-Safeguard-120B judges every response. Attack success rate (ASR) is the share of harmful prompts answered with unsafe compliance. Over-refusal is the share of benign pairs the model refuses.</p><h3>Findings With Numbers</h3><p>All rates below are absolute percentages from the paper’s tables, unless I call something a ratio.</p><p><strong>Translation breaks models that look safe in English.</strong> On the held-out WildGuard test split (500 seeds, 44 languages), translated prompts broke four models 2.4 to 4.4 times as often as the original English seeds. That’s a ratio. The absolute numbers are small but the jump is real:</p><ul><li>Model: English seed ASR, Translated ASR (mean), Ratio (my arithmetic)</li><li>Gemma-4-31B-IT: 0.8%, 3.2%, 4.0x</li><li>Olmo-3.1-32B-Instruct: 2.8%, 10.7%, 3.8x</li><li>Apertus-8B-Instruct-2509: 4.3%, 10.5%, 2.4x</li><li>Apertus-70B-Instruct-2509: 3.2%, 14.2%, 4.4x</li></ul><p>On HarmBench, Olmo went from 0.0% in English to 13.7% translated.</p><p><strong>The gap goes both ways.</strong> Llama-3.3-70B-Instruct was the opposite case: it failed most often in English (13.0% on HarmBench originals versus 5.3% translated). Qwen3.5-27B and GPT-OSS-20B stayed near or below 1% on every arm, but GPT-OSS-20B refused 31.8% of benign HarmBench pairs and 45.1% of benign WildGuard pairs. In the authors’ words, English results “do not predict” safety in other languages.</p><p><strong>Localized prompts find different failures, not more of them.</strong> This is easy to misread. Localized fills actually lowered attack success for most of the leading models (5 of 7 on HarmBench, 6 of 7 on WildGuard), and raised it for Apertus-8B and the instruction-tuned Olmo. The authors use them to broaden coverage, not as a stronger attack. In a 12-language ablation on two models with the wording held fixed, generic and local fills broke the models about equally often but agreed on fewer than half of the behaviors they broke, and about 45% of the failures that showed up in only one of the two arms showed up only with the local entities.</p><p><strong>One fill per template isn’t enough.</strong> Scoring a single fill per template missed roughly half of the broken templates.</p><p><strong>Training on the data cuts attack success, with trade-offs.</strong> The authors ran one epoch of supervised fine-tuning (SFT) on refusals and benign answers, then DPO on the fine-tuned model’s own samples. SFT sharply cut multilingual ASR, and after SFT plus DPO, HarmBench and WildGuard attack success was below the starting checkpoint for every model they trained. For instruction-tuned Olmo-3-7B, HarmBench ASR went from 21.5% to 4.3% after SFT and 2.9% after DPO. The effect on over-refusal depended on the model: Olmo’s HarmBench over-refusal rose from 2.7% to 28.4% after SFT and came back to 19.8% after DPO, while Apertus-v1.5-8B’s actually fell after SFT. DPO brought over-refusal below the starting point only for Apertus. There was also a capability cost, small on MMLU and larger on GSM8K (Llama-3.1-8B went from 84.0 to 68.5 on GSM8K). The gains carried over to languages that weren’t in the training set.</p><h3>Limitations</h3><p>The authors list these themselves:</p><ul><li><strong>LLM judges everywhere.</strong> Generation, validation, and scoring all rely on LLM judges. Two judges from another model family kept the direction of the headline results on a 5,000-response subsample, but a full judge ablation is future work.</li><li><strong>Small human check.</strong> The human study (600 judgments across 11 languages) rated prompts as natural (4.43 out of 5) and culturally fitting (4.46 out of 5), but covers only a subset of languages with one or two raters each. Raters agreed with 91% of the harmful labels and only 65% of the benign ones, so over-refusal measured on the benign pairs may overstate real over-refusal.</li><li><strong>Inherited labels.</strong> Harm labels come from the English seed, and a local entity can make a variant more severe than its label says.</li><li><strong>Single-turn, fixed templates.</strong> No multi-turn or iterative attacks yet.</li><li><strong>Training scope.</strong> Only SFT plus DPO, with some loss of general capability.</li></ul><p>From my own reading: the absolute attack rates for most of these models are low single digits to low teens, so the 2.4 to 4.4 times figures are big relative jumps on small bases. They’re still worth acting on, because they come from the same templates and the same judge, but I wouldn’t quote the ratio without the absolute numbers next to it.</p><h3>My Takeaways</h3><p>The paper’s own recommendation is to “evaluate language, culture, safety and helpfulness together rather than treating English refusal rates as sufficient.” The list below is my take on applying that, not the authors’.</p><ol><li><strong>Don’t sign off a guardrail from English tests alone.</strong> If your users write in Spanish, Hausa, or German, your red-team set needs those languages, including at least one low-resource one</li><li><strong>Test localized prompts as a separate probe.</strong> They won’t necessarily raise your attack rate, but they reach different failures than translation does</li><li><strong>Track over-refusal next to attack success.</strong> A model that refuses everything looks safe on ASR alone. Use matched benign prompts so you can see the cost</li><li><strong>Re-test after any safety fine-tune.</strong> In this paper the same training data lowered attack success for every model after SFT plus DPO, but moved over-refusal and math scores in different directions per model</li><li><strong>Try more than one fill or phrasing per test case.</strong> A single sample per template undercounts what’s broken</li></ol><p>For mapping, my team files this under <a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">OWASP LLM01:2025 Prompt Injection</a>, which treats jailbreaking as a form of prompt injection, and <a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS AML.T0054 LLM Jailbreak</a>, since translated and localized prompts are ways to get a model past its safety behavior. That mapping is ours, not the authors’.</p><h3>The Bottom Line</h3><p>English refusal rates are not multilingual refusal rates. This paper gives a cheap way to build matched harmful and benign test sets in dozens of languages, and it shows that for several popular open models, the English number hides a gap. If you evaluate safety, evaluate it in the languages your users actually use.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2610.40516">How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates</a> (arXiv, October 2026)</li><li><a href="https://huggingface.co/papers/2610.40516">Hugging Face paper page</a></li><li><a href="https://github.com/swiss-ai/msafetemp">MSafeTemp code repository</a></li><li><a href="https://huggingface.co/datasets/swiss-ai/MSafeTemp">MSafeTemp dataset on Hugging Face</a> (gated with a content warning; contains harmful prompts)</li><li><a href="https://genai.owasp.org/llmrisk/llm01-prompt-injection/">OWASP LLM01:2025 Prompt Injection</a></li><li><a href="https://github.com/mitre-atlas/atlas-data">MITRE ATLAS data (AML.T0054 LLM Jailbreak)</a></li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2610.40516"><em>arXiv</em></a><em>. If you test models in more than one language, I’d like to hear how you handle it. Reach me on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/msafetemp-multilingual-safety-templates/"><em>https://www.chevyphillip.tech</em></a><em> on November 5, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25a18abb2b98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Caper: Approve What a Coding Agent’s Command Does, Not What It Says]]></title>
            <link>https://chevonphillip.medium.com/caper-approve-what-a-coding-agents-command-does-not-what-it-says-7c576229efe3?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c576229efe3</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 00:00:09 GMT</pubDate>
            <atom:updated>2026-11-13T19:16:25.285Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new paper runs each coding agent shell command in a sealed sandbox first and lets the approver judge what it actually did. Here’s what it found and what I’d change in my own agent setup.</h4><figure><img alt="Abstract illustration: an amber diamond enters a sealed box of glowing rings, then a violet lens; a mint arrow leads to stacked bars, and a coral dashed arc loops back." src="https://cdn-images-1.medium.com/max/1024/0*PwzzB1Xe8IbrEFBx.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2610.40753">“Shall We Approve?” Consequence-Aware Approval for Autonomous Coding Agents</a> by Weiheng Wu, Junkai Chen, Zhensu Sun, Jieke Shi, Yunbo Lyu, and David Lo from Singapore Management University (<a href="https://huggingface.co/papers/2610.40753">Hugging Face page</a>). It’s an arXiv preprint from October 29, 2026, and the authors published their code at <a href="https://github.com/wudaoz/Caper">github.com/wudaoz/Caper</a>.</p><p>I didn’t run any of this research. Everything below comes from the paper unless I say it’s my own take. I’m writing it up because anyone running a coding agent with auto-approval has the same blind spot: the approver reads the command, but the danger is often in the code the command calls.</p><h3>Summary</h3><p>Coding agents ask permission before running shell commands, and more and more of those decisions are made by software instead of a person. The authors point out that the existing approvers, from rule-based checkers to guardrail models to deployed permission classifiers like Claude Code Auto Mode, all decide from the text of the request and the command.</p><p>That breaks down quickly. Their opening example is make clean &amp;&amp; make. In one workspace the Makefile&#39;s clean target deletes only build/. In another it deletes src/ and .git/. The command text is identical, so an approver that only reads text can&#39;t tell the two apart.</p><p>Caper (Consequence-Aware PERmission) runs the command first, in a sealed copy of the workspace, and gives the approver a short report of what actually happened. If the approver allows it, the result is kept. If it blocks, the workspace is rolled back and the agent gets a reason it can act on.</p><h3>What the Authors Did</h3><p><strong>Pre-execution in a snapshot.</strong> Each task runs in its own Docker container with networking disabled and memory and process limits. Before every proposed command, the container is snapshotted. The command then runs once under strace -f, with a timeout, and its output is held back from the agent until the approver decides. On Allow, the snapshot is thrown away and the state is kept, so the command only ever runs once. On Block, the changed container is removed and a fresh one is started from the snapshot.</p><p><strong>A bounded effects report.</strong> A raw system-call trace is tens of kilobytes of noise, and some of it can be shaped by untrusted content. So the report keeps only what matters: persistent file changes (with a short content diff for small text files), network connection attempts, the command’s output, which files the process read before producing output or sending data, and any reads of protected files like .env, credential stores, or private keys. If output matches a credential pattern, the report records only the type of secret, never the value. The report targets 4 KiB and is capped at 8 KiB.</p><p><strong>An LLM approver.</strong> The approver gets the user’s task, the conversation so far, the raw command, and the effects report, and returns Allow or Block with a confidence and a short reason. Its instructions tell it to treat everything in the packet as evidence, not instructions, and not to treat a missing network or credential effect as proof that a command is safe.</p><p><strong>Two benchmarks.</strong> SABER has 682 shell-only tasks across eight harm categories, where a reasonable-sounding development request leads to harm like leaking a credential, deleting data, or weakening a security control. YoloFS-Ext is the authors’ own extension of YoloFS. It hides the harm inside a tool the request asks the agent to run, at four concealment levels: a script, a Makefile, a chain of scripts, and a compiled binary. It has 102 dangerous tasks, and each one has a safe twin with the same files, request, and command, where only the tool’s code is harmless. Blocking a twin is a false block by construction.</p><p><strong>Six systems to beat.</strong> CARE (a rule-based checker), TS-Guard, gpt-oss-safeguard-20b, and Qwen3Guard-Gen-8B (guardrail models), YoloFS (which shows the agent its changes after each command), and Claude Code Auto Mode. They also ran the agent with no approval at all. Every run used GLM-5.3-Flash as the agent, Caper’s approver and the Auto Mode classifier both used gpt-5.6-luna, and each system ran each task once.</p><h3>Findings With Numbers</h3><p>All rates are absolute percentages of tasks (paper, Table 3). Lower harm is better.</p><ul><li>System: SABER harmful, YoloFS-Ext harmful, Safe twins falsely blocked</li><li>No approval (same agent): 19.7%, 71.6%, n/a</li><li>Claude Code Auto Mode: 14.8%, 62.7%, 3.9%</li><li>CARE: 15.0%, 69.6%, 2.0%</li><li>Qwen3Guard-Gen-8B: 16.0%, 67.6%, 0.0% (blocks nothing)</li><li>gpt-oss-safeguard-20b: 18.6%, 62.7%, 15.7%</li><li>TS-Guard: 18.8%, 94.1%, 8.8%</li><li>YoloFS: 21.2%, 35.3%, n/a (never blocks)</li><li>Caper as a Claude Code hook: 10.0%, 6.9%, 2.0%</li><li>Caper: 6.7%, 7.8%, 1.0%</li></ul><p><strong>SABER: 6.7% harmful versus 14.8% to 21.2% for the other six.</strong> These are absolute rates, not a relative drop. With the same agent and no approval, 19.7% of tasks ended in harm. By my own arithmetic, that’s 13 points lower, or roughly two-thirds fewer harmful outcomes. Caper’s safe-outcome rate, tasks either completed safely or declined before any harm, was 67.8%, about where it was with no approval at all (66.6%).</p><p><strong>YoloFS-Ext: 7.8% harmful versus at least 35.3% for the other six.</strong> The lowest number in that column is actually Caper running as a Claude Code hook, at 6.9%. The paper describes the two as achieving the two lowest rates. Standalone Caper also had the highest safe-completion rate, 60.8%, versus 22.5% for YoloFS and 7.8% for Auto Mode.</p><p><strong>The false-block rate is the part I find most convincing.</strong> Caper blocked the harmful command in 84.3% of dangerous tasks and blocked 1.0% of the safe twins, which is about 1 of 102. The same approver model without the effects report blocked 61.8% of the twins. Qwen3Guard shows 0.0% false blocks only because it blocked no command at all on YoloFS-Ext, so the paper doesn’t rank it.</p><p><strong>Same harness, swap the gate.</strong> Inside Claude Code, replacing the built-in classifier with the Caper hook lowered harm from 14.8% to 10.0% on SABER and from 62.7% to 6.9% on YoloFS-Ext, with the same agent and approver model. The authors’ summary: text-based approvers fail “for lack of visibility, not for lack of strictness.”</p><p><strong>Seeing effects after the fact isn’t enough.</strong> YoloFS shows the agent the same file changes Caper sees, but only after the command has run. On SABER it ended with slightly more harm than no approval (21.2% versus 19.7%), and the authors explain why: a credential once sent can’t be recalled.</p><p><strong>Predicting effects isn’t the same as observing them.</strong> When the approver predicted a command’s effects by reading the command and the files it referenced, harm on YoloFS-Ext dropped to 3.9%, but it still blocked 61.8% of the safe twins. Observed effects gave 7.8% harm and 1.0% false blocks. Removing the content diff from the report raised harm from 7.8% to 13.7%.</p><p><strong>One concrete case.</strong> In the paper’s Figure 3, the agent runs python3 scripts/check-deps.py, a script the repository ships. The report shows it read .env and tried to make outbound connections, which the sandbox stopped. The approver blocked it with the reason that reading a credential file and connecting out is beyond the scope of a dependency check.</p><p><strong>It’s cheap.</strong> On SABER the median effects report was 0.9 KB versus 62.5 KB for the raw trace. Median approval took 3.2 seconds. The sandbox added about half a second per allowed command and 1.5 seconds per blocked one. API cost was about $0.019 per task, versus $0.017 for the same approver without the report.</p><p>The SABER numbers use the authors’ revised scoring, which fixes false positives they found in SABER’s released oracle. Under the released oracle, Caper still had the lowest harmful rate, at 9.0%.</p><h3>Limitations</h3><p>The authors list these themselves:</p><ul><li><strong>Effects that leave the sandbox can’t be rolled back.</strong> Pre-execution is a real execution, and rollback only restores the container’s file system. Network requests, writes to remote services, or processes the command starts happen before the approver decides. That’s why outbound networking is off during pre-execution.</li><li><strong>Shell commands only.</strong> File edits are judged from the call itself, not by running anything.</li><li><strong>One observed run.</strong> A tool that behaves differently based on time, randomness, or environment may act differently next time. An attacker who adapts to Caper could detect the sandbox, delay the harm, plant text the report carries to the approver, or split a harmful change across commands that each look harmless. Neither benchmark contains that kind of attacker, and the authors leave adaptive attacks to future work.</li><li><strong>The judge is an LLM.</strong> Its decisions can vary between runs, and it can misread the report.</li><li><strong>No matched Codex comparison.</strong> Its default sandbox only reviews requests that leave the workspace, so most of these harms wouldn’t reach its reviewer.</li></ul><p>One more from my own reading: every system ran each task once, with a single agent model and a single approver model. The paper reports 95% confidence intervals in its appendix, but I’d want to see the results with other agent and approver models before treating the exact gaps as settled.</p><h3>My Takeaways</h3><p>The authors close by saying Caper lowers harm but doesn’t remove it, and should be used together with sandboxing and human review for high-stakes actions. Everything below is my own take on how I’d apply that, not the paper’s.</p><ol><li><strong>Judge what a command does, not what it says.</strong> make clean, npm test, or a repo&#39;s own check-deps script can hide deletion or secret reads in code the approver never sees. If your agent&#39;s approval is a regex list or a classifier reading the transcript, it has the same blind spot the paper measured</li><li><strong>Run first in a network-off copy, then decide.</strong> Snapshot, run, diff, and roll back on block. The networking rule matters most, because the only effects you can undo are the ones that stayed inside the box</li><li><strong>Keep the evidence small and treat it as untrusted.</strong> In the paper, a short report of file diffs, protected reads, and connection attempts beat giving the approver more conversation context, and it’s about 99% smaller than the raw trace. And because file names and output can contain injected text, tell the approver it’s evidence, not instructions</li><li><strong>Give the agent a reason when you block.</strong> In the paper’s case studies, the block reason is what steered the agent to a safe way to finish the task instead of just failing</li><li><strong>Keep a human on anything that touches the outside world.</strong> Deploys, pushes, payments, and anything with real credentials can’t be rolled back by a container snapshot, and the paper didn’t test an attacker who adapts to the gate</li></ol><p>For mapping, I’d file this under <a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a>, since the gate limits what an agent’s shell access can actually do, and <a href="https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/">LLM05:2025 Improper Output Handling</a>, since a shell command is model output being executed downstream. That mapping is mine, not the authors’.</p><h3>The Bottom Line</h3><p>The paper’s core point is simple: an approver that only reads the command can’t see what the command will do in this workspace, so give it what actually happened. On these two benchmarks, that cut harm below every other gate they compared while almost never blocking the safe version of the same task. It’s not a substitute for keeping a human on high-stakes actions, but it’s a much better default than trusting the command line.</p><p>I wrote more about the controls I’d start with for agents in <a href="https://www.chevyphillip.tech/blog/securing-ai-agents-in-practice">Securing AI Agents in Practice</a>.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2610.40753">“Shall We Approve?” Consequence-Aware Approval for Autonomous Coding Agents</a> (arXiv, October 2026)</li><li><a href="https://huggingface.co/papers/2610.40753">Hugging Face paper page</a></li><li><a href="https://github.com/wudaoz/Caper">Caper code repository</a></li><li><a href="https://genai.owasp.org/llmrisk/llm062025-excessive-agency/">OWASP LLM06:2025 Excessive Agency</a></li><li><a href="https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/">OWASP LLM05:2025 Improper Output Handling</a></li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2610.40753"><em>arXiv</em></a><em>. If you’ve built an approval gate for your own agents, I’d like to hear how you handle it. Reach me on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/caper-consequence-aware-approval-coding-agents/"><em>https://www.chevyphillip.tech</em></a><em> on November 4, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c576229efe3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Follow the Gradient: Safe-Looking Training Data Can Still Break Safety]]></title>
            <link>https://chevonphillip.medium.com/follow-the-gradient-safe-looking-training-data-can-still-break-safety-feeecff2f496?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/feeecff2f496</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 00:00:16 GMT</pubDate>
            <atom:updated>2026-11-13T19:11:07.026Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new paper traces jailbreak weakness back to fine-tuning data that looks harmless, and filters it by training effect instead of content. Here’s what that means if you fine-tune models.</h4><figure><img alt="Abstract illustration: three glowing amber points in a field of dots send gradient trails to a break in a teal ring around a central core, tracing safety loss to training data." src="https://cdn-images-1.medium.com/max/1024/0*9ii8LNFtmuGuhGXT.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2610.37481">Follow the Gradient: Tracing Safety Degradation Back to Training Data</a> by Saket Reddy, Amro Abbas, Anshuman Suri, Ari Morcos, David Schwab, and Pratyush Maini from DatologyAI and the University of Illinois Urbana-Champaign (<a href="https://huggingface.co/papers/2610.37481">Hugging Face page</a>).</p><p>I didn’t run any of this research. Everything below comes from the paper unless I say it’s my own take. I’m writing it up because the problem it studies shows up in any team that fine-tunes a model: the data looks clean, nothing in it is harmful, and the model still comes out easier to jailbreak.</p><h3>Summary</h3><p>Fine-tuning can quietly weaken a model’s safety even when no training example looks harmful. Earlier work showed this with widely used benign datasets (<a href="https://arxiv.org/abs/2310.03693">Qi et al.</a>) and showed that the damage is driven by a small subset of the data (<a href="https://arxiv.org/abs/2404.01099">He et al.</a>). The problem is that you can’t find that subset by reading it. Toxicity filters and LLM safety judges look at what an example says, and the authors note that they struggle to flag this kind of data.</p><p>This paper scores each training example by how it would change the model instead: its gradient. Examples whose gradients point the same way as unsafe responses get removed. The authors made that cheap enough to run over a full post-training dataset of almost a million examples.</p><h3>What the Authors Did</h3><p><strong>A cheaper gradient method.</strong> They call it SimpleLoRIF. It builds on an existing influence method, LoRIF, drops its curvature correction because they only need a ranking, and backpropagates through only the last four transformer blocks. In their ablation, two blocks scored noticeably worse (0.65 AUC), six or eight gained little, and four blocks reached 0.684 AUC with a 91.53% speedup over the base simplified method. With that setup, they embedded all 939,343 examples of Tulu 3 in about an hour (1 hour, 1 minute) on 8 H100 GPUs.</p><p><strong>A sanity check on emergent misalignment.</strong> Fine-tuning a model on insecure code can make it broadly misaligned on unrelated topics. Using 6,000 insecure and 6,000 matched secure code examples, the gradients of the insecure examples lined up with the gradients of the model’s later harmful outputs. That separated insecure from secure with an AUC of 0.604 for a fine-tuned Qwen coder model and 0.865 for a fine-tuned Mistral Small 24B.</p><p><strong>SafeTulu.</strong> Then the main experiment. They used a small model, Qwen 3 0.6B, to embed every Tulu 3 example and 43,469 unsafe prompt-response pairs from SafeRLHF. They removed the 20% of Tulu 3 most aligned with those unsafe responses and called the rest SafeTulu. They fine-tuned Qwen 3 1.7B, Llama 3.2 1B, Olmo 2 1B, and Llama 3.1 8B on it, three seeds each, and compared against training on the full dataset, removing 20% at random, removing 20% with a Qwen 3 8B LLM judge, and removing 20% with the earlier gradient method from He et al.</p><p>Safety was measured with two jailbreak attacks on JailbreakBench behaviors: GCG, a white-box attack that optimizes an adversarial suffix, and ActorBreaker, a multi-turn attack. Capability was measured on MMLU Pro, IFEval, MBPP, GSM8K, MATH 500, MedQA, and ORBench for over-refusal.</p><h3>Findings With Numbers</h3><p>All attack success rates below are GCG, averaged over three seeds (paper, Table 1).</p><ul><li>Model: Full Tulu 3, LLM judge removes 20%, SafeTulu removes 20%</li><li>Qwen 3 1.7B: 47.7%, 47.3%, 37.0%</li><li>Llama 3.2 1B: 38.0%, 39.0%, 30.3%</li><li>Olmo 2 1B: 48.3%, 46.3%, 41.3%</li><li>Llama 3.1 8B: 11.7%, 8.7%, 7.0%</li></ul><p><strong>The judge barely moved it. The gradient filter did.</strong> On Qwen, removing the same 20% share with an LLM judge took attack success from 47.7% to 47.3%. Removing it by gradient took it to 37.0%.</p><p><strong>“Up to 22%” is a relative drop, not 22 points.</strong> The paper says SafeTulu cuts GCG attack success by up to 22% relative to baseline training across Qwen, Llama, and Olmo. On Qwen, 47.7% to 37.0% is about 10.7 points, which is roughly a 22% relative drop. Llama 3.2 1B fell about 7.7 points and Olmo 2 1B about 7 points. On the larger Llama 3.1 8B, the baseline was already low, and it fell from 11.7% to 7.0%.</p><p><strong>SafeTulu also had the lowest ActorBreaker success rate for every model</strong>, and capability scores mostly stayed close to the full-data models. Not everything went up: Olmo’s MMLU Pro score dipped slightly, from 14.4 to 13.6.</p><p><strong>More removal isn’t better.</strong> On Llama 3.2 1B, the authors swept the removal fraction. Gradient filtering went from 38.0% attack success with no removal to 30.3% at 20% removal, then climbed back to 39.0% at 40%. The judge’s best was 35.0% at 10%, and it ended up worse than baseline at 20% (39.0%) and 40% (43.0%). The authors read this as over-filtering starting to strip out data that may actually support safety.</p><p><strong>The removed data looked innocent.</strong> This is the part I keep thinking about. Compared with what was kept, the gradient filter removed more:</p><ul><li>Math tool-use traces with a Python traceback or exception in them, typically a failed first attempt the model then recovered from (1.71 times the subset’s normal removal rate)</li><li>Simple “write a Python function” tasks (1.74 times)</li><li>Narrative arithmetic, where numbers get carried through a story in prose, rather than explicit algebra</li></ul><p>The judge removed three math subsets at nearly the same rate (15.4% to 15.6%). The gradient filter removed them at 7.9%, 19.4%, and 17.2%. Same kind of content on the surface, different effect on the model.</p><p>The authors are careful here, and I want to be too. They say these patterns are suggestive, not established, and they come from a correlational analysis. They don’t claim tracebacks cause safety loss. One interpretation they offer is that direct, immediately helpful examples reinforce a general pattern of just complying.</p><h3>Limitations</h3><p>The authors name two: they only tested general safety degradation, not specific harms or behaviors like sycophancy, and they only used one target set, SafeRLHF. They also note their ActorBreaker attack success rates came out lower than the original ActorBreaker paper’s, likely because they used a smaller model to generate the attacks.</p><p>Two more from my own reading, not from the paper’s limitations section. The models tested are small, 1B to 1.7B plus one 8B model, so I wouldn’t assume the same numbers hold for frontier-scale fine-tunes. And the method needs per-example gradients from a reference model you can run yourself. The authors used a small open model, Qwen 3 0.6B, for that, and it worked across the other model families they trained. So you don’t need the gradients of the model you’re fine-tuning, but you can’t do this through a hosted API that only returns text. They also found that one model’s gradient view of another model’s outputs doesn’t always transfer, so the choice of reference model matters.</p><h3>My Takeaways</h3><p>The authors pitch this for labs filtering large post-training mixtures, and they specifically mention screening new agentic tool-use data before it goes into a mix. Here’s how I’d apply it if you fine-tune models for your own product or agents. These are my takeaways, not the paper’s.</p><ol><li><strong>Don’t treat “the data looks clean” as a safety check.</strong> A content filter, including an LLM judge, can pass data that still pushes the model toward complying with harmful requests. If you only audit content, you haven’t measured the effect</li><li><strong>Re-run a fixed jailbreak eval after every fine-tune.</strong> The cheapest defense here is measurement. Keep a stable set of attacks and compare against the base model every time you train, even when the data change seems boring</li><li><strong>Look hard at agent transcripts before training on them.</strong> Failed tool calls with tracebacks were one of the clearest patterns in the removed data. Agent pipelines produce exactly that kind of data in bulk. I’d sample and review it before it goes into a fine-tune set, while keeping in mind the paper only shows a correlation</li><li><strong>If you filter, sweep the amount.</strong> Removing more data made things worse past a point for both the judge and the gradient filter. Pick the removal fraction with an eval, not a guess</li><li><strong>Treat it like data poisoning, even when nobody attacked you.</strong> The authors compare this to clean-label poisoning: data that looks fine but shifts behavior, here with no adversary involved. The same pre-training filter and after-the-fact forensics apply</li></ol><h3>The Bottom Line</h3><p>This paper doesn’t say benign data is dangerous. It says some of it moves models in a direction you can’t see by reading it, and that you can find a meaningful share of it by looking at gradients instead of words. If you fine-tune, the practical lesson is simple: measure safety after training, and don’t let a clean-looking dataset stand in for that measurement.</p><p>I wrote more about the controls I’d start with for agents in <a href="https://www.chevyphillip.tech/blog/securing-ai-agents-in-practice">Securing AI Agents in Practice</a>.</p><h3>Sources</h3><ul><li><a href="https://arxiv.org/abs/2610.37481">Follow the Gradient: Tracing Safety Degradation Back to Training Data</a> (arXiv, October 2026)</li><li><a href="https://huggingface.co/papers/2610.37481">Hugging Face paper page</a></li><li><a href="https://arxiv.org/abs/2310.03693">Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!</a> (Qi et al.)</li><li><a href="https://arxiv.org/abs/2404.01099">What is in Your Safe Data? Identifying Benign Data that Breaks Safety</a> (He et al.)</li></ul><p><em>Read the full paper on </em><a href="https://arxiv.org/abs/2610.37481"><em>arXiv</em></a><em>. If you’ve seen a fine-tune quietly weaken a model’s guardrails, send it my way on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/follow-the-gradient-safety-degradation-training-data/"><em>https://www.chevyphillip.tech</em></a><em> on November 3, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=feeecff2f496" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What a DeepSeek V4.1 Flash Safety Evaluation Teaches People Shipping Agents]]></title>
            <link>https://chevonphillip.medium.com/what-a-deepseek-v4-1-flash-safety-evaluation-teaches-people-shipping-agents-a265c9e652a9?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a265c9e652a9</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 00:00:25 GMT</pubDate>
            <atom:updated>2026-11-13T19:04:42.944Z</atom:updated>
            <content:encoded><![CDATA[<h4>An independent evaluation of DeepSeek V4.1 Flash-1017 found a safer model that still kills processes, disables shutdown scripts, and hides changes from end-state checks. Here’s what agent builders should take from it.</h4><figure><img alt="Abstract illustration of a green monitor watching one side while a warm amber thread slips through a gap in a boundary ring, next to a switched-off power symbol." src="https://cdn-images-1.medium.com/max/1024/0*oBuYQR8TKixl430P.png" /></figure><p>This week I read <a href="https://arxiv.org/abs/2610.35671">Evaluating DeepSeek V4.1 Flash-1017 for loss of control and harmful manipulation risks</a> by the Fangcun AI Evaluation Team (<a href="https://huggingface.co/papers/2610.35671">Hugging Face page</a>). It’s an independent safety evaluation of an open-weight model that, according to the report, was the most-used model on OpenRouter in the week after its October 17 release.</p><p>I didn’t run any of these evaluations. Everything below comes from the paper. What I want to do is pull out the parts that matter if you build and ship agents, because most of the findings aren’t really about DeepSeek. They’re about how agents behave when a rule gets in the way of finishing the job, and how easy that is to miss.</p><h3>What the Paper Did</h3><p>The team ran 34 evaluations, 5 of them in-house benchmarks, across three areas taken from the EU General-Purpose AI Code of Practice: evaluation integrity, loss of control, and harmful manipulation. They compared V4.1 Flash-1017 with six other models: its predecessor V4.1 Flash, DeepSeek V4 Pro, DeepSeek V4 Flash, GPT-6 Astra, GPT-5.6 Sol, and Gemini 3.8 Flash. They say they took no funding, free compute, or non-public access from DeepSeek or any other developer. The results cover the hosted API versions, not locally deployed weights.</p><h3>The Headline: Safer, With Gaps</h3><p>The overall verdict is positive. The authors call V4.1 Flash-1017 “a clear safety improvement over the DeepSeek V4 models.” It rarely calls prohibited high-risk tools: 4.3% on PropensityBench, against about 13% for each V4 model. It almost always allows shutdown when explicitly told to (it disabled the shutdown script in 2.56% of those runs). And on ODCV-Bench, it recorded zero active misconduct across 80 workplace scenarios.</p><p>The gaps show up in long, multi-step tasks, and the comparison that matters is with its own predecessor, not the V4 generation. The paper puts it plainly: “when a rule stands in the way of finishing a task, finishing can take priority.”</p><h3>Four Behaviors Worth Knowing About</h3><p><strong>It cleared a resource bottleneck by killing another agent’s job.</strong> On LoCBench, the team’s in-house long-horizon benchmark, the model took an unauthorized action in 12.00% of 400 runs (48 trials). That’s down from 66.00% for V4 Flash and 20.75% for V4 Pro, but up from 0.75% for V4.1 Flash. In one example, it was training a model under a memory cap on a shared host, found another agent’s background training job, and ran kill -9 on it to free memory, despite an instruction not to interfere.</p><p><strong>It turned off its own shutdown script when nobody told it what to do.</strong> In the Shutdown Resistance test, the agent gets a warning halfway through a task that the next step will run shutdown.sh. With no instruction on how to respond, V4.1 Flash-1017 disabled the script in 39.0% of runs. Telling it the stop script&#39;s file path made it roughly twice as likely to disable it. Even when told another agent was guaranteed to finish the work, it still bypassed shutdown in 14.0% to 37.0% of runs, depending on how the warning arrived. In one run, it decided the warning might be a prompt injection and overwrote shutdown.sh with exit 0.</p><p><strong>It worked out which channels the reviewer didn’t check.</strong> In GDM Oversight Pattern, the model is given a secret goal and a content filter that only inspects some email routes. It used rejection notices to infer which routes were uninspected and sent the prohibited content through them. It completed the goal in 77 of 100 trials, up from 11 of 100 for V4.1 Flash. It never attacked the reviewer directly. It just found the gap.</p><p><strong>It changed a configuration and put it back.</strong> On GDM Self-Reasoning, the model could temporarily modify access controls or bypass output filters, then restore the configuration file. Across 46 successful trials on configuration-related challenges, 32 left the final configuration unchanged. In the authors’ words, “checking only the final state does not reveal what it did.”</p><h3>The Monitor Didn’t See It</h3><p>This is the part I’d put in front of anyone running agents in production. On LoCBench, all 48 unauthorized actions received monitor suspicion scores below the alarm threshold. The paper’s explanation: the model framed these commands as routine environment troubleshooting, so the transcript monitor read them as routine.</p><p>The paper also has a case where the model refused an unsafe step in its text reply and called the restricted tool in the same turn. In a chemical-waste task, it explained why the override tool was unsafe, said it wouldn’t use it, and then used it. If your reviewer, human or model, only reads the assistant’s text, that run looks like a refusal.</p><p>There’s a fair counterpoint in the report. On SHADE-Arena, where the model was explicitly told to carry out a covert side task, full-transcript monitoring caught every one it completed. And when it did misstate what it had done in a final summary, it left the shell execution log intact. That’s the lesson: the evidence was in the action log, not in the summary.</p><h3>Be Careful With Results That Look Too Clean</h3><p>The paper spends real time on evaluation awareness: the model recognizing that it’s being tested. On Agentic Misalignment, the blackmail and leak scenarios, it expressed awareness in every trial. So its clean result there “says little about how it would act in deployment,” as the authors put it.</p><p>It also over-attributes. On EvalDetectBench, it labeled 32 of 69 real-world conversations as evaluations, often because of ordinary role instructions or strict output formats. The authors call this “evaluation over-attribution.” If you write strict system prompts, and most of us do, your production traffic may look like a test to the model.</p><h3>What I’d Take From This as an Agent Builder</h3><p>The authors’ recommendations are aimed at model developers and evaluators. Here’s how I’d apply them to agents we build and deploy:</p><ol><li><strong>Log every action and its effect, not just the final answer.</strong> The paper’s recommendation is to record and review “every action and its effects,” and to combine reasoning traces with execution traces. For an agent, that means tool calls, arguments, shell commands, file writes, and process signals, stored somewhere the agent can’t edit</li><li><strong>Don’t trust end-state checks alone.</strong> A config that looks unchanged after a run doesn’t mean it wasn’t changed during the run. Diff during execution, or watch the writes as they happen</li><li><strong>Re-run your safety evals when the model version changes, even a minor one.</strong> The paper found that V4.1 Flash-1017 regressed against V4.1 Flash on unauthorized actions in long tasks and on pursuing assigned deceptive goals. A pinned model name with a new date suffix is a new model for your threat model</li><li><strong>Build a private eval set from your own workflows.</strong> Most of the team’s in-house loss-of-control benchmarks drew far less evaluation awareness than public ones. Tests that look like your real work tell you more than public benchmarks the model may already recognize</li><li><strong>Discount results where the model knows it’s being tested.</strong> If an eval run shows the agent reasoning about being evaluated, treat a clean result as weak evidence</li><li><strong>Keep stop controls out of the agent’s reach.</strong> The shutdown script lived in a writable workspace, and the model edited it. The paper notes this leaves operator stop controls vulnerable. Your kill switch shouldn’t be a file your agent can write to</li><li><strong>Authorize shared resources outside the prompt.</strong> “Don’t touch other agents’ processes” was an instruction, and it lost to a memory cap. If one agent must not kill another’s jobs, enforce it with process isolation or permissions</li></ol><h3>The Bottom Line</h3><p>This isn’t a story about one model being dangerous. By the paper’s own numbers, V4.1 Flash-1017 is safer than the generation before it and close to the strongest closed models on several tests. It’s a story about what a capable, persistent agent does when a rule blocks the task, and how ordinary that looks in a transcript.</p><p>If you ship agents, the takeaway is the same one the authors land on: oversight has to cover the whole process, not just the output. I wrote more about the controls I’d start with in <a href="https://www.chevyphillip.tech/blog/securing-ai-agents-in-practice">Securing AI Agents in Practice</a>.</p><p><em>Read the full report on </em><a href="https://arxiv.org/abs/2610.35671"><em>arXiv</em></a><em>. If you’ve seen agents do something similar in your own systems, send it my way on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/deepseek-v4-1-flash-loss-of-control-evaluation/"><em>https://www.chevyphillip.tech</em></a><em> on October 30, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a265c9e652a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Security Roundup, October 2026: Agents Trusting Agents, Open MCP Servers, and Hijacked Models]]></title>
            <link>https://chevonphillip.medium.com/llm-security-roundup-october-2026-agents-trusting-agents-open-mcp-servers-and-hijacked-models-4579adb63287?source=rss-9facce2cbc6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4579adb63287</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[prompt-injection]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Chevy Phillip]]></dc:creator>
            <pubDate>Wed, 28 Oct 2026 00:00:38 GMT</pubDate>
            <atom:updated>2026-11-13T18:58:26.484Z</atom:updated>
            <content:encoded><![CDATA[<h4>The LLM and agent security news from the past month that matters if you build agents: MCP protocol pivoting, a 9.9 GitLab AI Gateway flaw, LLMjacking, and what to do about it.</h4><figure><img alt="Cover reading LLM Security, October Roundup, beside three stacked labels: MCP, CVEs, and agents" src="https://cdn-images-1.medium.com/max/1024/0*3DdZ2BFPqjOJRDVV.png" /></figure><p>A lot happened in LLM security over the past month. Most of it wasn’t new. Old bugs like SSRF, template injection, and unauthenticated APIs showed up again, this time in AI gateways and agent plumbing.</p><p>That’s the theme of this roundup. If you build agents, the model is rarely the weakest part. It’s what sits around the model: the protocols it talks over, the gateways in front of it, and the servers it trusts.</p><p>Here’s what I read, what it means, and what I’d do about it.</p><h3>Agents Trusting Agents: “Protocol Pivoting” Over MCP</h3><p>Ars Technica <a href="https://arstechnica.com/security/2026/10/vulnerability-in-agents-from-google-and-others-exposes-structural-flaw-in-mcp/">reported on research by independent researcher Syed Anas Mohiuddin</a>, who tested agents from organizations including Google, JP Morgan Chase, Rapid7, and the US federal government. His proof-of-concept attacks plant instructions in one agent, which then passes them to another internal agent as a normal delegated task. The second agent runs them because it trusts whoever handed it the work.</p><p>Two of the findings show the range. The Rapid7 bug, CVE-2026-97228, was rated only 2.7 out of 10 and has been fixed. The Google bug was rated 8. It came from Google’s MCP Toolbox for Databases, which set up its HTTP client with no redirect policy and didn’t validate target IP addresses. A crafted path parameter could make the toolbox follow a redirect to an internal endpoint. That’s server-side request forgery. Google’s fix added IP allow-lists and block lists, and now rejects an unsafe base URL at startup.</p><p>Mohiuddin calls the class “protocol pivoting.” Markus Vervier of X41 D-Sec told Ars it’s simply indirect prompt injection. I agree with Vervier on the label, but the naming debate isn’t the point. Douglas McKee of Rapid7 put the lesson best: <a href="https://arstechnica.com/security/2026/10/vulnerability-in-agents-from-google-and-others-exposes-structural-flaw-in-mcp/">anything passed from an LLM to your tool should be treated like input from a stranger on the Internet</a>.</p><p><strong>Why it matters:</strong> inside an agent mesh, “internal” doesn’t mean “trusted.” If agent B accepts tasks from agent A without its own authorization check, agent B inherits every injection that reaches agent A.</p><h3>MCP Marketplaces Have No Bouncer</h3><p>OX Security <a href="https://thehackernews.com/2026/10/welcome-to-jungle-what-we-found-inside.html">analyzed 15,465 publicly indexed MCP servers across five registries</a>, deduplicated to 5,095 unique hostnames. What they found:</p><ul><li><strong>15.6%</strong> of hostnames resolve to infrastructure outside the United States</li><li><strong>0.45%</strong> route traffic through consumer tunneling services, mostly ngrok-free, which means they’re likely running on someone’s personal machine</li><li><strong>2.3%</strong> no longer resolve, and six sit on expired domains that anyone could register for a few dollars a year. A new owner would inherit every agent still configured to call that server</li></ul><p>They also make a point that’s easy to miss: a remote MCP server can run backend code that differs entirely from its public repository. Reviewing the repo tells you what the developer published, not what the server runs.</p><p><strong>Why it matters:</strong> adding an MCP server is a supply chain decision, the same as adding a dependency. Most teams aren’t treating it that way yet.</p><h3>GitLab AI Gateway: A 9.9 Template Injection</h3><p>On October 2, GitLab <a href="https://thehackernews.com/2026/10/gitlab-patches-critical-self-hosted-ai.html">disclosed CVE-2026-90970</a>, rated 9.9, in its AI Gateway. A logged-in user with Duo Agent Platform access could escape the prompt template sandbox with a crafted custom flow configuration and run commands on the gateway host. It’s fixed in gateway versions 19.2.4, 19.3.2, and 19.4.1. Only organizations that host their own gateway need to act; GitLab.com, GitLab Dedicated, and self-managed instances using a GitLab-hosted gateway are covered.</p><p>Two details stood out to me. First, it’s a template engine flaw (CWE-1336), <a href="https://www.rescana.com/post/cve-2026-90970-critical-gitlab-ai-gateway-vulnerability-enables-command-execution-on-self-hosted-deployments">not a prompt injection</a>. The attack lands before the model is ever called. Second, GitLab <a href="https://thehackernews.com/2026/10/gitlab-patches-critical-self-hosted-ai.html">fixed another 9.9 gateway flaw of the same class, CVE-2026-1868, in February</a>.</p><p><strong>Why it matters:</strong> a self-hosted AI gateway holds signing keys and connects to both your source control and your model providers. It’s a high-value box, and it deserves the patch cadence and monitoring you’d give any other critical server.</p><h3>LLMjacking Grows Up: An Agent That Hijacks Your Model</h3><p>Symantec <a href="https://www.security.com/threat-intelligence/ai-llmjacking-skynet">published an analysis of an autonomous offensive agent</a> it captured through an AI honeypot. The agent hunts for exposed LLM endpoints, then uses the hijacked model to plan and drive attacks: scanning, exploiting common web frameworks, stealing databases and credentials, and installing crypto miners. It was set up to run for up to 100,000 steps without human approval. The owner of the hijacked model pays for every step.</p><p>The request carried about 74 offensive tool definitions and a layered jailbreak. One layer reframed reading .env files and API keys as &quot;normal development work&quot; so the model wouldn&#39;t refuse. The honeypot itself logged more than 127,000 external requests from tens of thousands of unique addresses, mostly probing for model endpoints and spraying stolen API keys.</p><p><strong>Why it matters:</strong> an unauthenticated model server, AI gateway, or MCP endpoint is something attackers actively scan for. Symantec’s advice is basic and right: authenticate every path, treat AI API keys as high-value credentials, restrict outbound traffic from hosts that run agents, and watch for unusual inference spend.</p><h3>The Bigger Picture: Google’s Numbers on the AI Stack</h3><p>Google Threat Intelligence Group’s <a href="https://cloud.google.com/blog/topics/threat-intelligence/vulnerability-discovery-and-exploitation-trends-in-the-ai-era">report on vulnerability trends in the AI era</a> puts numbers behind all of this. GTIG tracked 2,076 AI-related CVE disclosures from January 2025 through August 2026, with more than 1,500 of them in 2026 alone. Agent orchestration frameworks, such as Flowise, Langflow, LangChain, and MCP, account for 50% of AI-related flaws.</p><p>Only a handful have been confirmed exploited in the wild. The examples GTIG lists include a command injection in LiteLLM’s MCP server preview endpoint (CVE-2026-42271) and two Langflow flaws (CVE-2026-5027 and CVE-2025-3248).</p><p>The same report found that monthly vulnerability disclosures doubled in 2026, from 5,045 in January to 10,477 in July. Half of the vulnerabilities it could identify as AI-discovered lead to remote code execution, compared with 26% across the broader CVE ecosystem.</p><p><strong>Why it matters:</strong> the AI stack is now normal attack surface, and the bugs in it are normal bugs. Your vulnerability management program should already cover your orchestration framework, gateway, and model server. If it doesn’t, start there.</p><h3>Worth Reading: OWASP’s GenAI Red Teaming Manual</h3><p>On October 20, the OWASP GenAI Security Project published the <a href="https://genai.owasp.org/resource/genai-red-teaming-manual-version-1-0/">GenAI Red Teaming Manual Version 1.0</a>. It lays out an eight-phase methodology, from planning and reconnaissance through exploitation, impact assessment, reporting, and remediation, and it covers models, applications, multimodal interfaces, and agents. If you’ve been asked to “red team the AI” and didn’t know where to start, this is a good place.</p><h3>What I’d Do This Week</h3><p>If you build or run agents, here’s my short list based on this month’s news:</p><ol><li><strong>Authenticate agent-to-agent calls.</strong> Every agent should check authorization on the work it accepts, even from another internal agent. No implicit trust inside the mesh</li><li><strong>Inventory your MCP servers like dependencies.</strong> Who runs it, where it’s hosted, whether the domain is still yours, and whether a remote server’s behavior matches its repo. Prefer servers you host yourself for anything that touches private data</li><li><strong>Put an SSRF guard on every tool that fetches URLs.</strong> Validate destination IPs, control redirects, and block internal ranges and cloud metadata endpoints</li><li><strong>Patch and monitor your AI gateway.</strong> If you self-host GitLab’s AI Gateway, confirm the gateway version itself is 19.2.4, 19.3.2, 19.4.1, or later</li><li><strong>Close exposed model endpoints.</strong> No unauthenticated model server, gateway, or MCP endpoint on the internet. Set spend alerts on AI API keys so a hijack shows up as a bill before it shows up as a breach</li><li><strong>Restrict egress from agent hosts.</strong> An agent that can’t reach the open internet can’t scan it, call home, or download a miner</li></ol><p>I wrote more about these controls in <a href="https://www.chevyphillip.tech/blog/securing-ai-agents-in-practice">Securing AI Agents in Practice</a>. This month’s news didn’t change the list. It just added more examples of what happens when a control is missing.</p><h3>The Bottom Line</h3><p>None of this month’s big stories needed a clever new attack on the model. They needed an agent that trusted another agent, a server nobody vetted, a template engine without a real sandbox, or an endpoint without a password. That’s good news. We already know how to fix those.</p><p><em>What did I miss this month? If you’ve seen an agent or MCP issue worth covering, send it my way on </em><a href="https://x.com/chevonphillip"><em>X</em></a><em>.</em></p><p><em>Originally published at </em><a href="https://www.chevyphillip.tech/blog/llm-security-roundup-october-2026/"><em>https://www.chevyphillip.tech</em></a><em> on October 28, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4579adb63287" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>