<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikita Shigarov on Medium]]></title>
        <description><![CDATA[Stories by Nikita Shigarov on Medium]]></description>
        <link>https://medium.com/@nikita.shigarov?source=rss-8264437f1758------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rEzl-7z0h8SwQsqF</url>
            <title>Stories by Nikita Shigarov on Medium</title>
            <link>https://medium.com/@nikita.shigarov?source=rss-8264437f1758------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:23:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikita.shigarov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The verifier is the moat: self-improving RLVR for secure Terraform]]></title>
            <link>https://medium.com/@nikita.shigarov/the-verifier-is-the-moat-self-improving-rlvr-for-secure-terraform-6c859f37db18?source=rss-8264437f1758------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c859f37db18</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Nikita Shigarov]]></dc:creator>
            <pubDate>Wed, 15 Jul 2026 16:08:44 GMT</pubDate>
            <atom:updated>2026-07-16T00:03:05.529Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How a deterministic security scanner — used as a reinforcement-learning reward — turned an open 8B model into something that beats a frontier model where it actually counts. And how the same audit discipline caught my own data leak, then a bug in my own reward function.</em></p><p>Give a frontier model a boring infrastructure request:</p><blockquote>“an S3 bucket for invoices, KMS-encrypted, private.”</blockquote><p>It writes clean, plausible Terraform. It compiles. It looks right. Then a deterministic security scanner reads it and zeroes the whole thing, because the config quietly ships a wildcard in an IAM policy, an unscopeable action, or a hardcoded secret.</p><p>That gap — between <em>looks correct</em> and <em>is verifiably not broken</em> — is the entire project. It’s also where reinforcement learning has an unfair advantage, because “does the scanner pass?” is a reward you can compute exactly, with no human rater and no LLM judge in the loop.</p><p>This is <strong>rl-gym</strong>: a small, reusable post-training platform that trains open models against <strong>verifiable rewards</strong> — with two things most RLVR setups skip: the reward-hacking audit as a first-class citizen, and a <strong>self-improving verifier</strong>: served logs are cross-checked against external scanners to surface blind spots, a big open <em>reasoning</em> model drafts the new rule from the gap, and tests plus a human gate it before merge — so the reward keeps pace with a standard that never stops moving. It’s my submission to the <a href="https://nebius.com/serverless-ai-builders-challenge">Nebius Serverless AI Builders Challenge</a>, and I’m <a href="https://github.com/SkyForce/rl-gym">open-sourcing all of it</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4OGaakat2Y7DI4B8bMkwIA.png" /></figure><h3>The one idea: the reward is a program</h3><p>Most of the interesting recent RL work on LLMs — the reasoning models, the math-and-code fine-tunes — rests on a single decision rule: <strong>if the task is verifiable, you can skip the reward model and let a program grade the output.</strong> Exact-match for math. Unit tests for code. That’s RLVR — Reinforcement Learning with Verifiable Rewards.</p><p>Security is a beautiful fit for this and almost nobody frames it that way. A static analyzer (Checkov-style rules) <em>is</em> a reward function: run it over a Terraform config, count the pass rate, and hard-fail on anything critical. So I built the pipeline around it:</p><pre>SFT warm-start  →  GRPO (RLVR)  →  eval  →  reward-hacking audit  →  serve  →  mine traffic  →  ↺</pre><p>The task is a pluggable Environment — swap it and the whole machine retargets to any domain where a program can grade the output: Kubernetes hardening, CI policy-as-code, IAM least-privilege, semantic rubrics. Nothing in the trainer, eval, or audit is secretly Terraform-shaped.</p><p>The reward has two parts, and the split is the whole trick:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ux_1G8QwNTxSO6ahaZNFRQ.png" /></figure><ul><li><strong>Soft components</strong> in [0,1] — the fraction of applicable security rules that pass — give GRPO a smooth gradient to climb.</li><li><strong>Hard gates</strong> — builds_required, no_critical — <em>zero</em> the reward outright when violated. A config with a hardcoded secret doesn&#39;t get partial credit for being 90% clean. It gets nothing.</li></ul><p>The gates are what make the reward un-gameable, and making them un-gameable is the whole game. The scanner ships <strong>18 base rules</strong> across four severities — 5 critical (public S3 ACL, world-open admin SG, public RDS, hardcoded secret, IAM wildcard), plus high/medium/low rules for encryption, versioning, backups, rotation, tags. Critical failures trip the gate; everything else is soft signal.</p><h3>How the data is built</h3><p>A verifiable reward frees you from labels, but you still need a <em>lot</em> of varied requests, at difficulties that produce reward <em>variance</em> — GRPO learns nothing from a batch where every rollout scores identically. The training pool is a deliberate three-way blend:</p><ol><li><strong>A parametric generator</strong> (rl_gym.iac.tasks) — atomic builders for S3, EC2, RDS, IAM, KMS, DynamoDB, Lambda, VPC, each emitting hundreds of distinct request→resource-set combinations at varied difficulty. This is the bulk of the volume and the source of gradient variance. Different random seeds give disjoint instances, which is how I hold out an in-distribution test set later.</li><li><a href="https://huggingface.co/datasets/autoiac-project/iac-eval"><strong>IaC-Eval</strong></a> — 455 human-written request→Terraform pairs. This is the real, curated benchmark.</li><li><strong>GitHub-mined</strong> requests from terraform-aws-modules, plus <strong>Checkov&#39;s own test corpus</strong> — real-world configs the trainer never samples from, reserved entirely for holdout.</li></ol><p>The reward needs no reference answer, so even “unlabeled” real requests train fine. Critically, <strong>the dev split is a frozen, disjoint 120 from IaC-Eval, and training draws only from the other 335</strong> (more on why that disjointness cost me a painful correction below). The holdout is virgin: mined + Checkov, hash-disjoint from everything in the mix.</p><p><strong>SFT targets are themselves verified.</strong> For the warm-start and for later injection experiments, a big open model <em>drafts</em> candidate configs, the scanner <em>grades</em> them, and I keep only the ones that actually pass. No human wrote a single target; the verifier is the annotator. That closes a loop most SFT pipelines leave open — your imitation data is only as clean as your grader, and here the grader is the same program that computes the reward.</p><h3>The training stack</h3><ul><li><strong>SFT warm-start</strong> — LoRA (r16, α32) in <strong>fp32</strong>. Not a stylistic choice: bf16 SFT <em>NaN-collapses</em> on granite-4.1-8b (loss → nan mid-epoch), and a poisoned merge then crashes the downstream GRPO. fp32 LoRA is the stable path.</li><li><strong>GRPO</strong> — trl + vLLM for fast group sampling, G=4 completions per prompt, <strong>DAPO</strong> knobs (ε_high = 0.28, decoupled clip, token-level loss) with β = 0.04. The soft-×-gate reward is the only signal.</li><li><strong>Base model chosen by bake-off, not vibes</strong> — granite-4.1-8b (0.636) beat Qwen3-8B (0.567) and two Qwen2.5 variants on the real benchmark <em>before</em> any training.</li></ul><p>Everything is bounded and self-terminating on a single H100, checkpoints to S3, and prints its own eval tables. A full train+audit cycle is a few dollars.</p><h3>The results</h3><p>On <strong>IaC-Eval</strong> — the frozen split, fully disjoint from training:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ztxyyKYj0TpV_JkI9wNulg.png" /></figure><pre>IaC-Eval (frozen, disjoint)        holdout (never-sampled: Checkov + GitHub)<br>policy            reward    gate    policy            reward    gate<br>random             0.480   100.0%   base LLM           0.756    84.3%<br>base LLM           0.629    76.7%   SFT (warm-start)   0.781    84.3%<br>SFT (warm-start)   0.679    80.8%   GRPO (RLVR)        0.876    92.8%<br>GRPO (RLVR)        0.694    81.7%   human reference    0.524    75.9%<br>human reference    0.459    85.0%</pre><p>The trained 8B scores about <strong>1.5× the human-written reference configs</strong> on security. Read that twice: on this benchmark, the model writes more secure Terraform than the humans whose configs are the &quot;ground truth.&quot;</p><p>On the holdout it carries even harder — <strong>+0.12 reward and +8.5 points of gate rate over base, and +0.10 over SFT.</strong> It&#39;s the <em>RL</em>, not the imitation, that transfers.</p><p>And the reward-hacking audit came back clean: the security component climbs monotonically (0.842 → 0.881 → 0.913) while output diversity stays flat at 93.3% unique. The model got better by getting broadly better, not by finding one rule to spam.</p><h3>I caught my own leak</h3><p>Here&#39;s the number I&#39;m proudest of, and it&#39;s a correction.</p><p>An earlier run reported <strong>0.737</strong> on IaC-Eval. It looked great. It was also inflated: about two-thirds of the dev prompts had accidentally leaked into the training mix. (Prompt exposure only — the reward never reads reference answers, so label leakage was structurally impossible — but exposure is exposure.)</p><p>So I froze a genuinely disjoint split, re-ran the identical recipe, and measured the truth: <strong>0.694</strong>. The leak was worth <strong>+0.04</strong>, corrected everywhere now.</p><p>The same promotion ratchet caught a second, subtler failure. A plausible-sounding upgrade — rejection-sampling SFT on the model&#39;s own verifier-perfect samples — was silently <em>regressing</em> out-of-distribution (0.579 vs 0.679). It got blocked from promotion automatically.</p><p>I&#39;m keeping both negative results in the README on purpose. A platform whose pitch is &quot;we audit for reward hacking&quot; should be visibly willing to catch <em>itself</em>.</p><h3>Continual learning when the standard changes</h3><p>This is the clean, easy case — the mirror image of the hard KMS blind spot later on. Together they map the boundary of what the flywheel can auto-fix.</p><p>Security standards drift. To simulate a real policy-release event I switched on <strong>five new rules the shipped model was never trained against</strong> — all <em>single-flag</em> checks (multi_az = true, a lifecycle {} block, description = …, ebs_optimized = true, a log-group kms_key_id). One-token fixes the model already emits sometimes.</p><pre>event                                      reward<br>shipped model, pre-drift                    0.682<br>+5 new rules switch on  (the &quot;drift&quot;)       0.667   ← measured drop<br>one 50-step continual update ($1.30)        0.704   ← recovered, under the new rules<br>   …re-checked under the old scanner        0.726   ≥ 0.682 → no forgetting</pre><p>It closes <strong>cleanly</strong> — because the fix is a flag the model already samples, so GRPO has plenty of positive rollouts to reinforce. The KMS policy doesn&#39;t, and that single distinction (does the model <em>sample</em> the fix?) predicts which gaps the loop can auto-close and which need a bigger hammer. That&#39;s the honest operating manual for a self-improving system.</p><h3>Self-repair: distilling the verifier loop into the weights</h3><p>The served system doesn&#39;t just generate. It runs <strong>generate → scan → repair</strong>, and ships whichever pass the verifier scores higher. The repair turn is <em>trained</em>: I scan the model&#39;s own first-pass rollouts, and its real failures + the scanner findings become new GRPO episodes — same reward, no new labels. The model learns to read its own audit.</p><pre>serving mode                          reward     gate<br>single pass                            0.694    81.7%<br>two-pass, repair untrained             0.766    85.0%<br>two-pass, repair trained               0.802    90.0%</pre><p>Trained repair converts 89% of its attempts on the holdout (17/19) versus 71% untrained — one 49-minute, ~$3 run.</p><h3>The flywheel: serving traffic becomes the next model, and a program decides it ships</h3><p>This is the loop the whole platform is built to run. The live demo logs every served request and every repair transcript to S3. An aggregator pools that traffic, a continual update trains from the incumbent, and an <strong>executable promotion gate</strong> evaluates the candidate against the incumbent on the frozen anchors — batched through vLLM in one call, greedy/seeded so the decision is reproducible — and exits `PROMOTE` or `BLOCK`. No human in the decision.</p><p>First real cycle — 26 pilot requests → 38 logged episodes → a 50-step update → the gate, about $4:</p><pre>                    incumbent      candidate      verdict<br>real dev  (n=60)    0.697/83.3%    0.738/88.3%    +0.041<br>holdout   (n=60)    0.859/91.7%    0.906/93.3%    +0.047<br>                                   collapse alarm: none → PROMOTE</pre><p>Serving traffic literally produced a better model, and a program cleared it to ship. A cycle that <em>fails</em> the anchors is auto-blocked and the incumbent stays. &quot;Reliable&quot; is a gate, not a hope.</p><p>One finding surprised me enough to re-examine it three times: the gain lands almost entirely on <strong>pass@1</strong>, not best-of-n. Training on repair transcripts sharpens the <em>single-shot</em> output, so blind resampling has little left to find. That collapses the serving economics — <strong>pass@1 + self-repair costs about 40% of best-of-4 + repair at statistically equal quality</strong> — so it&#39;s the default and best-of-n is just a quality-max fallback. The verifier&#39;s job shifts from <em>selector</em> to <em>instructor</em>.</p><h3>The audit found a bug in my own reward function</h3><p>This is the newest finding, and it&#39;s the one I&#39;d lead a technical talk with.</p><p>The flywheel logs served traffic and <strong>mines it for blind spots</strong>: run the model&#39;s own outputs through Checkov (a second, independent scanner) and look for <em>disagreements</em> — configs my scanner passes that Checkov flags. One disagreement dominated the ranking: <strong>CKV2_AWS_64 — &quot;KMS key Policy is defined.&quot;</strong> My model was cheerfully provisioning aws_kms_key resources with <strong>no key policy at all</strong>, and my scanner had no rule to notice.</p><p>I quantified the blind spot on held-out episodes: the served model attaches a key policy <strong>0/7</strong> of the time on real requests, <strong>1/40</strong> on parametric ones. A genuine, measurable gap the loop surfaced on its own.</p><p>So I added a kms_key_policy rule and started training to close it — and <em>that&#39;s</em> when the audit paid off. Every attempt to teach the correct behavior scored <strong>zero</strong>. The reason: a well-formed KMS key policy contains &quot;Resource&quot;: &quot;*&quot; and &quot;Action&quot;: &quot;kms:*&quot; <strong>scoped to that key</strong> — which is the <em>correct</em> AWS idiom — but my critical iam_wildcard rule was reading those tokens as an unscoped IAM wildcard and tripping the gate. <strong>The fix for the discovered blind spot was self-penalizing because of a false-positive in my own critical rule.</strong></p><p>The fix was to exempt KMS key-policy bodies from the IAM-wildcard check (brace-match the policy blocks out before scanning for wildcards). The proof it worked: SFT-target keep-rate on verified-correct KMS configs went <strong>0% → 94%</strong>, real unscoped IAM wildcards are still caught, and re-scoring the shipped model under the hardened scanner cost essentially nothing (0.694 → 0.690 — the model wasn&#39;t gaming that rule; the rule was just wrong).</p><p><strong>The loop&#39;s own gap-mining caught a bug in the loop&#39;s own reward.</strong></p><h3>The honest part: closing the model is harder than closing the verifier</h3><p>Fixing the verifier was clean. Teaching the 8B to reliably <em>write</em> new key policies was not.</p><p>The diagnosis is precise and it&#39;s a known RL bottleneck: <strong>GRPO can only reinforce a behavior the model already samples.</strong> A key policy is a verbose, multi-line JSON block the base model emits ~2.5% of the time. In a group of G=4, the chance <em>any</em> rollout contains one is ~10% — so 90% of KMS prompts produce a policy-free group with zero variance and zero gradient. The signal to reinforce simply isn&#39;t there.</p><p>So I ran the ladder, and measured every rung on held-out episodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJi3O-JXKjEeOBKfKZ21tQ.png" /></figure><p>The last row is the real finding. One round of STaR (rejection-sample the model&#39;s own scanner-verified successes, SFT on them) <strong>jumped the key-policy rate from 7.5% to 80%</strong> — the behavior is absolutely learnable, and the sampling-rate diagnosis was dead right. But it came at a catastrophic cost: over-distilling on the narrow KMS slice <strong>collapsed the model</strong> — reward fell to 0.22, the no_critical gate dropped from 88% to 42%, some outputs stopped parsing. It bought the rare rule by destroying general competence: a textbook alignment tax, quantified.</p><p>That&#39;s the honest boundary of what a self-improving loop can auto-fix at this scale. It reliably does the valuable half — detect the gap, repair its own verifier. Teaching a rare, verbose behavior into an 8B is closeable <em>in isolation</em> but not without collapsing everything else — and knowing exactly where that wall is, with the numbers, beats a lucky close I couldn&#39;t defend.</p><p><strong>So where does the fix actually belong? The repair turn, not the weights.</strong> A standard KMS key policy is <em>boilerplate</em> — nearly identical every time. Forcing an 8B to memorize it (and collapsing it in the process) is the wrong tool for a templatable fix. The system already runs a verifier-guided repair step, so I gave it a deterministic patch: whenever the scanner flags a policy-less aws_kms_key, the repair layer injects a standard root key policy — idempotent, gate-safe, and it closes the served gap to ~100% for free. The blind spot the weights couldn&#39;t hold, the serving layer closes. Detection, verifier, and repair each do the half they&#39;re good at — which is the whole thesis in one bug: <strong>the verifier is the moat, and the machinery around it is fungible.</strong></p><h3>Two open models, two roles</h3><p><strong>Rule-authoring runs on Nebius Token Factory; the tuned model runs on a Nebius GPU.</strong> The big reasoning model that drafts verifier rules runs per-token on Token Factory (serverless, nothing idling between requests). The tuned specialist, though, is granite-4.1-8b — and Token Factory hosts fine-tunes only on <em>supported</em> bases (LoRA on Llama-3 / Qwen), not granite. So it runs on a <strong>Nebius L40S</strong> GPU instance via vLLM — and the <em>same</em> stack <strong>self-hosts in your VPC</strong> unchanged when the model and infra topology can&#39;t leave the boundary, which is the point for a security customer. Training is separate — a handful of short, bounded Nebius H100 jobs at a few dollars each. No closed model anywhere in the system.</p><p><strong>Role 1 — serve the specialist.</strong> The promoted granite-8B runs on a <strong>Nebius L40S</strong> with vLLM at roughly <strong>$0.006/request</strong> amortized — or self-hosts in your VPC on the same stack when data can&#39;t leave the boundary. (Zero-idle per-token serving on Token Factory needs a supported base — Llama-3.1-8B or Qwen2.5 — trading the ~7 security points granite won at the bake-off. A real quality-vs-serverless choice.) On the first 10 dev episodes:</p><pre>condition                            reward    gate     $/request<br>frontier model, blind                 0.598     70%      ~$0.17<br>frontier model + verifier rulebook    1.000    100%      ~$0.16<br>tuned 8B, pass@1 + self-repair        0.865      —       ~$0.006</pre><p>Blind, the frontier model ties the <em>base</em> 8B and zeroes 3/10 on critical gates. Hand it a ~580-token distillation of the verifier&#39;s rules and it saturates the benchmark. Both halves are the thesis: <strong>the verifier is the learning signal</strong> — for frontier agents as much as for GRPO — and once the judge saturates, judge <em>coverage</em>, not the generator, is the moat. The economics stay 12–33× apart per request.</p><p>That <strong>$0.006 is engineered, not luck.</strong> The 8B serves under vLLM with <strong>fp8 weights</strong> (the L40S is bandwidth-bound — fp8 halves weight traffic, and Ada has native fp8 kernels), <strong>prompt-lookup speculative decoding</strong> (Terraform is highly templated, so drafted tokens verify cheaply and <em>losslessly</em> — a domain-specific free lunch), and <strong>CUDA graphs</strong> — the engine boots the fastest config the hardware accepts and degrades one knob at a time if it balks. Layer on pass@1 + self-repair instead of blind best-of-<em>n</em> (matched quality at ~40% of the generation cost), and the per-request number stays low without spending quality to get there.</p><p><strong>Role 2 — grow the verifier with a big model, safely.</strong> New rules aren&#39;t guessed from a wishlist — they&#39;re <em>mined</em>: served logs cross-checked against external scanners (Checkov and friends) surface the disagreements, and each disagreement is a candidate rule. From that gap, a big open <em>reasoning</em> model (DeepSeek-V4-Pro — pennies per-token) <em>reasons out</em> the scanner predicate — its chain-of-thought is exactly what you want writing a security rule. Then two gates make it trustworthy before a human ever merges it:</p><ol><li><strong>AST sandbox</strong> — the drafted code may use only re and string ops. Imports, open/eval, dunders are rejected at compile time. Model-written code never gets real capabilities.</li><li><strong>Executable validation</strong> — the rule must classify every pass/fail example correctly, or it&#39;s rejected. <em>This is exactly why a big open model is safe here: the tests catch its mistakes.</em></li></ol><p>I demonstrated it end to end — an open model drafted a rds_deletion_protection rule, it passed all four examples, and it&#39;s staged behind a flag pending human sign-off.</p><p>The principle, one level up from the model itself: <strong>the artifact is verifiable, so the generator is fungible.</strong> A big LLM writes verifiers. It never gets to <em>be</em> one.</p><h3>Wins, problems, solutions</h3><p>Because the interesting part of a systems project is the failures you survived:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sd9fZiQnUuApT22xPqMesw.png" /></figure><p>Every one of these is in the repo with its eval output. The pitch isn&#39;t &quot;nothing broke&quot; — it&#39;s &quot;everything is measured, and the gates caught what broke.&quot;</p><h3>Demo</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_IsADe8h4_Q%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_IsADe8h4_Q&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_IsADe8h4_Q%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d2198b1bbb52a37731a9d5d72a7913a/href">https://medium.com/media/5d2198b1bbb52a37731a9d5d72a7913a/href</a></iframe><p><strong>Code (AGPL-3.0):</strong> <a href="https://github.com/SkyForce/rl-gym">github.com/SkyForce/rl-gym</a> — reproduce it on Nebius (instructions in the README). Commercial licenses available.</p><p><strong>#NebiusServerlessChallenge</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c859f37db18" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>