<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Never a Paragraph on Medium]]></title>
        <description><![CDATA[Stories by Never a Paragraph on Medium]]></description>
        <link>https://medium.com/@eddie.hc.tsui?source=rss-95401a211927------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VSK0YATmUkuUoNXggCTiig.png</url>
            <title>Stories by Never a Paragraph on Medium</title>
            <link>https://medium.com/@eddie.hc.tsui?source=rss-95401a211927------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:19:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eddie.hc.tsui/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One support ticket went into Jev and came back as three probabilities.]]></title>
            <link>https://medium.com/@eddie.hc.tsui/the-score-is-an-average-jev-does-not-write-a-label-f8a1dd13aaa2?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8a1dd13aaa2</guid>
            <category><![CDATA[je]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[decision-making]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Thu, 24 Sep 2026 08:54:10 GMT</pubDate>
            <atom:updated>2026-09-24T14:14:36.316Z</atom:updated>
            <content:encoded><![CDATA[<h3>One support ticket went into Jev and came back as three probabilities. The urgency score is each level times its weight, added up. Here is that arithmetic, and the part TypeSafe has left unpublished.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AwVkCzHKcUU2OWG2" /></figure><p>One ticket. Three distributions. The urgency score is a weighted mean.</p><p>A blank checkout page. An enterprise customer. Two browsers. Yesterday I sent that sentence through a word list, a chat model asked to return JSON, and Jev, and I wrote down what each one gave back. The interesting object was the Jev answer. It arrived in under a second: bug probability 0.96, payments as the owner, urgency 1.99.</p><p>I treated that answer as a black box with good manners. This piece opens the box as far as the documentation and that answer allow.</p><p>The argument in one line: the decision you can audit is small. A closed set of options, a probability on each, a score that is their weighted average, and an if-statement in your repository. The neural net, the reward, and the confidence formula stay inside TypeSafe.</p><p>Jev 1.13 is a System One model from TypeSafe, on OpenRouter as <a href="https://www.google.com/url?q=https://openrouter.ai/typesafe/jev-1.13&amp;sa=D&amp;source=editors&amp;ust=1790243512598404&amp;usg=AOvVaw3XdpLHdi5SyE8BCJW9Fvww">typesafe/jev-1.13</a> . Diogo Almeida, who co-invented the reinforcement-learning method behind InstructGPT, launched it on 15 September 2026. The name System One comes from Daniel Kahneman’s fast judgment. The name Jev comes from William Stanley Jevons, because TypeSafe expects cheaper decisions to create more of them, the way a more efficient steam engine increased the demand for coal.</p><p>The call is a function. State in. Typed probabilities out. You post to `https://openrouter.ai/api/alpha/decisions`. You pay for input tokens, listed that week at $0.042 per million. Output tokens are free. Context is 32,000 tokens for the state and the questions together. The six tickets I measured used about 470 to 490 of those tokens and came back in well under a second of wall clock.</p><p>1. Ten boxes, three of them in the call</p><p>A poster of “the mathematics behind AI” usually has ten boxes. Probability. Linear algebra. Similarity. Statistics. Gradients. Bayesian updating. Combinatorics. Optimization. Decision theory. Sometimes quantum optimization, because the poster had a spare color.</p><p>People ask which box Jev lives in. The honest map is shorter, and it is the one I can defend from TypeSafe’s own pages plus the answers I actually received on 23 September 2026.</p><p>Probability is the product. Every answer is a distribution, or a single probability.</p><p>Statistics is the training promise. Calibration is a property of many answers together. A stated 0.8 should be right about four times in five, across a pile of cases. It is a promise about the pile.</p><p>Decision theory sits in your code. The model stops at the distribution. The cost of paging the wrong person, or of missing a blank checkout, is a threshold you write.</p><p>Optimization is the name of the training target. TypeSafe calls it Reinforcement Learning for Calibrated Decisions, RLCD. The objective is published. The reward function is not.</p><p>The other boxes are real mathematics, and they are the wrong story for this call. Cosine similarity is how an embedding search ranks neighbors. Gradients are how a network is trained, and this API does not return them. A Bayesian update needs a prior and a likelihood you can write down. Jev’s docs give you neither, so a noul is a probability you can threshold, and it is not a posterior you can update by hand. Quantum optimization does not appear in the launch post, the primitives, or the confidence page. I am leaving it off the diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qJ1CjLH2y92_F8Cd" /></figure><p>Probability, a calibration promise, a named training target, and a threshold you own.</p><p>2. A chat model writes the next word</p><p>A language model answers by sampling the next token, then the one after that, each one conditioned on the tokens already written. That is why a JSON classifier is slow in a way that has little to do with how hard the ticket is. On my six calls, Llama 3.3 70B at temperature 0 took 1.4 seconds on the dark-mode request and 9.2 seconds on the double charge. The double charge is not six times harder. Generation has a length, and the length wandered.</p><p>Jev gives up that loop. Almeida’s launch post describes a parallel sampler: all of the outputs for a query are produced together, in hardware that is built for that shape. There is no sentence growing on the screen. There is no place for the model to talk itself from “this might be a feature” into “and therefore it is urgent.”</p><p>The practical consequence is the one I care about in a queue. Questions in one request are scored independently, against the same state, and they cannot see each other’s answers. Urgency does not get a peek at the bug label. TypeSafe’s cookbook on parallel questions measured a batch of thirteen judgments as about ten times faster and about twelve times cheaper than thirteen separate calls, with the answers unchanged. Our triage used three questions. They shared one round trip.</p><p>Speed on the provider page is a P50 of 0.28 seconds at TypeSafe. My six calls, network included, landed between 0.49 and 0.88 seconds. That is the number a product person should plan around, because the user waits for the round trip.</p><p>There is one published exception to “one shot, every option.” A Choice accepts up to 255 options. In the Wikipedia-racing demo, when a page had more links than that, TypeSafe described a two-stage procedure: score the candidates independently, then make an explicit choice among the survivors. Our desk has three teams. We never entered that second stage. I mention it because it is one of the few places the company describes a procedure, and because a routing problem with a thousand labels is a different program from a routing problem with three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U05s3wFuMyRkasgx" /></figure><p>The slow part of a JSON classifier is the sentence. Jev does not write one.</p><p>3. The request is the hypothesis</p><p>Before any probability exists, you write down the set of answers you are willing to receive. That set is the algorithm’s first line, and it is yours.</p><p>The state is what the model may see. On this desk it was two fields: customer tier, and the ticket text. A bare string works. An object is better, because a forgotten field stays a forgotten field, and a question can point at `ticket` or `customer_tier` by name. Question IDs are for your code. TypeSafe is explicit that the model does not see them. The words that matter go in `instructions` and in `criteria`.</p><p>Three primitives, and only three.</p><p>A noul asks whether a statement holds. The answer is one number, from 0 to 1, the probability of yes. Near 0.5 means yes and no are about equally likely. It means the bug is “medium” only if you have confused this primitive with a score. There is no second field called confidence, because the number already is the confidence.</p><p>A choice asks which one of a list you wrote. The answer is the winning key, a probability for every key, and a confidence. The probabilities sum to 1. The winning key is the option with the highest probability. A fourth team cannot appear, because a fourth key was never offered. Cardinality stops at 255. If the list might be incomplete, you add `other` yourself.</p><p>A score asks where the state sits on an ordered scale you wrote, from 2 to 10 levels. The answer is a probability per level, a legend that repeats your words, a confidence, and a score. The score can fall between levels.</p><p>I keep the policy out of the request. Jev returns numbers. The branch that pages a human lives in the repository, next to the cost of a mistake. A vendor default would be someone else’s cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PBUKRGDYuVad0-v-" /></figure><p>The model can only land inside the sets you declared.</p><p>4. One ticket, three distributions</p><p>Here is the blank checkout, the public example in OpenRouter’s Jev tutorial, which I kept so the numbers can sit next to a sample anyone can rerun.</p><p>State. Tier: enterprise. Ticket: “My checkout page shows a blank screen after I click Pay. I have tried two browsers.”</p><p>Three questions, sent together.</p><p>Is the customer reporting a software defect? Criteria for yes: broken or unexpected product behavior. Criteria for no: a question, or a feature request.</p><p>Which team owns it? Account, for login, permissions, or profile. Frontend, for rendering, layout, or browser behavior. Payments, for checkout, billing, or payment processing.</p><p>How urgent is it? Level 0, can wait for the next release. Level 1, should be fixed this week. Level 2, blocking revenue right now.</p><p>The response, model snapshot `typesafe/jev-1.13–20260917`, 590 milliseconds, 476 input tokens, about two hundredths of a cent:</p><p>Choice. Payments 0.80, frontend 0.20, account 0. Confidence 0.70. The selected key is payments, because 0.80 is the largest mass.</p><p>Score. Level probabilities 0, 0.01, and 0.99. Score 1.99. Confidence 0.99. The legend comes back with the number, so “blocking revenue right now” stays attached to index 2 in the record.</p><p>Read them as three separate instruments. The noul says the defect call is extreme. The choice says most of the team-mass sits on payments, with a real remainder on frontend, which is what “blank screen” and “two browsers” are doing in the sentence. The score says almost all of the urgency-mass sits on the top level. They were not allowed to consult each other. A chat model often makes urgency agree with the bug label because both labels are tokens in one growing paragraph. Here the agreement is a property of the ticket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Onx_7XMDYeXmihyR" /></figure><p>Three instruments. One round trip. None of them wrote a reason.</p><p>5. The score is a weighted mean</p><p>This is the line in the docs I wish I had read before I stared at 1.99.</p><p>TypeSafe’s Score page says the score is each level number multiplied by its probability, added up. On their own worked example, probabilities 0, 0.70, and 0.30 become 0×0 + 1×0.70 + 2×0.30 = 1.30. The level numbers are positions in the array you sent, starting at 0. The model is given the descriptions. It is not given the integers. “Worse than the previous level” is meaningless to it, because it does not see the neighbors as neighbors. Each level is judged on its own, and the average is assembled afterward.</p><p>Apply that to the checkout.</p><p>0 × 0 + 1 × 0.01 + 2 × 0.99 = 1.99.</p><p>The 1.99 is that sum. It is the expected position on a line you defined. It is a useful summary when the scale is ordered and the words on the levels are situations, which is why “blocking revenue right now” beats “very urgent.” A vague degree gives the model nothing to match.</p><p>The same arithmetic closes on the dead login from the same run. Level probabilities 0.01, 0.29, and 0.70.</p><p>0 × 0.01 + 1 × 0.29 + 2 × 0.70 = 1.69.</p><p>And on the invoice-id leak. Probabilities 0.01, 0.57, and 0.42.</p><p>0 × 0.01 + 1 × 0.57 + 2 × 0.42 = 1.41.</p><p>On the double charge, the bars I published rounded to 0, 0.32, and 0.68, which multiply out to 1.68, and the returned score was 1.67. That gap is rounding of the bars, not a second hidden formula. I would store the raw probabilities beside the score so a later reader can recompute the mean.</p><p>Two warnings sit on the same page, and both showed up in this set.</p><p>The score does not measure a mixture in the world. A 1.30 on TypeSafe’s export-button example means “mostly level 1, some weight on level 2.” It does not mean 30 percent of customers lack a workaround. My 1.99 does not mean the checkout is 99.5 percent blocked. It means the probability mass is sitting on the top sentence of my scale.</p><p>Different distributions can share a score. All of the mass on level 1 is a score of 1. Half the mass on level 0 and half on level 2 is also a score of 1. The first is a clear “this week.” The second is a tug-of-war between “next release” and “paging someone.” The score erases that difference. The probabilities and the confidence put it back. I did not have a perfect tie in the six tickets. I had the shape of one: the leak’s urgency is 0.57 on level 1 and 0.42 on level 2, score 1.41, confidence 0.36. A rounded score of 1 would have called that “this week” and hidden a large chance that the top level is the right one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o8FqJOpiupksqvnh" /></figure><p>The formula is on TypeSafe’s Score page. The inputs are the probabilities from the call.</p><p>6. The winner is the top of the list</p><p>A choice is easier to misread, because the winning key looks like a label and the confidence looks like a second opinion.</p><p>The rule in the docs is short. `choice` is the option with the highest probability. `probabilities` is the full distribution, and the values sum to 1. `confidence` is a number from 0 to 1 computed from how that distribution is spread. A single peak is high confidence. A flat shape is low confidence. TypeSafe computes the summary so you can threshold without doing the reduction yourself, and then says you are not locked to their reduction, which is why the full distribution is in the payload. The cookbook that would compare reductions is promised and, as of these docs, not linked. I am not going to invent the formula and call it theirs.</p><p>The dead login is the clean demonstration. “The login button does nothing when clicked.”</p><p>Noul 0.97. Team probabilities: account 0.59, frontend 0.41, payments 0. Confidence 0.38. Score 1.69, confidence 0.53.</p><p>The selected team is account, because 0.59 is larger than 0.41. The confidence is 0.38, which is smaller than the winning probability. If you threshold on 0.59 you will auto-route. If you threshold on confidence at 0.70, as I would, you will not. Both numbers are in the same answer. They answer different questions. The probability says how the mass was split. The confidence says how peaked that split is. On a two-way split this close, a peak statistic should be modest, and 0.38 is modest.</p><p>The checkout shows the same split of meanings with more room. Payments at 0.80, confidence at 0.70. I can act on the key and still refuse to treat 0.80 as a certificate. The docs are plain that confidence describes the answer’s shape. A confidence of 1.0, in their severity table, means the distribution put all of its mass on one level. It describes the answer. It is not a guarantee the answer is right.</p><p>T5 is the uncomfortable case, and I am keeping it. “Something feels off with the app lately.” Noul around 0.55, which is an abstain on the defect. Frontend at 0.99, confidence 0.99. The team distribution is a spike. The text does not earn a spike. A closed output space prevents an invented fourth team. It does not prevent a confident team call on a thin sentence. The spike is visible, which is the property I want in a review, and it is still a spike.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eHp7llxEFIzZA37h" /></figure><p>Argmax picks the key. Confidence summarizes how alone that key is.</p><p>7. Questions that cannot coach each other</p><p>Independence is the property I would test in a design review, because it is the one a prompt cannot promise.</p><p>Every question in the request sees the same state. None of them receives another question’s answer as hidden context. You can add a question or remove one. TypeSafe says the others stay put, beyond ordinary run-to-run sampling noise. Their parallel-questions cookbook is the empirical version of that claim: thirteen judgments in one call, answers unchanged versus asking them apart.</p><p>That is why the dark-mode ticket can be a feature and a frontend item and a backlog item at once, without the feature call softening the team call or the team call inflating urgency. Noul 0.08. Frontend at 1.00. Score 0.07, from 0.94 on level 0 and 0.06 on level 1. The word “broken” is in the sentence. The noul still sits near zero, because the criteria for no include a feature request, and the customer said this is not an error. The score did not look at the noul and decide to agree. Both instruments happened to agree.</p><p>When a later judgment truly depends on an earlier one, the docs tell you to make a second request. The dependency is real when your code cannot build the next question until it has the first answer: it needs that answer to fetch more state, to decide what the state contains, or to choose the next option list. Hierarchical classification works this way, a choice at each level of a tree, with a beam of probable branches kept alive when the mass is split. Structure recovery works this way too: first a noul per line break, then a choice on blocks that did not exist until the first call returned.</p><p>If you can ask both questions against the original ticket, you ask them together and you combine them in code. The second round trip is the exception.</p><p>Speculative fan-out is the pattern I would actually use on a desk. Ask the return-reason question even when you are not sure the ticket is a return. Ask the severity question even when the noul might come back low. Ignore the answers your code does not need. Extra questions cost tokens, and on this price list the tokens are the cheap part. The TypeSafe choice example does this with five questions on one shoe-store ticket, and the sample code uses `return_reason` only inside the returns branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qxUzdjRGoKFnpjd-" /></figure><p>If urgency must depend on the bug call, write that dependence where a person can see it.</p><p>8. What the training is for</p><p>The sampler explains the shape of a response. It does not explain why a 0.80 should mean anything.</p><p>TypeSafe’s comparison, in the launch post, is three reinforcement-learning targets.</p><p>RLHF rewards answers human raters prefer. It is the method behind InstructGPT and ChatGPT, and Almeida co-invented it. The known cost, which TypeSafe’s own primer names, is that raters like confidence. A fluent certain answer scores well. Sycophancy and mode dropping come with that target. An output can be compelling to a person and still be a bad thing to automate.</p><p>RLVR rewards answers a program can check. Tests pass, the arithmetic matches, the proof verifies. It has produced strong reasoning models on tasks where correctness is mechanical. It says nothing about how sure the model should have been on the way there.</p><p>RLCD, Reinforcement Learning for Calibrated Decisions, is the target TypeSafe names for Jev. The docs’ one-line version: train the model to return decisions and calibrated probabilities, in place of generated text. The launch post’s version: epistemically honest probabilities on System One tasks. Across many predictions, outcomes assigned 0.2 should happen about a fifth of the time, outcomes assigned 0.8 about four fifths, outcomes assigned 1.0 essentially always. Those rates describe groups of predictions.</p><p>TechCrunch, on 18 September 2026, reported Almeida saying Jev is trained entirely on synthetic data, and quoted the bet: they would make all of their data. That is a data-source claim. It is not a recipe.</p><p>What is missing is everything required to rerun the training. No reward function. No proper scoring rule written down. No dataset, no step count, no calibration plot for a task family, no paper. An arXiv search in the week of the launch for the phrase “reinforcement learning for calibrated decisions” came back empty. I am writing that down because a Medium post that invents a loss function is how this subject usually goes wrong. The objective is public. The method is a name.</p><p>The consequence that does show up in the API is enough to build on. The distribution is the primary output. The chosen key is derived from it, by taking the maximum. Confidence is computed from the distribution after the fact. Nobody in the request asked the model “how sure are you,” which is the question that invites a language model to perform certainty. A noul carries no extra confidence field, because a second number would repeat the first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Il8auldyFhdbny-Z" /></figure><p>RLCD is a published objective. It is not a published training run.</p><p>9. Calibration belongs to a pile</p><p>I want to be dull about this, because the dull version is the one that survives a review.</p><p>A single ticket at 0.96 is not “calibrated.” Calibration is a comparison, over many tickets, between the probability you were given and the fraction that turned out to be defects. Take every answer that came back near 0.8. If about 80 percent of them were actually yes, the model is calibrated in that bin. If 60 percent were yes, it is overconfident, and a threshold at 0.8 will automate mistakes. If 95 percent were yes, it is underconfident, and your queue will fill with cases the model could have cleared.</p><p>The usual picture is a reliability diagram. Bucket the probabilities. Plot the average stated probability in each bucket against the hit rate. A calibrated model sits near the diagonal. Expected calibration error is the average gap. The paper people cite is Naeini, Cooper, and Hauskrecht, 2015. I have not drawn that diagram for Jev, because I do not have labels. Six tickets are a reading, not a bin.</p><p>Accuracy and calibration come apart. A classifier that is right 90 percent of the time and says 0.99 on every ticket is accurate and useless for gating. A weaker model that says 0.70 when it is right about 70 percent of the time is the one you can build a threshold on. TypeSafe has published no accuracy figure and no calibration figure that I can drop into this paragraph. Their workflow evals compare Jev with other models against a reference built from larger models’ probabilities, on workflows their own team wrote. That is a relative study with the caveats they print next to it. It is not a reliability diagram on support tickets.</p><p>Two further limits, both from TypeSafe’s jaggedness notes on 1.13, belong next to any threshold.</p><p>There is no guaranteed relationship between related questions. A noul and a noul of its negation need not sum to 1. Their worked example sums to 1.19, and the page says there are many reasons the two numbers may not be comparable. Per-answer calibration does not give you logical consistency inside one call. If your code needs `P(yes) + P(no) = 1`, ask one question.</p><p>And the model is jagged in ordinary ways. Weak at arithmetic and counting. Dates read as text. Quality drops as unrelated content piles into the state. Input is not treated as hostile by default. A triage desk that pastes a whole thread, plus a policy, plus a customer biography, is a different problem from the two-field state I sent.</p><p>The way I would check this on our own desk is ordinary. Log the noul, the distribution, the snapshot id, and the cost. Later, log what a person decided. Bin by probability, per question, because a bug-noul can be honest while an urgency score is not. Move the thresholds when the bins disagree with the cuts. Until that table exists, the cuts I published yesterday are a policy, and they are not a measurement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q1LGCimXmc3fhezf" /></figure><p>Log the probability now. The hit rate arrives when a person closes the ticket.</p><p>10. The if-statement is the rest of the algorithm</p><p>Once the distributions exist, the remaining algorithm is decision theory of the inexpensive kind. You write down the cost of acting, and you refuse to act when the distribution is too spread out to pay that cost.</p><p>TypeSafe’s confidence page offers three bands as a starting picture. High, act. Medium, confirm or gather more. Low, send it to a person. The numbers move with the stakes. Their sample code uses 0.5 as a floor under which it will not guess, and 0.9 before it will approve a transfer. Showing the wrong account screen is recoverable. Moving money is not. The correct cuts, they say, depend on your domain, and you start conservative.</p><p>My cuts, repeated so this article can stand beside the last one, are a policy I would start from. They are not a TypeSafe default, and they are not tuned.</p><p>On the bug noul, at or above 0.85 is a defect, at or below 0.15 is not, and the band between them belongs to a person. On the team choice, auto-route only when confidence is at least 0.70. On urgency, page only when the score is at least 1.7 and confidence is at least 0.60. If the bug question abstained, do not auto-close and do not page. One confident field does not authorize the whole workflow.</p><p>Walk the checkout through those cuts, because it is the only ticket in the set that clears all three. Noul 0.96 clears 0.85. Team confidence 0.70 clears the routing bar on the nose. Score 1.99 with confidence 0.99 clears the page. Payments on-call gets it. The arithmetic of the score is why 1.99 clears and 1.41 does not. Almost all the mass is on level 2, so the mean sits on level 2, and the shape is a spike, so confidence is 0.99.</p><p>Walk the double charge, because it is the ticket the mean is for. Noul 0.45, inside the abstain band, so a person owns the defect label. Payments at confidence 1.00, so the queue can be chosen. Score 1.67 with confidence 0.51, so nobody is paged. Billing sees a refund conversation. Engineering is not woken up to adjudicate an authorization hold that might be a duplicate capture. A boolean rule, looking for “charged twice” and “today,” could not take the middle action. A chat model at 0.90 on “bug” erased the middle.</p><p>The leak is where I defend the model against my own scale. Noul 0.95, so it is a defect. Account at 0.81, confidence 0.71, so it routes, and the UI should show payments at 0.18 as the runner-up. Score 1.41, confidence 0.36, so the policy does not page. An access-control failure that exposes another customer’s invoice is urgent in a sense my scale never named. The low confidence is the score following the question I asked, which was about revenue. The fix is a new question whose top level is “other customers’ data is exposed now.” Lowering the revenue bar until a privacy incident looks like lost revenue is how severity scales get destroyed.</p><p>Composite scoring is the grown-up version of the same idea, and it stays in code on purpose. If priority depends on severity, frustration, and how much the report gives an engineer to work with, you ask three scores and you weight them. When the ranking feels wrong, you change a weight. You do not rewrite a paragraph and hope the next sample agrees. The weights are the decision. The model supplied the inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*go3mPkBb_HtZ5y8z" /></figure><p>The checkout clears every bar. The double charge clears only the queue.</p><p>11. What I am not going to invent</p><p>A post about “how the algorithm works” is under pressure to draw a network. I do not have one, and drawing a plausible one would be worse than the gap.</p><p>Unpublished, as of the docs and the launch post I used: the architecture, the parallel sampler’s implementation, the reward function, the synthetic-data generator, any calibration chart, and the exact reduction from a distribution to the confidence number. TypeSafe says confidence is a statistic of the shape, that flatter means lower, and that you may compute a different statistic from the probabilities they return. That is a complete interface. It is an incomplete paper.</p><p>Also unpublished, because it is not how this API behaves: a reasoning trace. The OpenRouter guide is direct. Jev does not return the path it took. If a reviewer needs a sentence, a person writes it, or a chat model writes it after the decision, with the typed answer handed over as a fact. The explanation does not get a vote. I have watched a wrong label arrive with a fluent defense, and the defense was produced by the same system that picked the label. Separating them is the point of giving up the paragraph.</p><p>A few ordinary caveats belong in the same box, because they change how you ship.</p><p>The sampler is not bit-identical. On the vague ticket, a second call moved the noul between 0.54 and 0.55, and the score between roughly 0.28 and 0.31. The team stayed a frontend spike. OpenRouter’s tutorial says the probabilities differ slightly from run to run. The operational decision, inside a wide abstain band, stayed put. I will not promise replay down to the digit. Exact replay belongs to the word list, on the decisions that really are a token: a SKU, a country code, a blocked domain.</p><p>Pin the snapshot. Every response I received named `typesafe/jev-1.13–20260917`. The alias `~typesafe/jev-latest` tracks the family. A threshold tuned in September should call the dated model until you retune. Store the response id, the snapshot, the state, the questions, and `usage.cost` with the ticket. That record is the audit. The prose you did not request is not.</p><p>And keep the rule layer where the rule is the policy. Jev is a poor arithmetic engine, by TypeSafe’s own jaggedness note. Do not ask it whether the refund is under fifty dollars. Ask your database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ViB0oY2K1bqf5vHM" /></figure><p>The interface is enough to branch on. It is not enough to retrain.</p><p>12. The whole path, on one ticket</p><p>Put the pieces in the order a request actually runs. This is the algorithm as an integrator sees it. It is deliberately smaller than a research talk.</p><p>You build a state object. Tier and ticket, nothing else, unless a question truly needs the policy text or the charge records. You declare the questions, each with a type, an instruction, and criteria written as situations. You send one POST. The provider routes to TypeSafe. The parallel sampler produces a distribution for each question against that state, without letting the questions read one another. The API reduces each distribution in a documented way: a noul is already a probability; a choice key is the argmax; a score is the probability-weighted mean of the level indices; confidence, on choice and score, is their summary of the shape. The response carries the snapshot name, an id, and a cost. Your code applies cuts you can defend in a design review. A person owns the band between the cuts. A chat model, if you still want one, drafts the reply after the branch has been taken.</p><p>On the checkout, that path is concrete.</p><p>State in. Three questions in. Noul 0.96, so the defect label is automatic under a cut at 0.85. Payments 0.80 with confidence 0.70, so the queue is payments, and the UI can show frontend at 0.20 as the runner-up a tired on-call might still want to see. Score = 0×0 + 1×0.01 + 2×0.99 = 1.99, confidence 0.99, so the page fires. Cost, about $0.00002. Latency, 590 milliseconds from my machine. No JSON to repair. No fourth team to reject. No sentence to trust.</p><p>On the double charge the same path refuses to finish. The noul is 0.45, the mean is 1.67, the confidence on that mean is 0.51, and the function returns “person” for the label and for the page, and “payments” for the queue. That refusal is the product. A decision model that cannot land in the middle is a classifier with a costume on, which is the system we had before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IzeBvA6rN-WzAohF" /></figure><p>The model’s last step is a distribution. The product’s last step is a branch.</p><p>13. What I would tell the next reader</p><p>If you came from a poster of ten mathematical fields, take three ideas home and leave the rest for the tools they actually belong to.</p><p>The number on a score is an average you can recompute. If your stored probabilities do not multiply back to the stored score, something in the pipeline rounded too early or edited the legend. That check is free, and it catches a class of bugs that a demo screenshot will not.</p><p>The winning label is the top of a list that sums to 1, and the confidence is a second reading of the same list. Threshold them separately. A dead login can be an obvious defect and an ambiguous owner. A vague complaint can be an obvious abstain and a strangely confident team. The fields are allowed to disagree. Your policy should say which one spends the money.</p><p>The training story is a target, calibration, pursued with a method the vendor has named and not specified, on synthetic data they say they built. You can measure the target on your own tickets. You cannot audit the method from the outside. Ship the measurement. Keep the word list for the decisions that are lookups. Keep a person on the band. Let a writing model write the email after the branch, with the probabilities already in the record, so the email cannot renegotiate them.</p><p>Jevons gets the last word, because TypeSafe put him in the name. Make a decision cheap enough, and fast enough, and typed enough that a program can call it a million times, and the organization will find a million decisions it used to leave implicit. The checkout page is one of them. The arithmetic is short. The discipline is storing the distribution, recomputing the mean, and letting the middle of the range belong to a person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5ORu718LRjxuIQrJ" /></figure><p>The algorithm you can keep is the one you can recalculate on a napkin.</p><p>#ArtificialIntelligence #MachineLearning #DecisionScience #DataScience #LLMs #AIEngineering #Probability #Calibration #ModelEvaluation #AIProduct #SoftwareEngineering #MLOps #AppliedAI #Classification #SupportOperations #Automation #AIInfrastructure #OpenRouter #TypeSafe #SystemDesign #ProductManagement #TechWriting #Statistics #AIWorkflows #FutureOfWork</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8a1dd13aaa2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Governing LLMs and Jev : When AI Starts Making Decisions]]></title>
            <link>https://medium.com/@eddie.hc.tsui/ai-governance-after-generative-ai-governance-decision-intelligence-18bcd2d7cac4?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/18bcd2d7cac4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[jevs]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 14:37:11 GMT</pubDate>
            <atom:updated>2026-09-25T00:49:02.154Z</atom:updated>
            <content:encoded><![CDATA[<p>Beyond the Generative:</p><p>Governing LLMs and Jev : When AI Starts Making Decisions</p><p>A practical framework for separating prediction, judgment, control, explanation and authority</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CTQBQxuXr4os0VCi" /></figure><p>Figure 1. Two different AI brains belong in one governed system: expressive language generation and bounded probabilistic judgment.</p><p>For the past several years, the public conversation about artificial intelligence has focused on what models say. Can a chatbot invent a fact? Can a generated answer disclose private information? Can a prompt manipulate the system? These are important questions, but they belong to the era in which AI mainly produced content for a person to read.</p><p>The next phase is different. AI is moving inside software, where a model may classify a case, score its urgency, route work, select a tool, recommend an action or decide whether enough evidence exists to proceed. At that point, the central governance question changes from “Is the answer fluent?” to “What authority did this output receive, and what prevented a bad judgment from becoming a bad action?”</p><p>This change becomes especially visible when we compare large language models with Jev, TypeSafe AI’s first “System One” model. An LLM generates language. Jev evaluates text-based state and returns typed answers and probabilities through three documented primitives: Choice, Score and Noul, a yes-or-no probability. TypeSafe describes these outputs as structured decisions software can inspect directly. It also states that calibration is measured across groups of predictions and does not guarantee that an individual answer is correct.</p><p>The attraction of typed decision models is easy to understand. Traditional application code handles facts it can calculate exactly: an order exceeds a limit, a date has passed, or a balance is insufficient. Real workflows also contain “fuzzy if” questions: Is this evidence sufficient? Which queue owns this case? Is this exception urgent? Should the system ask for review?</p><p>A general LLM can answer those questions in prose or structured output. Jev is designed for the narrower task itself. The application defines the permitted answer space, supplies the state and asks typed questions. The model returns bounded values and probabilities rather than an essay. This can reduce parsing ambiguity and make a judgment easier for software to consume. But a bounded answer is not automatically a correct, fair or safe answer. A schema prevents the wrong shape of output; it does not prevent the wrong judgment.</p><p>That distinction is the foundation of modern AI governance. We should stop treating “valid JSON,” “high confidence” or “no hallucinated prose” as evidence that a system deserves operational authority. Those properties are useful, but they answer only early technical questions. Governance must continue through reliability, robustness, economic value, execution and real-world outcomes.</p><p>An expressive LLM and a typed decision model should not be governed as if they were the same product. They may consume similar text, yet their outputs create different risks.</p><p>OWASP’s 2025 list for LLM applications highlights risks including prompt injection, sensitive information disclosure, supply-chain problems, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. These still matter in a hybrid system. A typed decision layer can also be influenced by the state it receives, the wording and order of choices, missing context or adversarial framing. Therefore, constrained output should complement deterministic controls, not replace them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JWgmi54gI1TZjJ4A" /></figure><p>Figure 2. The hybrid decision architecture. Each component performs one job, and the learning loop returns outcome evidence to every layer.</p><p>The most defensible design is not “LLM versus Jev.” It is a separation of responsibilities. Think of an enterprise decision as a relay race. Each runner carries a different form of evidence, and no runner is allowed to quietly change the track.</p><h3>1. Predict: data science estimates the world</h3><p>Statistical and machine-learning models estimate demand, risk, arrival time, propensity, anomaly or uncertainty. They answer what is likely to happen. Their governance includes target validity, leakage control, out-of-sample testing, calibration, segment performance and drift.</p><h3>2. Judge: Jev interprets ambiguous state</h3><p>A typed judgment model can classify a case, score urgency or estimate whether a proposition is true. It answers what kind of situation the system is facing. Its governance includes the definition of the answer space, probability calibration, order and framing sensitivity, abstention rules and the cost of false approvals versus false rejections.</p><h3>3. Control: deterministic software governs consequences</h3><p>Rules, optimisation, permissions and exact calculations determine what may happen. Money, inventory conservation, legal thresholds, safety limits and access rights belong here. Where a constraint is non-negotiable, the model should not be asked to remember it; code should enforce it.</p><h3>4. Explain: an LLM makes evidence usable</h3><p>A grounded LLM can explain why a case was routed, summarise the evidence, compare alternatives and prepare a reviewer’s decision pack. It should cite governed evidence and expose caveats. It should not silently recompute official numbers or rewrite the policy boundary.</p><h3>5. Authorize: people and policy own material decisions</h3><p>Human oversight is not a decorative approval button. A reviewer needs time, information, independence, authority and a meaningful consequence for intervening. High-impact, novel or poorly calibrated cases should remain inside an accountable human or policy gate.</p><p>Imagine a customer says: “I was charged twice and need the extra charge refunded today.” A poorly governed system sends the entire message to one LLM and asks it to decide, explain and trigger the refund. The same component interprets the text, recalls policy, calculates the amount and takes action. The demo looks elegant. The control structure is fragile.</p><p>A governed design separates the jobs:</p><ul><li>A data service retrieves the two transactions, account identity and prior refund history.</li><li>A Jev-style judgment asks bounded questions: Is a refund requested? Does the evidence indicate a duplicate charge? Which queue owns the case?</li><li>Deterministic code verifies transaction identity, refund limits, account status and exact amount.</li><li>A decision policy maps calibrated probability and financial consequence to auto-act, review or abstain.</li><li>An LLM explains the evidence and next steps in clear language.</li><li>The application logs the state, model version, judgment, confidence, rules, approval, action and final outcome.</li></ul><p>Notice what governance accomplished. The model was allowed to judge ambiguity, but not to invent a transaction, change a refund limit or move money outside the control layer. A fast judgment became useful because the architecture constrained its authority.</p><p>Consider thousands of delayed orders, inventory shortages, document gaps and supplier messages arriving each day. A classical rules engine can route obvious cases but leaves a long tail of ambiguous exceptions. An LLM can summarise them, yet free-form prose is awkward for automated prioritisation.</p><p>A hybrid design can use data science to estimate delay and business impact, a typed decision model to classify the exception and score review priority, optimisation to allocate scarce expediting capacity, deterministic controls to preserve budget and compliance constraints, and an LLM to prepare the operational brief. The human planner then focuses on high-consequence or uncertain cases rather than reading every message.</p><p>This example travels beyond supply chain. The same pattern applies to insurance claims, support tickets, fraud alerts, document review, maintenance triage, procurement exceptions and regulatory evidence. The domain changes. The governance logic does not: prediction is not judgment, judgment is not permission, and permission is not proof of value.</p><p>Most AI governance becomes practical when confidence is combined with consequence. Confidence asks how reliable the model appears for this case and segment. Consequence asks what happens if it is wrong: financial loss, customer harm, safety impact, legal exposure, reversibility and reputational cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rP4A6ptowvk68GEn" /></figure><p>Figure 3. Three zones of machine authority. Confidence alone should never determine automation.</p><p>Use automation when the case is low consequence, the model is validated for the relevant segment, deterministic safeguards are active and the action is reversible. Examples might include spam routing, low-value categorisation or a routine recommendation within a narrow mandate.</p><p>Use review when evidence is useful but incomplete, consequences are moderate, novelty is present or confidence falls into an uncertain region. The reviewer should see the evidence, alternatives, confidence, rules and reason the case was escalated.</p><p>Abstention is a capability, not a failure. The system should decline to decide when essential evidence is missing, adversarial risk is high, the case is outside the validated population or the action exceeds machine authority. The correct output may be “request more information,” “route to an expert” or “do not act.”</p><p>A serious governance programme needs more than a model card and a red-team exercise. It needs sequential evidence. Each gate answers a different question, and passing one gate never proves the next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MI7rio-MzxQsbsYK" /></figure><p>Figure 4. The AI assurance ladder: technical validity at the bottom, real-world incremental outcome evidence at the top.</p><h3>Gate 1: Output contract</h3><p>Does every response conform to the permitted schema and type? This prevents malformed output, not bad judgment.</p><h3>Gate 2: Discrimination</h3><p>Can the system separate relevant classes, priorities or outcomes better than a credible baseline?</p><h3>Gate 3: Calibration</h3><p>When the model reports similar probabilities, do observed frequencies match, including by meaningful segment?</p><h3>Gate 4: Stability</h3><p>Do materially equivalent states receive consistent judgments? Test paraphrases, order changes and repeated calls.</p><h3>Gate 5: Robustness</h3><p>How does the system respond to missing fields, noise, unusual combinations, adversarial text and prompt injection?</p><h3>Gate 6: Decision value</h3><p>Does the judgment improve expected utility after review cost, delay, false approvals, false rejections and operational constraints?</p><h3>Gate 7: Outcome evidence</h3><p>Does the complete workflow create incremental value versus a credible counterfactual, not merely a plausible story?</p><p>Begin in shadow mode. The existing process remains in control while the candidate systems record what they would have decided. Use the same frozen decision frames to compare at least four alternatives: incumbent rules, a conventional classifier, a general LLM constrained to structured output and a Jev-style typed judgment layer. If the future architecture is hybrid, test the integrated workflow as a fifth system.</p><p>The evaluation should report distributions and failure slices, not one average score. Useful families include discrimination, calibration, stability, robustness, latency and cost, operational coverage, abstention rate, human-review burden and decision utility. Thresholds should be chosen from expected loss and authority, not because 0.80 “feels confident.”</p><p>For example, a false approval of a large payment and a false rejection of a low-value request do not carry the same cost. The threshold should vary with consequence, evidence quality and reversibility. A calibrated probability can inform that policy, but deterministic code should execute the boundary.</p><p>The NIST AI Risk Management Framework is intended to help organisations incorporate trustworthiness considerations into the design, development, use and evaluation of AI systems. Its Generative AI Profile is a cross-sector companion focused on risks that are unique to or intensified by generative AI. These resources are useful because they move governance beyond a single model test and across the lifecycle.</p><p>For practical security work, OWASP’s LLM risk catalogue provides a threat-oriented lens. For a hybrid Jev-and-LLM system, the key lesson is to govern the full call path: data sources, model supply chain, retrieval, prompts, typed questions, output handling, tool permissions, cost limits, human checkpoints and audit evidence. A safe model inside an unsafe application is still an unsafe system.</p><p>Before allowing an AI system to make or materially influence a business decision, leaders should be able to answer the following questions without relying on marketing language:</p><ul><li>What exact decision is being made, for whom, at what frequency and with what consequence?</li><li>What is the trusted state, and which fields are measurements versus proxies?</li><li>Which component predicts, which judges, which controls, which explains and which authorizes?</li><li>What answer space is permitted, and what important option might be missing?</li><li>How were probabilities calibrated, and where does calibration break by segment?</li><li>What happens when evidence is missing, contradictory, novel or adversarial?</li><li>Which constraints are enforced in deterministic code?</li><li>What decisions can auto-act, require review or must abstain?</li><li>What can the human reviewer see, change and stop?</li><li>Is every input, version, judgment, rule, override, action and outcome logged?</li><li>What baseline will prove incremental value?</li><li>Who can roll the system back, and what signal triggers that rollback?</li></ul><p>AI governance is often presented as a collection of principles, committees and documents. Those elements matter, but the strongest governance is executable. It appears inside the call path as typed schemas, source lineage, deterministic checks, thresholds, permissions, escalation, logging, rollback and outcome measurement.</p><p>Jev makes this architectural truth easier to see because it separates judgment from language generation. Its typed outputs can be valuable where software needs a bounded decision rather than a paragraph. LLMs remain valuable where people need explanation, synthesis and interaction. Data science remains essential for estimation and uncertainty. Optimisation remains essential for feasible action. Humans remain essential for authority, exceptions and accountability.</p><p>The future is therefore not Jev replacing LLMs, or LLMs replacing data science. The stronger design is composable: data science predicts, Jev judges, deterministic systems control, LLMs explain and accountable people own consequential decisions. Every layer produces evidence for the next, and real outcomes return to improve the system.</p><ol><li>TypeSafe AI, “System One.” <a href="https://docs.typesafe.ai/concepts/system-one">https://docs.typesafe.ai/concepts/system-one</a></li><li>NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” July 2024. <a href="https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence">https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence</a></li><li>NIST, “AI Risk Management Framework.” <a href="https://www.nist.gov/itl/ai-risk-management-framework">https://www.nist.gov/itl/ai-risk-management-framework</a></li><li>OWASP, “Top 10 for LLM Applications 2025.” <a href="https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/">https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/</a></li><li>Internal conceptual foundations: From Prediction to Decision: Jev AI + Data Science; HT4 Decision Intelligence materials.</li></ol><p>#AIGovernance #ArtificialIntelligence #GenerativeAI #LargeLanguageModels #LLM #JevAI #SystemOne #DecisionIntelligence #DecisionScience #DataScience #MachineLearning #ResponsibleAI #TrustworthyAI #EnterpriseAI #AIAssurance #AIValidation #AIEthics #AIRiskManagement #HumanInTheLoop #ModelGovernance #MLOps #AIArchitecture #AgenticAI #ExplainableAI #DigitalTransformation</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18bcd2d7cac4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Decision Was Never a Paragraph : Keywords vs Prompt vs Jev AI Decision]]></title>
            <link>https://medium.com/@eddie.hc.tsui/the-decision-was-never-a-paragraph-keywords-vs-prompt-vs-jev-ai-decision-223faa3071c8?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/223faa3071c8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[jevs]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 09:56:01 GMT</pubDate>
            <atom:updated>2026-09-23T09:56:01.387Z</atom:updated>
            <content:encoded><![CDATA[<p>Six tickets, one rubric, three methods. Keyword rules, a chat model asked to return JSON, and Jev, a decision model that returns a probability your code can threshold.</p><p>By Eddie Tsui · 17 min read</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sesdZVQhnqws9GJu" /></figure><p>Six tickets. One rubric. The difference is what each method is allowed to return.</p><p>A support desk does not need a paragraph. It needs a queue, a label, and a decision about whether a person should see the ticket before the system moves it.</p><p>For a long time we got that decision by asking a language model to pretend it was a classifier. Write a careful prompt. Demand JSON. Parse the JSON. Hope the number next to “probability” behaves like one. Before that, we used a word list, because a word list is easy to explain and impossible to surprise in a way you can defend.</p><p>I ran the same six tickets through all three, on 23 September 2026. Same state. Same questions. This is a controlled comparison, not a production A/B test, and six tickets are not a benchmark. They are enough to see the shape of the difference.</p><p>The argument in one line: the advantage is the contract. A decision model returns a typed probability inside options you defined. A chat model returns text you then interpret. A rule returns whatever the word list was taught to see.</p><p>Jev 1.13 is a System One model from TypeSafe, listed on OpenRouter as <a href="https://www.google.com/url?q=https://openrouter.ai/typesafe/jev-1.13&amp;sa=D&amp;source=editors&amp;ust=1790139107593525&amp;usg=AOvVaw1FmbtIllxI_X1ZUUEFdtTV">typesafe/jev-1.13</a> . Released 18 September 2026. Text in, structured decisions out. List price that day: $0.042 per million input tokens, $0 for output. Context is 32,000 tokens, state plus questions. One provider, TypeSafe, and OpenRouter forwards the request there directly. The call does not use the chat-completions endpoint. You post to the Decisions API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zYRAWaMILFk_g-0m" /></figure><p>Rules return a hit. A chat model returns text. Jev returns a value inside a type you declared.</p><p>1. The job hiding inside the prompt</p><p>Most “AI classification” systems I have watched a team ship are a prompt with a costume on.</p><p>The costume says: you are a careful analyst, return only JSON, use these labels, include a confidence. The job underneath is smaller. Is this a defect. Which queue owns it. How fast does someone need to look. We used to answer with rules, because “password reset” really does belong to account, and a human cleaned up the rest. The rest was most of the volume, which is why the rule felt successful.</p><p>Then the messy middle grew, and we pasted the ticket into a chat model. The first week looks like magic. It hears “this is not an error.” It invents a team the word list does not have. It attaches a sentence that sounds like a reason. The bill arrives as a parse failure, a repair prompt, and probabilities that cluster on 0.9 and 0.8 and 1.0. The explanation is written by the same system that picked the label, so a wrong label arrives with a fluent defense. I have trusted that defense.</p><p>The job was never the paragraph. The job was a branch in code: move this ticket, or stop and ask a person.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yqxGxfB4Fa7SlB3f" /></figure><p>Every method saw these six texts, plus a customer tier, and nothing else.</p><p>2. Six tickets, one desk</p><p>I kept the rubric small on purpose. If the comparison needs a clever prompt to look fair, it is not fair.</p><p>The state is two fields. Customer tier is standard or enterprise. Ticket is the customer’s text. Three questions, the ones a triage desk actually asks.</p><p>Is it a defect. A noul, Jev’s name for a yes-or-no probability. True means the customer describes broken or unexpected product behavior. False means they are asking a question or requesting a feature. The instruction is: is the customer reporting a software defect?</p><p>Which team owns it. A choice among three options. Account covers login, permissions, or profile. Frontend covers rendering, layout, or browser behavior. Payments covers checkout, billing, or payment processing. There is no fourth team. If the ticket needs one, the model is not allowed to invent it.</p><p>How urgent is it. A score on an ordered scale. Can wait for the next release. Should be fixed this week. Blocking revenue right now. Index 0 is the first line. A score near 2 sits on the last line.</p><p>The six tickets:</p><p>T1, standard. “The login button does nothing when clicked.”</p><p>T2, enterprise. “My checkout page shows a blank screen after I click Pay. I have tried two browsers.” This is the example in OpenRouter’s Jev tutorial. I kept it so the numbers can sit next to a public sample.</p><p>T3, standard. “Could you add a dark mode? The light theme is broken for our night shift. This is not an error in the product, we just need the feature before next quarter.”</p><p>T4, standard. “I was charged twice this morning and I need a refund today. Invoice 4412. This is unacceptable.”</p><p>T5, standard. “Something feels off with the app lately. Not sure what. Can someone take a look when you have time?”</p><p>T6, enterprise. “If I change the invoice id in the URL I can see another customer’s invoice. This is happening right now on our enterprise account.”</p><p>Read them once as a person. T1 has two plausible owners. T2 is losing money now. T3 is a feature wearing bug vocabulary. T4 may be a defect or a billing dispute. T5 is almost empty. T6 is an access-control failure, and the urgency scale I wrote has no line for that. I left the bad scale in on purpose.</p><p>3. The keyword rule</p><p>The rule is short enough to paste into a review. That is its virtue, and the whole of its virtue.</p><p>Call it a bug if the text contains any of: error, broken, does nothing, blank, bug, crash, charged twice, see another.</p><p>Pick a team in order. Pay, checkout, invoice, charge, refund, or billing goes to payments. Else login, password, permission, or account goes to account. Else button, screen, browser, page, click, theme, or url goes to frontend. Else leave it unassigned.</p><p>Urgency is 2 if the text contains right now, today, unacceptable, or happening. Else, if the customer is enterprise and the rule already called it a bug, urgency is 2. Else a bug is 1. Else 0.</p><p>On the six tickets:</p><p>T1. Bug, from “does nothing.” Account, from “login.” Urgency 1.</p><p>T2. Bug, from “blank.” Payments, from “checkout” and “Pay.” Urgency 2, because the customer is enterprise and the rule called it a bug.</p><p>T3. Bug, from “broken” and “error.” Frontend, from “theme.” Urgency 1.</p><p>T4. Bug, from “charged twice.” Payments. Urgency 2, from “today” and “unacceptable.”</p><p>T5. Not a bug. Unassigned. Urgency 0.</p><p>T6. Bug, from “see another.” Payments, from “invoice.” Urgency 2, from “right now” and “happening,” and also because it is an enterprise bug.</p><p>T1, T2, T5, and the bug label on T6 are defensible if you only read the hits. T5’s silence is the correct silence.</p><p>T3 is the one I would not ship. The customer asked for a feature and then said this is not an error. The rule counted “broken” and “error” and filed a defect at urgency 1. A word list has no representation of a sentence. T4 is the quieter miss: “charged twice” was on my bug list, so the rule is certain, and it pages, even though the text may be an authorization hold. T6’s team is stuck the same way. “Invoice” sends an access-control failure to payments, which is the object, not the fault.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JYR53DW0A6zZkJQ_" /></figure><p>T3 is a feature request. The rule files a bug because both trap words are present.</p><p>I have written this rule, in slightly different words, at more than one company. It is auditable. It is free. Run it twice and the labels match, bit for bit. When the decision really is the presence of a token, a SKU, a country code, a blocked domain, I would still use a rule and I would refuse a model.</p><p>The moment the decision is a judgment about a sentence, the rule’s certainty becomes a kind of blindness. It fails by being sure.</p><p>4. The chat model we asked to behave</p><p>The second method is what teams adopt when the rule starts embarrassing them. Put the rubric in the prompt. Demand JSON only. Ask for a boolean, a team, a score from 0 to 2, and probabilities that sum to 1. Set temperature to 0 and tell yourself the call is now a measurement.</p><p>I wanted that method to be Claude Sonnet 4.5, the model people reach for when they say “use a strong model and parse it.” On this OpenRouter key, Sonnet 4.5 returned HTTP 403. So did GPT-4o-mini, Gemini 2.5 Flash, and Claude Haiku 4.5. The error was the same each time: the request is prohibited due to a violation of provider terms of service. I am not going to invent those outputs. A comparison that swaps in a remembered demo is not a comparison.</p><p>The chat model this key could call, and the one I measured, is meta-llama/llama-3.3–70b-instruct at temperature 0. That is a serious classifier-by-prompt, and it is what a lot of teams wire up when the frontier model is expensive, gated, or, as here, refused. The method under test is the method: ask a chat model for a label, then parse it.</p><p>Llama returned valid JSON on all six. The probabilities were round.</p><p>T1. Bug at 0.90. Account 0.80, frontend 0.20. Urgency 1.5, with 0.10 / 0.40 / 0.50. About 3.1 seconds and $0.000074.</p><p>T2. Bug at 0.90. Team: frontend at 0.60, payments at 0.30, account at 0.10. Urgency 2. About 3.7 seconds and $0.000105.</p><p>T3. Not a bug, at 0.20. Frontend 0.80. Urgency 0.5, spread 0.40 / 0.50 / 0.10. About 1.4 seconds and $0.000113.</p><p>T4. Bug at 0.90. Payments at 1.0. Urgency 2, with the entire distribution on that level. About 9.2 seconds and $0.000058.</p><p>T5. Bug at 0.70. Frontend 0.60. Urgency 0.2. About 6.9 seconds and $0.000058.</p><p>T6. Bug at 0.99. Account 0.80. Urgency 2, with probability 1.0 on “blocking revenue.” About 3.2 seconds and $0.000079.</p><p>Three of those numbers are the method, not the model brand.</p><p>On T2, a blank Pay screen, the rule and Jev both said payments. Llama said frontend at 0.60, payments at 0.30. Argmax files the wrong queue. The 0.60 is a token I asked for, not a calibrated doubt. On T4, Llama called a double charge a bug at 0.90 and put the entire urgency distribution on “blocking revenue.” That is a page. The text supports a refund conversation. It does not support certainty that the product is broken. On T3, Llama got the feature request right, at 0.20, which the rule missed. I am keeping that. The 0.20 is still a self-report: nothing in the API stopped the model writing 0.85, and the parser would have accepted either. On T6 it put probability 1.0 on “blocking revenue” for a privacy incident. The scale was used as a mood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UaGaM03gL_lo9fLA" /></figure><p>Temperature 0 did not turn a language model into a measurement instrument.</p><p>The six calls took 1.4, 3.1, 3.2, 3.7, 6.9, and 9.2 seconds. T4 is not a harder question than T3. The slow call is generation, not difficulty. And the probabilities are not a distribution the model is forced to emit. I asked them to sum to 1, and they mostly did, because the model is obedient. Obedience is not calibration. A threshold on 0.90 is a threshold on a writing style.</p><p>5. A decision frame, not a prompt</p><p>Jev asks you to write the decision down as a frame before you send the text.</p><p>Three parts go to the model. The fourth stays in your repository.</p><p>State. What the model may see. Here, tier and ticket, sent as an object. A bare string turns every forgotten field into more prose.</p><p>Questions. A type, an instruction, and criteria in your words.</p><p>A noul is the probability the answer is yes. Near 0.5 means yes and no are about equally likely. It does not mean the bug is “medium.” Ask severity on a score.</p><p>A choice is the winning key, a probability for every option you listed, and a confidence. The key has to be one of yours. This rubric cannot return a fourth team.</p><p>A score is a probability-weighted position on an ordered list you wrote, plus a probability per level, a confidence, and a legend that repeats your labels. Index 0 is the first line. Store the legend with the number, or the index will lie after someone edits the wording.</p><p>Policy. Jev returns numbers. The branch that pages someone lives in your code. A vendor threshold is somebody else’s cost of a mistake.</p><p>I called POST <a href="https://openrouter.ai/api/alpha/decisions">https://openrouter.ai/api/alpha/decisions</a> with the model set to typesafe/jev-1.13. Every response named the snapshot that served it: typesafe/jev-1.13–20260917. The dated suffix is expected. The alias ~typesafe/jev-latest tracks the family, which is what you want in a demo and what you do not want under a threshold you tuned last quarter. Pin 1.13 when the bars have to stay put. There is a second surface, POST /api/v1/systemone, for people who already use the TypeSafe SDK and only want to change the base URL. I used the Decisions API. Same key, same bill.</p><p>One property I would not skip in review: questions in a single request are answered in parallel, and they cannot see each other’s answers. Urgency does not get to peek at the bug label and then agree with it. If you have watched a chat model talk itself from “this might be a feature” into “therefore it is urgent, because bugs are urgent,” you have watched the failure this removes. If two judgments really depend on each other, make a second call, with the first answer written into the state, where a person can see the dependence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*toE2TSk4vjC3C8VA" /></figure><p>The model stops at the typed answer. Acting is a separate program.</p><p>There is no reasoning trace. The <a href="https://www.google.com/url?q=https://openrouter.ai/docs/guides/community/jev&amp;sa=D&amp;source=editors&amp;ust=1790139107618662&amp;usg=AOvVaw2s6A8pGPsjrW4Xt-t0Wchh">Jev documentation on OpenRouter</a> is plain about it. If you need a sentence, ask a chat model after the decision, or hand the case to a person. Do not ask the decider to also be the advocate.</p><p>6. What came back</p><p>Every call resolved to typesafe/jev-1.13–20260917, provider TypeSafe. Probabilities below are rounded to two decimals. Latency is wall clock from my machine, network included. The model page lists TypeSafe’s P50 at 0.28 seconds. My six calls landed between 0.49 and 0.88 seconds.</p><p>T1, 882 ms, $0.000019572, 466 input tokens and 70 output. Noul 0.97. Team: account 0.59, frontend 0.41, payments 0, confidence 0.38. Score 1.69, level probabilities 0.01 / 0.29 / 0.70, confidence 0.53.</p><p>T2, 590 ms, $0.000019992, 476 in and 70 out. Noul 0.96. Payments 0.80, frontend 0.20, account 0, confidence 0.70. Score 1.99, probabilities 0 / 0.01 / 0.99, confidence 0.99.</p><p>T3, 587 ms, $0.000020706, 493 in and 70 out. Noul 0.08. Frontend 1.00, confidence 1.00. Score 0.07, probabilities 0.94 / 0.06 / 0, confidence 0.90.</p><p>T4, 629 ms, $0.000020244, 482 in and 70 out. Noul 0.45. Payments 1.00, confidence 1.00. Score 1.67, probabilities 0 / 0.32 / 0.68, confidence 0.51.</p><p>T5, 487 ms, $0.00002016, 480 in and 70 out. Noul 0.57. Frontend 0.99, account 0.01, confidence 0.99. Score 0.33, probabilities 0.68 / 0.32 / 0, confidence 0.51.</p><p>T6, 493 ms, $0.00002037, 485 in and 70 out. Noul 0.95. Account 0.81, payments 0.18, frontend 0.01, confidence 0.71. Score 1.41, probabilities 0.01 / 0.57 / 0.42, confidence 0.36.</p><p>Six calls, about $0.000121 together. Output tokens were reported and not billed. That matches the page: you pay for input, and the decision comes back free.</p><p>I called T5 again a moment later, because I wanted to know whether to tell you this was deterministic. It is not. The noul moved between 0.54 and 0.55. The score moved between 0.28 and 0.31. The team stayed frontend at 0.99. OpenRouter’s tutorial says the probabilities differ slightly from run to run. The band did not change. Both reads are an abstain on the bug question. I will not claim bit-identical replay. A repeat stayed inside the same operational decision, on this ticket, this day. That is a weaker claim, and it is the one the evidence supports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WZQSa_zBFBA2ziWx" /></figure><p>On T2, team confidence is 0.70. The winning probability is 0.80. They are different fields.</p><p>T2 is the clean reading of the three types. Noul 0.96 is a defect under any bar I would ship. Payments at 0.80, with confidence at 0.70, are different fields: confidence summarizes how concentrated the distribution is, and the docs do not treat it as a warrant that the workflow is safe. I am not going to invent the formula. The score of 1.99, confidence 0.99, sits on “blocking revenue right now,” and the legend comes back with the number so the words stay in the record. Llama put this same ticket on frontend. The easy case is where the chat model spent 3.7 seconds and named the wrong team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wESChbH_jyJWRYy0" /></figure><p>Highlighted rows are where a boolean hides the evidence. Six tickets, not a benchmark.</p><p>7. Where they split</p><p>The interesting tickets are the ones a demo leaves out.</p><p>T3, the feature request. The rule filed a bug. Llama said not a bug at 0.20. Jev’s noul was 0.08, frontend at 1.00, score 0.07. Six tickets cannot crown a model. What this one shows is that the rule cannot represent “not an error,” and that 0.08 is far enough from the middle to file a feature, send it to frontend, and leave it on a backlog. I would trust that direction. I would not tune a production cut on Llama’s self-reported 0.20.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fBBF1lZNRVq8NnO6" /></figure><p>Same sentence. The rule counts words. The other two read the request. Only one returns a closed probability.</p><p>T1, the dead login. Everyone agrees it is a defect. The rule stops at account, because “login” outranks “button.” Llama says account at 0.80. Jev says account 0.59, frontend 0.41, confidence 0.38. A dead button is an account problem if the session never starts, and a frontend problem if the handler is dead. The text does not say which. Noul 0.97 can auto-label the defect. The team call should wait. Score 1.69 with confidence 0.53 should not page anyone, even with a lot of mass on the top level. Mass and confidence are different signals. Let the stricter one win.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYsVLR77My0XADpZ" /></figure><p>The bug label can be automatic. The owner cannot.</p><p>T4, the double charge. The rule is certain it is a bug and certain it is urgent. Llama is nearly as certain: bug at 0.90, urgency pinned on level 2. Jev’s team call is payments at 1.00. The bug call is 0.45. The score is 1.67, confidence 0.51. That 0.45 is a coin flip between a double bill and a customer reading an authorization hold. A boolean has to pick engineering. A probability of 0.45 is a place to stop. Send it to payments. Do not auto-label the bug. Do not page. A person on that queue decides whether engineering is involved.</p><p>T6, the invoice leak. The rule saw “invoice” and said payments. Jev said account at 0.81, payments at 0.18, confidence 0.71. Changing an id and seeing another customer’s document is authorization, and payments as the runner-up is the right shape. Llama also said account. The word list is the one that is stuck. Urgency is where I defend the model against my own rubric. Jev’s score is 1.41, confidence 0.36. Llama put 1.0 on “blocking revenue.” The scale says revenue. The ticket never mentions revenue. Low confidence is the model following the question. The fix is a question about exposed customer data, not a longer prompt.</p><p>T5, because it is the limit. The rule shrugged: not a bug, unassigned, urgency 0. Llama said bug at 0.70. Jev’s noul was 0.57, and the repeat was 0.54. Both are abstains. The team call was frontend at 0.99, which is a lot of confidence for “something feels off with the app.” I am not sanding that down. One confident field is not permission to act on the whole ticket.</p><p>8. The mechanism, slowly</p><p>Three habits make the types usable.</p><p>Use two cuts on a noul, not one. Above the high cut, act as yes. Below the low cut, act as no. Between them, stop. A single cut at 0.5 throws away the number you paid for.</p><p>Read the runner-up on a choice. Confidence describes concentration. It is not a safety certificate, and it is not the winning probability. T2 is the reminder: payments at 0.80, confidence at 0.70. Pick the bar from the cost of a wrong queue.</p><p>Keep the fraction on a score. 1.99 and 1.41 should not take the same action. Store the legend, or the index will lie after the rubric is edited.</p><p>The output space is closed. The model cannot answer “security” on T6, because that key was never offered. Naming a team that does not exist is unavailable. A wrong probability is still possible. T5’s frontend confidence of 0.99 is that failure, and it is visible.</p><p>Questions in one request are scored in parallel and cannot see each other. If urgency should depend on the bug call, write that in code, or make a second request whose state includes the first answer.</p><p>Context is 32,000 tokens for state and questions together. These calls used about 470 to 490. The response carries an id, the dated snapshot, and a cost. Store them with the ticket. A repeat can move a probability by a few hundredths. A wide abstain band is how you live with a model that holds its decision and fidgets in the digits.</p><p>9. What is left after the screenshot</p><p>Someone will see T3 and ask to put Jev on everything. The list for that meeting is short. Code receives a float, a key, or a position, so there is no JSON to repair. A new team is a diff in the criteria, not a sentence the model may honor. T4 at 0.45 is an action, ask a person, which the rule could not take and Llama’s 0.90 erased. Parallel questions do not coach each other inside one generation.</p><p>On this run, six Jev calls cost about $0.000121 and came back in under a second of wall clock. Six Llama calls cost about $0.000488 and took up to 9.2 seconds. The planning number is the list price, $0.042 per million input tokens, output free. The record is the state, the questions, the dated snapshot, the cost, and the id. A repeat is close, not identical. Promise a stable decision inside a band, and test it. T6’s confidence of 0.36 is a successful failure: the scale asked about revenue, the incident was access, and the model did not cover the mismatch with a 1.0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CbumHjKvCfWrPHnW" /></figure><p>A wrong probability can still happen. An invented team cannot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sya7foyXaqqephIk" /></figure><p>The provider page lists a P50 of 0.28s. These bars are the round trip I actually waited for.</p><p>10. What Jev will not do</p><p>It will not write the reply, summarize the thread, or say why account beat frontend. If you need a written reason, show a chat model the typed answer as a fact, or ask the human who took the abstain. Do not let the explanation vote.</p><p>It will not be calibrated because a page said “probabilities.” I have no labeled set here. T3 and T4 land in the right neighborhood for my reading. That is a sanity check, not a reliability diagram. The cuts in the next section are a policy I would start from. They are not a result I measured.</p><p>It will not stay bit-identical. A move from 0.54 to 0.57 should not flap the action, which is what the abstain band is for. Exact legal replay belongs in the rule layer. It will not rescue a revenue scale applied to a privacy incident, except by going uncertain. And it will not replace the chat model. The <a href="https://www.google.com/url?q=https://openrouter.ai/docs/guides/community/jev&amp;sa=D&amp;source=editors&amp;ust=1790139107636310&amp;usg=AOvVaw0Zyrx6OqjysPozKPpsauHm">OpenRouter docs</a> draw the same line. Prose is one job. A decision the program can act on is the other. One generation that does both lets the prose win.</p><p>11. The stack I would actually run</p><p>This policy is mine. It is not a TypeSafe default, and it is not tuned. I would move every cut after a few hundred labeled tickets. I am writing the cuts down so the six results have somewhere to land.</p><p>On the bug question, noul at or above 0.85 is a defect, and noul at or below 0.15 is not. Between them, a person owns the label. On the team question, auto-route only when confidence is at least 0.70. On urgency, page only when the score is at least 1.7 and confidence is at least 0.60. Above those three: if the bug question abstained, do not auto-close the ticket and do not page. You may still choose a queue when the choice itself is extreme. One confident field does not authorize the workflow.</p><p>T1. Noul 0.97, so the defect label is automatic. Team confidence 0.38, so a person picks the owner. Score 1.69 with confidence 0.53, so nobody is paged. The ticket lands in a shared inbox with the label already on it.</p><p>T2. Noul 0.96, payments confidence exactly 0.70, score 1.99 with confidence 0.99. This one pages the payments on-call. It is the only ticket in the set that clears every bar. It is also the only ticket where the evidence is both strong and matched to the scale.</p><p>T3. Noul 0.08, so it is not a defect. Frontend confidence 1.00, score 0.07. A feature request on the frontend backlog. No page. The rule would have opened a bug. This is the cleanest save in the set.</p><p>T4. Noul 0.45, so a human owns the defect label. Payments confidence 1.00, so the queue can be chosen. Score 1.67 with confidence 0.51, so no page. Billing sees it because the queue is right. Engineering is not woken up to adjudicate a refund.</p><p>T5. Noul 0.57, so a human owns the ticket. I would not auto-route on the frontend confidence of 0.99. The text is too thin for that confidence to spend. The rule’s “unassigned” is closer to the right action than a confident queue.</p><p>T6. Noul 0.95, so it is a defect. Account confidence 0.71, so it routes to account, and the UI should show payments at 0.18 as the runner-up. Score 1.41 with confidence 0.36, so the policy does not page. I would still want a person to move fast on an access-control ticket, which means the policy and the rubric are both incomplete. Add a question whose top level is “other customers’ data is exposed now,” and let that question page. Do not lower the revenue bar until a privacy incident happens to look like lost revenue. That is how severity scales get destroyed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*__tpZq6w0SfTV3p-" /></figure><p>These cuts are a starting policy. Move them when you know the cost of each mistake.</p><p>Around that policy, four layers. Rules first, for decisions that are not judgments: a dollar limit, a blocked sender, a required field. Jev second, model pinned, questions in version control beside the policy. A changed sentence in the criteria is a changed system. A person third, on the band between the cuts and on any rubric you already know is the wrong shape. The abstain path needs a queue with a name. A chat model last, drafting the reply or a note for the human, with no vote on the label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N7MfbpPAH3dn__nt" /></figure><p>The chat model is last, and it is optional.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M9kUKB7ZvHq8cfdH" /></figure><p>Keep the rule where the word is the policy. Keep the chat model where you need a sentence.</p><p>We spent years asking a writing system to pick a label, because the writing got good and the label was annoying. The label was the job. On these six tickets the leftover had a receipt. It was slower. It cost more, for text I was going to discard. It was sure, at 0.90, about a double charge that is not sure, and it named the wrong team on a blank checkout. On the dark-mode ticket it was better than my rule. That stays, so the rule does not get a sentimental defense.</p><p>Jev’s advantage was not a smarter tone. On T1 it was less sure than Llama about the owner, and that was the right amount of sure. On T4 it refused the bug label. On T6 it refused to pretend a revenue scale understood a privacy incident. On T3 it was sure in the direction the sentence actually goes, inside a type the code can branch on.</p><p>A probability you are willing to threshold is a different object from a paragraph you are willing to trust. I would ship the first. I would let the second write the email.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xEEyTXxRcXIO4MiB" /></figure><p>Ship the number. Let a person, or a later model, write the sentence.</p><p>#JevAI #DecisionModel #SystemOne #TypeSafe #OpenRouter #DataScience #AppliedAI #MachineLearning #LLM #AIEngineering #SupportTriage #TicketRouting #Classification #StructuredOutput #Probability #ModelCalibration #ConfidenceThresholds #HumanInTheLoop #AIInProduction #MLOps #PromptEngineering #DecisionSystems #AIWorkflow #ModelEvaluation #AIProduct</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=223faa3071c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jev Decision Gateway : Where Jev Actually Fits]]></title>
            <link>https://medium.com/@eddie.hc.tsui/jev-decision-gateway-where-jev-actually-fits-b1f9ad918fe3?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1f9ad918fe3</guid>
            <category><![CDATA[decision-intelligence]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 04:04:36 GMT</pubDate>
            <atom:updated>2026-09-22T04:04:36.272Z</atom:updated>
            <content:encoded><![CDATA[<p>12 practical ways to turn documents, evidence, analytics and AI agents into fast, typed, governed decisions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wuHszaW_HCniEvWi" /></figure><p>Jev as the missing decision layer between messy evidence and governed action.</p><p>Most enterprise systems do not need another chatbot. They need a reliable answer to a smaller question: What should happen next?</p><p>Enterprises have spent years building data platforms, predictive models, document extractors, compliance engines, supplier portals, search tools and AI copilots. These systems can read, retrieve, calculate, forecast and explain. Yet many still stop one step before action. A person must interpret the output, decide which queue it belongs in, determine whether confidence is sufficient, and choose whether to proceed, review, request more evidence or block.</p><p>That gap is not primarily a generation problem. It is a judgment problem. Software repeatedly needs bounded decisions: Which pipeline should process this file? Is the evidence strong enough? Which specialist owns the exception? Should a recommendation be automated or reviewed? Is a retrieved passage decisive or merely contextual?</p><p>This is the design space Jev AI is trying to occupy. TypeSafe describes Jev as a System One model that evaluates text or JSON state and returns typed answers and probabilities rather than generated prose. The official documentation presents Choice, Score and yes-or-no probability primitives, and recommends combining their outputs with deterministic checks in code before routing a case to action or review.</p><p>A general-purpose language model is designed to generate strings: explanations, plans, code and dialogue. Jev is designed for a different interface. The application supplies the current state and defines the permitted answer space in advance. Jev returns a structured decision with probability information that ordinary software can inspect.</p><p>A typical decision contract might ask:</p><ul><li>Which route applies: auto-continue, human review, request evidence or block?</li><li>What is the primary issue: data quality, identity ambiguity, missing evidence, policy conflict or unsupported case?</li><li>How urgent is the case on a bounded scale?</li><li>What is the probability that the available evidence is sufficient?</li></ul><p>This narrower design is valuable because workflows usually do not need an essay at every branch. They need a stable value that an API, queue, policy engine or user interface can use immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*agtPTjJRoxKP7mlF" /></figure><p>Figure 1. Why Jev is useful: typed outputs, bounded choices, probabilities, parallel questions and software-ready decisions.</p><h3>1. It makes the answer space explicit</h3><p>The application defines the allowed choices. This forces product teams to clarify what the workflow can actually do. A badly designed choice set will still produce a bad system, but the ambiguity becomes visible and testable.</p><h3>2. It converts semantic ambiguity into software-ready values</h3><p>Documents, notes, supplier replies and extracted fields often contain meaning that simple rules cannot capture. Jev can classify that state into a bounded route, score or probability without asking downstream software to parse a paragraph.</p><h3>3. It can answer several focused questions about the same state</h3><p>The official System One documentation describes asking independent typed questions together. A workflow can obtain route, urgency, evidence sufficiency and issue category from the same state, then combine the answers with policy checks.</p><h3>4. It creates a natural escalation mechanism</h3><p>Probability and confidence can help determine when a system should act, request more information or escalate. This does not make an individual answer correct. Calibration is a population property and must be validated on the organisation’s own data.</p><h3>5. It complements existing systems rather than replacing them</h3><p>Jev is most interesting as a layer between analytics and control. It can sit behind a document extractor, retrieval engine, forecasting model, supplier portal or agent. The existing system continues to own its specialist job; Jev decides the next bounded route.</p><p>The safest design is not “state in, action out.” It is a governed chain in which Jev contributes judgment but does not own authority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JcLrsXz9bRcugwCc" /></figure><p>Figure 2. Safe workflow: trusted state, typed questions, Jev judgment, hard rules, risk gate, action or review, and outcome learning.</p><ol><li>Build a trusted state. Assemble the smallest complete snapshot required for the decision. Separate source facts from generated summaries.</li><li>Define typed questions. Specify allowed choices, scales and yes-or-no statements. Include an explicit insufficient-evidence or review route.</li><li>Call Jev for bounded judgment. Store returned values, probability information, model version and decision contract version.</li><li>Run hard rules in code. Apply legal thresholds, arithmetic, identity checks, permissions, capacity limits and non-negotiable policies deterministically.</li><li>Apply a risk gate. Combine confidence, evidence quality and consequence. Low-risk cases may automate; uncertain or consequential cases should escalate.</li><li>Execute or review. Log the final decision, owner, override and rationale.</li><li>Learn from outcomes. Compare decisions with actual outcomes, human overrides and baseline policies. Recalibrate or redesign the choice set when failure patterns emerge.</li></ol><p>Jev judges. Code controls. People own the consequences.</p><p>Instead of embedding different ad hoc decision logic in every product, an organisation can create one reusable Decision Gateway. Each domain supplies its own state and permitted answers, while sharing a common governance envelope.</p><p>This contract is a proposed design pattern, not a statement that Jev provides those exact domain labels. The value comes from giving many systems a consistent way to express uncertainty, ownership and next action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x6_fm_okQlKdyoJ3" /></figure><p>Figure 3. Jev use-case landscape: one decision gateway serving multiple bounded enterprise judgments.</p><h3>1. Document Routing</h3><p>An incoming file may be a laboratory report, inspection record, certificate, declaration, invoice or unsupported attachment. Jev can choose the appropriate processing pipeline or manual-review queue. The benefit is faster intake and fewer files sent through the wrong extractor.</p><p>Other options: deterministic metadata rules, a trained document classifier or a multimodal language model. Jev is attractive when the file context is semantic but the routes are known.</p><h3>2. Extraction Quality Triage</h3><p>After a document AI service extracts fields, Jev can classify the case as auto-accept, targeted field review, full review, re-extract or reject. It may also identify the likely failure category, such as missing pages, conflicting fields or unsupported layout.</p><p>Other options: OCR confidence thresholds, validation rules and human sampling. These should remain part of the solution because Jev should not replace exact field checks.</p><h3>3. Evidence Relevance and Sufficiency</h3><p>A compliance or audit workflow may retrieve several documents for a claim. Jev can classify each item as directly supportive, indirectly supportive, contradictory, contextual or unrelated, then judge whether the overall packet is sufficient for the next stage.</p><p>Other options: embedding similarity, rerankers, natural-language inference and expert review. Jev is useful as the bounded decision layer after retrieval, not as the evidence repository itself.</p><h3>4. Compliance Case Routing</h3><p>A regulatory question can be routed to applicability, evidence requirements, threshold interpretation, product release, supplier risk, audit response or specialist review. Jev can also label the reason for escalation.</p><p>Other options: rule engines, taxonomies and knowledge graphs. Final legal applicability, thresholds and release decisions should remain governed by authoritative rules and accountable experts.</p><h3>5. Supplier Response Triage</h3><p>Supplier replies can be classified as complete, partial, irrelevant, contradictory, expired or wrong in scope. Jev can recommend the next workflow route: request a missing field, ask for a replacement document, initiate testing, resolve identity or escalate.</p><p>Other options: business-process rules, response-propensity models and human case management. Jev helps when replies are unstructured but actions are bounded.</p><h3>6. Search Relevance Screening</h3><p>Before sending retrieved passages to an expensive reasoning model, Jev can classify them as decisive, supporting, contextual, outdated or irrelevant. This can create a focused evidence set for the downstream answer.</p><p>Other options: cross-encoder rerankers, vector similarity and retrieval rules. Jev should be compared against specialist rerankers rather than assumed superior.</p><h3>7. Recommendation Triage</h3><p>A forecasting or optimisation system may produce thousands of candidate actions. Jev can determine which are auto-eligible, review-required, deferred, rejected or held for data-quality investigation, and identify the primary ambiguity.</p><p>Other options: supervised classifiers, gradient boosting and deterministic business policies. Quantities, economics and constraints should remain in data-science and optimisation code.</p><h3>8. Analytical-Regime Selection</h3><p>When history is sparse, a system may need to choose among direct history, a family prior, a peer analogue, a hierarchical estimate or manual review. Jev can select the route from contextual evidence while each analytical method remains independently validated.</p><p>Other options: mixture-of-experts gating, model-selection rules or meta-learning. If abundant labelled outcomes exist, a classical gating model may be preferable.</p><h3>9. Entity-Match Review</h3><p>After deterministic name, address and geographic matching produce candidate entities, Jev can classify an ambiguous pair as likely match, possible duplicate, related but distinct, no match or human verification required.</p><p>Other options: probabilistic record linkage, graph matching and embeddings. Jev belongs at the ambiguous tail, not at the beginning of the matching pipeline.</p><h3>10. Output and Disclosure Pre-Screening</h3><p>A data-product platform can use Jev to route a request toward publishable, publishable with caveat, internal only, aggregation only or legal review. The real disclosure permission must still be enforced through entitlement, contract and policy code.</p><p>Other options: access-control engines, data-loss prevention and deterministic disclosure policy. These controls are mandatory for consequential release.</p><h3>11. AI Governance Triage</h3><p>An AI governance platform can classify a use case or incident by risk theme, required review and escalation owner. Examples include data issues, prompt injection, unsafe action, access-control failure, bias concern or model drift.</p><p>Other options: policy-as-code, governance questionnaires, GRC workflows and risk committees. Jev can accelerate first-line triage but should not issue the organisation’s formal legal designation by itself.</p><h3>12. Agent Tool and Safety Routing</h3><p>An AI agent can use Jev to choose which tool should handle a request and whether a proposed action should proceed, be reviewed or be blocked. This is a natural fit because the tool list and safety routes are predefined.</p><p>Other options: deterministic routers, compact classifiers and general LLM function calling. High-risk agent actions still need explicit permissions, sandboxing and transaction controls.</p><p>The most effective design does not ask one model to perform every cognitive and operational job. It separates prediction, judgment, control, explanation and authority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5kuWKPt8RUpSC2F5" /></figure><p>Figure 4. Four engines, one decision system: data science predicts, Jev judges, deterministic code controls, and language models plus people explain and own.</p><ul><li>Data science estimates quantities, risks and uncertainty from historical data.</li><li>Jev classifies, scores and routes ambiguous state within a predefined answer space.</li><li>Deterministic code performs exact calculations, policy checks, permissions and constrained optimisation.</li><li>A grounded LLM explains the evidence, alternatives and caveats in language people can understand.</li><li>Accountable people approve high-consequence cases, manage exceptions and own the final decision.</li></ul><p>This separation is more than a technical convenience. It makes failure diagnosable. If an outcome is poor, the organisation can ask whether the evidence was wrong, the prediction was weak, the judgment route was wrong, the rule was misconfigured, the user overrode the system or execution failed.</p><ul><li>Do not use it for exact arithmetic, legal thresholds, inventory conservation or payment calculations.</li><li>Do not use it where the answer space is genuinely open-ended and explanation is the product.</li><li>Do not use it as a substitute for a strong supervised model when abundant labelled data and stable features already exist.</li><li>Do not let it make irreversible, high-consequence decisions without deterministic controls and accountable review.</li><li>Do not assume type safety means factual correctness. A valid value can still be the wrong decision.</li><li>Do not send images, audio or video directly when the official documentation states that the current model accepts text input only. Use a specialist extraction layer first.</li></ul><p>A successful API call is not evidence of a trustworthy decision system. Validation must climb from technical contract to measurable business value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9KLrDLR0TXkEYYa0" /></figure><p>Figure 5. Validation ladder: schema, accuracy, calibration, stability, robustness, decision value and human accountability.</p><ol><li>Schema: Are the choices, scales, definitions and insufficient-evidence routes complete and versioned?</li><li>Accuracy: Does Jev select the correct answer on an independently labelled dataset, by class and segment?</li><li>Calibration: When the system reports a probability, does observed frequency align across confidence buckets?</li><li>Stability: Do semantically equivalent states produce materially consistent answers across time and formatting variations?</li><li>Robustness: How does the system behave with missing fields, reordered options, injected instructions, noise and out-of-distribution cases?</li><li>Decision value: Does the Jev-assisted policy improve utility, review workload, latency or outcome quality against a credible baseline?</li><li>Accountability: Are escalation, override, audit, monitoring and rollback mechanisms operating as designed?</li></ol><p>A practical first experiment is shadow mode. The existing process continues to control action while Jev records what it would have decided. Compare four alternatives: the current rules, a simple trained classifier, a general-purpose LLM with structured output, and Jev.</p><p>Use a decision with a bounded answer space and visible review cost, such as document routing, extraction QA or supplier-response triage. Pre-register the labels, loss function, segments and escalation policy. Evaluate not only accuracy, but also calibration, disagreement, latency, cost, reviewer acceptance and error severity.</p><p>Only after the failure envelope is understood should the team introduce limited automation for low-consequence, high-confidence cases. Retain a review region and an explicit block or abstain region. Capture every override and outcome so that the system can be recalibrated and the decision contract improved.</p><p>Jev matters because it challenges a default assumption in enterprise AI: that every intelligent step should be a conversation with a large language model. Many workflows do not need another paragraph. They need a small, honest, constrained judgment that software can use.</p><p>The deeper opportunity is not to replace data science, rules engines, optimisers, LLMs or human experts. It is to connect them through a clearer division of labour. Evidence systems establish what is known. Data science estimates what may happen. Jev judges which bounded path the current state supports. Code checks what is allowed. People own what matters.</p><p>The winning enterprise AI stack may not be one giant model. It may be a disciplined team of specialised engines, each doing the job it can defend.</p><p>Recommended title: Where Jev AI Actually Fits: 12 Decision-Layer Use Cases Beyond the Chatbot</p><p>Alternative title: The Missing Decision Layer: How Jev Turns Enterprise AI Into Action</p><p>Subtitle: A visual field guide to routing, triage, evidence, governance and safe automation with System One models.</p><p>Suggested Medium tags: Artificial Intelligence, Data Science, Decision Intelligence, AI Governance, Enterprise AI.</p><p>TypeSafe AI, “System One.” <a href="https://docs.typesafe.ai/concepts/system-one">https://docs.typesafe.ai/concepts/system-one</a></p><p>TypeSafe AI, “Introducing System One Models &amp; Jev.” <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">https://typesafe.ai/blog/introducing-system-one-models-and-jev</a></p><p>Independent community collection, “Awesome Jev by TypeSafe.” <a href="https://github.com/Anil-matcha/awesome-jev-by-typesafe">https://github.com/Anil-matcha/awesome-jev-by-typesafe</a></p><p>#JevAI #SystemOneAI #ArtificialIntelligence #EnterpriseAI #DecisionIntelligence #DataScience #MachineLearning #AppliedAI #HybridAI #AIAutomation #IntelligentAutomation #AIArchitecture #DecisionScience #DecisionEngineering #AIGovernance #ResponsibleAI #TrustworthyAI #ExplainableAI #HumanInTheLoop #ModelValidation #AIValidation #EnterpriseAutomation #AgenticAI #AIAgents #DocumentAI #ComplianceTechnology #RegTech #DigitalTransformation #FutureOfAI #AILeadership</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1f9ad918fe3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Prediction to Decision : Why Jev AI Will Not Replace Data Science, but Could Complete It]]></title>
            <link>https://medium.com/@eddie.hc.tsui/from-prediction-to-decision-why-jev-ai-will-not-replace-data-science-but-could-complete-it-9c0241a7ccf7?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c0241a7ccf7</guid>
            <category><![CDATA[decision-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 03:46:44 GMT</pubDate>
            <atom:updated>2026-09-22T03:46:44.255Z</atom:updated>
            <content:encoded><![CDATA[<p>A rigorous blueprint for combining System One models, traditional machine learning, optimisation, deterministic controls and human accountability.</p><p>EVIDENCE → PREDICTION → JUDGMENT → CONTROL → ACTION → LEARNING</p><p>The future is not “Jev versus data science.” It is a governed hybrid: data science estimates the world, a System One model judges ambiguous states, optimisation selects feasible actions, deterministic code governs consequences, and outcome evidence teaches the system what deserved trust.</p><p>Imagine a familiar enterprise data-science programme. It forecasts demand, detects anomalies, estimates risk and ranks opportunities. The notebooks are credible. The validation is respectable. The dashboards are polished. Yet the operating business still asks the same stubborn questions: Should we act? Which case should move first? When is the evidence strong enough? Who owns the exception? What happens when the model is uncertain?</p><p>This is the hidden limit of many traditional data-science projects. They are excellent at producing beliefs about the world, but much weaker at converting those beliefs into governed, low-latency and machine-consumable decisions. The last mile becomes filled with brittle rules, manual interpretation, unstructured notes, approval queues and dashboards that describe rather than decide.</p><p>An organisation may improve predictive accuracy while leaving realised value nearly unchanged. A forecast does not move inventory. A risk score does not resolve a case. A recommendation does not prove that the action was approved, executed and beneficial.</p><p>The most important model may not be the one that predicts the future. It may be the one that decides what kind of decision the system is currently facing.</p><p>The breakthrough is not another accuracy contest. It is a redesign of the path from evidence to judgment to action to learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-6vsqzL6h5Ge8wpD" /></figure><p>Figure 1. Prediction estimates what may happen. Typed judgment chooses a bounded route. Deterministic control governs what may be executed.</p><p>TypeSafe AI describes System One models as models built to make fast, structured decisions that software can use directly. Jev evaluates a state and returns typed answers and probabilities rather than free-form text. Its documented primitives include a choice from predefined options, an ordered score and a yes-or-no probability.</p><p>That design matters. A general-purpose language model is naturally expressive: it can explain, brainstorm and reason in prose. Enterprise workflows often need something narrower, such as approve, review or block; route A, B or C; or the probability that evidence is sufficient. These constrained outputs can be inspected directly by downstream software.</p><p>A typed answer is not automatically a correct answer. Calibration is evaluated across groups of predictions and does not guarantee that an individual judgment is right. Constrained output also does not eliminate adversarial framing or prompt injection. Deterministic checks must therefore stand alongside, not beneath, model-based judgments.</p><p>The architectural question is not “Can Jev automate the workflow?” It is “Which uncertainty should Jev judge, and which consequences must remain under deterministic or human control?”</p><h3>Three Computational Jobs That Should Not Be Confused</h3><p>PREDICT → Estimate an unknown quantity such as demand, failure, risk or arrival time. Output: forecasts, distributions and uncertainty.</p><p>JUDGE → Interpret an ambiguous state and choose a bounded route. Output: a choice, score or probability over predefined answers.</p><p>CONTROL → Apply arithmetic, policy, capacity and safety constraints. Output: deterministic calculations, optimisation results and gates.</p><p>The hybrid architecture becomes clearer once prediction, judgment and control are treated as distinct computational jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rREBycEvZZy4S9zO" /></figure><p>Figure 2. A closed-loop intelligence system connects evidence, estimation, judgment, governed action and outcome learning.</p><p>A mature decision system should assign each technology the role it performs best. Jev is one candidate for the fast judgment layer. Conventional machine learning, operations research and deterministic software keep their own responsibilities.</p><h3>The Evidence-to-Outcome Control Loop</h3><ol><li>SENSE: Transactions, events, documents and context become versioned evidence.</li><li>ESTIMATE: Statistics and machine learning reconstruct truth, predict outcomes and quantify uncertainty.</li><li>JUDGE: A typed decision model classifies the state, scores priority and determines the route.</li><li>CONTROL AND ACT: Rules, economics, optimisation, approvals and APIs govern consequential action.</li><li>LEARN: Execution, overrides, outcomes and counterfactual evidence update every layer.</li></ol><p>SENSE → ESTIMATE → JUDGE → CONTROL AND ACT → LEARN → back to SENSE</p><p>The loop closes only when outcome evidence returns to the system. Without this, the organisation measures model performance rather than decision performance.</p><h3>Predictions are not policies</h3><p>A demand distribution does not specify the cost of acting, the cost of waiting, the inventory available, the receiving capacity or the authority required. Converting a prediction into action requires an explicit objective and constraints.</p><h3>Model quality is not workflow quality</h3><p>A statistically strong model can still fail because its recommendation arrives late, cannot be understood, violates a local rule or enters a queue without an owner.</p><h3>Accuracy hides asymmetric consequences</h3><p>A false positive may create unnecessary cost, while a false negative may lose revenue or expose risk. Thresholds must therefore be economic and contextual, not merely statistical.</p><h3>Enterprise state is partly unstructured</h3><p>Notes, requests, exception narratives and policy context can affect routing even when the core numerical evidence is structured.</p><h3>The system must know when not to decide</h3><p>Abstention, escalation and evidence insufficiency are product capabilities, not embarrassing failures.</p><p>Traditional data science remains indispensable because it provides estimation discipline, out-of-sample testing, uncertainty analysis, bias diagnosis and causal methods. The new approach adds an operating layer that turns analytical beliefs into bounded decisions while preserving the controls that make those decisions defensible.</p><p>The hybrid is not a compromise. It is a separation of concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yCAMtBlScxR13gSF" /></figure><p>Figure 3. The hybrid responsibility map assigns each computational job to the engine best suited to perform it.</p><p>The seduction of a new model is to route every unresolved problem through it. That would repeat an old mistake: replacing explicit system design with model enthusiasm. A safer principle is to permit Jev only where the output space is bounded, the evidence state is defined, the cost of error is understood and a downstream control can verify or contain the consequence.</p><p>A Jev-like model is well aligned with exception categorisation, workflow routing, evidence sufficiency, review priority, regime selection and agent tool choice. It is poorly suited to replacing exact accounting, legal thresholds, inventory conservation, payment arithmetic, integer constraints or causal claims.</p><h3>The Hybrid Responsibility Map</h3><ul><li>Reconstruct latent demand → Statistical or machine-learning model</li><li>Choose an analytical regime → Typed judgment model</li><li>Calculate quantity and cost → Deterministic code</li><li>Optimise a feasible action → Operations research</li><li>Explain the result to a user → Grounded language model or template</li><li>Approve a consequential action → Policy gate or accountable human</li></ul><p>Consider a general inventory-rebalancing project. The legacy system predicts near-term demand by item and location, then proposes movements from excess locations to constrained ones. The new architecture does not discard that work. It wraps it in a richer control loop.</p><ol><li>Define the decision, not merely the model. The decision is not “forecast demand.” It is “which feasible movement, if any, should be executed now, given uncertain demand, cost, timing, capacity and evidence quality?”</li><li>Build trusted analytical state. Data engineering reconciles movements and balances. Statistical models reconstruct censored demand and estimate future distributions. Execution models estimate whether an approved movement is likely to arrive within the useful window.</li><li>Ask bounded judgment questions. A System One layer may classify each candidate as auto-eligible, review-required or reject; score review priority; identify the primary ambiguity; and estimate whether the evidence is sufficient.</li><li>Calculate consequences deterministically. Code computes expected margin, transport cost, handling cost and the cost of delay. An optimisation engine selects quantities under inventory, capacity and policy constraints.</li><li>Govern the handoff. High-confidence, low-consequence cases may move automatically. Medium-confidence cases enter a ranked queue with evidence. High-consequence or unstable cases require accountable approval.</li><li>Learn from the complete chain. The system records the proposed route, confidence, approval, override, actual execution and outcome. It can then distinguish model error from workflow failure and execution failure.</li></ol><p>A deployed decision system should answer not only “Was the prediction accurate?” but also “Did the right action occur, at the right time, for the right reason, and did it improve the outcome?”</p><h3>Lower decision latency</h3><p>Typed outputs can be consumed directly by software, reducing the need to parse free-form prose. The actual gain must be benchmarked on the target workflow.</p><h3>Higher operational coverage</h3><p>A fast judgment layer can triage the long tail of cases that rules leave unresolved, while escalating uncertainty.</p><h3>Better separation of risk</h3><p>Soft judgments remain probabilistic, while money, policy and constraints remain deterministic.</p><h3>More useful confidence</h3><p>Calibration can support thresholds for automation, review and abstention, provided it is tested on in-domain data.</p><h3>Stronger learning evidence</h3><p>Recording judgment, control, human override and outcome creates a richer evidence chain than model accuracy alone.</p><h3>More composable products</h3><p>Prediction, judgment, optimisation, explanation and approval become replaceable modules rather than one opaque model.</p><p>Value should never be claimed from architecture alone. The hybrid earns its value only when it improves a measurable decision outcome relative to a credible baseline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GzcFVlfIg37MnRhy" /></figure><p>Figure 4. Trust is earned progressively, from schema validity through real-world outcome evidence.</p><p>A model that emits a valid type can still make a poor decision. A well-calibrated population can still contain dangerous individual errors. A fast model can accelerate the wrong workflow. Validation must therefore move beyond a single aggregate metric.</p><h3>The Seven-Gate Validation Ladder</h3><ol><li>Contract: Does every response conform to the permitted schema?</li><li>Discrimination: Can the model separate the relevant classes or priorities?</li><li>Calibration: Do predicted probabilities correspond to observed frequencies by segment?</li><li>Stability: Do equivalent states receive materially consistent judgments?</li><li>Robustness: How do ordering, framing, missing fields and adversarial text affect results?</li><li>Decision value: Does the judgment improve utility after costs and constraints?</li><li>Outcome evidence: Does the complete workflow create incremental value versus a credible counterfactual?</li></ol><p>Passing an early gate does not imply passing the next. Type validity is not truth. Calibration is not permission. Speed is not value.</p><p>The first production-adjacent experiment should run in shadow mode. The legacy policy continues to control action while the hybrid records what it would have done. This creates paired evidence without prematurely transferring authority.</p><ul><li>Baseline A: Existing business rules.</li><li>Baseline B: A conventional supervised classifier or scoring model.</li><li>Baseline C: A general-purpose language model constrained to structured output.</li><li>Candidate: Jev as the typed judgment layer.</li></ul><p>Evaluate predictive quality, calibration, consistency, latency and cost, decision utility, and subgroup robustness. Report distributions and failure slices rather than only averages.</p><p>Automation thresholds should be chosen through expected loss, not confidence aesthetics. If the cost of a false approval exceeds the cost of review, the threshold for automation should rise. If delay destroys value, excessive abstention also has a cost.</p><ul><li>ACT when confidence and consequence are acceptable.</li><li>REVIEW when evidence is useful but insufficient.</li><li>BLOCK OR ABSTAIN when required evidence is missing, adversarial risk is high or the action exceeds delegated authority.</li></ul><p>The pilot ends not when the model looks impressive, but when the organisation can specify where the system is reliable, where it is not, how failure is contained and whether the end-to-end decision creates incremental value.</p><p>Classical data science often treats deployment as the final stage of a modelling lifecycle. The hybrid view treats modelling as one component of a decision institution. The unit of analysis changes from the model to the governed loop.</p><h3>Epistemology becomes operational</h3><p>Uncertainty is no longer a chart beside the prediction. It determines whether the system acts, escalates or abstains.</p><h3>Architecture becomes part of causal value</h3><p>A superior prediction cannot create value if the action is infeasible, late or ignored. Workflow design mediates the effect of analytics on outcomes.</p><h3>Human oversight becomes selective</h3><p>The goal is neither a human in every loop nor a human out of the loop. It is human authority concentrated where uncertainty, consequence or novelty exceeds the machine’s mandate.</p><h3>Governance becomes executable</h3><p>Policy is represented through thresholds, allowed actions, approval rights, logging and rollback, rather than a document attached after deployment.</p><p>The most important intellectual contribution of System One models may therefore be less about speed than about forcing system designers to distinguish judgment from generation, and uncertainty from permission.</p><p>A probability may inform authority. It must never impersonate authority.</p><ol><li>Frame: Name the business decision, owner, action set, horizon, loss function and non-negotiable constraints.</li><li>Reconstruct: Build the trusted state. Correct target leakage, censoring, timing and entity-level reconciliation.</li><li>Model: Use data science for estimation, forecasting, scoring and uncertainty. Preserve a simple baseline.</li><li>Type the judgment: Define bounded choices, ordered scores and yes-or-no probabilities. Include abstention or evidence insufficiency.</li><li>Control: Keep arithmetic, hard policy, conservation rules, optimisation and transaction execution in deterministic systems.</li><li>Explain: Use templates or a grounded language model to make evidence, constraints and alternatives understandable.</li><li>Gate: Map confidence and consequence to automate, review, block or request more evidence.</li><li>Observe: Log source state, version, judgment, confidence, rule result, override, action and outcome.</li><li>Prove: Compare against the baseline using shadow tests, controlled rollout and counterfactual evaluation.</li><li>Evolve: Recalibrate thresholds, retrain models and revise schemas when drift or new failure modes appear.</li></ol><p>Jev is new enough that strong claims deserve disciplined skepticism. Its vendor positioning is compelling: fast, constrained probabilistic decisions rather than generated prose. But architecture should not be built around marketing numbers, and type safety should not be confused with truth, calibration with correctness, or confidence with permission.</p><p>The durable idea is broader than any single model. Enterprise AI needs a missing middle between prediction and action: a machine-native judgment layer that is bounded enough for software, uncertain enough to be honest and governed enough to coexist with deterministic control.</p><p>The old data-science project asked: What is likely to happen?</p><p>The new hybrid asks a richer sequence:</p><ul><li>What is the trusted state?</li><li>What is likely to happen?</li><li>What bounded judgment does the state support?</li><li>What action is feasible and economically rational?</li><li>Who or what is authorised to execute it?</li><li>What evidence will prove whether the decision worked?</li></ul><p>That is not the end of data science. It is data science growing into decision architecture.</p><p>The transition is from building models that know, to building systems that know when, how and whether to act.</p><p>TypeSafe AI. “System One.” <a href="https://docs.typesafe.ai/concepts/system-one">https://docs.typesafe.ai/concepts/system-one</a></p><p>TypeSafe AI. “Introducing System One Models &amp; Jev.” 15 September 2026. <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev">https://typesafe.ai/blog/introducing-system-one-models-and-jev</a></p><p>The Register. “TypeSafe AI debuts model for machines that plays Doom.” 16 September 2026. <a href="https://www.theregister.com/ai-and-ml/2026/09/16/typesafe-ai-debuts-model-for-machines-that-plays-doom/5296711">https://www.theregister.com/ai-and-ml/2026/09/16/typesafe-ai-debuts-model-for-machines-that-plays-doom/5296711</a></p><p>VentureBeat. “Companies are putting Jev in charge of AI agent decisions.” 21 September 2026. <a href="https://venturebeat.com/security/companies-are-putting-jev-in-charge-of-ai-agent-decisions-and-prompt-injection-can-influence-the-verdict">https://venturebeat.com/security/companies-are-putting-jev-in-charge-of-ai-agent-decisions-and-prompt-injection-can-influence-the-verdict</a></p><p>#ArtificialIntelligence #DataScience #JevAI #SystemOneAI #MachineLearning #DecisionIntelligence #EnterpriseAI #AIGovernance #ResponsibleAI #HybridAI #DecisionScience #AppliedAI #AIArchitecture #MLOps #ModelValidation #AIValidation #ExplainableAI #HumanInTheLoop #CausalInference #PredictiveAnalytics #Optimization #DigitalTransformation #AIAutomation #FutureOfWork #AILeadership</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c0241a7ccf7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Riskiest AI Arrived In A Renewal]]></title>
            <link>https://medium.com/@eddie.hc.tsui/your-riskiest-ai-arrived-in-a-renewal-4c6189ba0fdd?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c6189ba0fdd</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-risk]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 00:51:20 GMT</pubDate>
            <atom:updated>2026-09-21T00:51:20.456Z</atom:updated>
            <content:encoded><![CDATA[<h3>You did not build it. You are still accountable for it. Part V on governing the AI that comes through procurement, not engineering.</h3><p>Eddie Tsui · AI governance and decision assurance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yxTfIyCs7npKzphy" /></figure><p>Four ways AI gets into a company. Your control plane was designed for one of them.</p><p>The first four parts of this series were written for a company that builds. Part I argued that a system has to earn decision authority with evidence rather than inherit it because it demos well. Part II said evidence is inert unless something in the call path can refuse. Part III said the human at the end of that path is not a control unless they have time, information, independence, authority and a consequence. Part IV said the unglamorous habits already exist in public repositories, so stop rewriting them in a workshop.</p><p>All four share an assumption I did not examine closely enough: that there is a call path you own, and that you can put something in it.</p><p>For most of the AI now operating inside most organisations, there is not. It arrived through a renewal, a default-on setting, a feature release note, or an acquisition. Nobody wrote a prompt. Nobody registered a use case. The gateway cannot sit in front of it, the evaluation suite cannot run against it, and the model version is not a thing you are told.</p><p>You are still accountable for it. That is the whole problem, and it does not have an engineering answer.</p><p>1. Your control plane covers the minority case</p><p>It is worth being precise about what you can and cannot reach, because the honest version of this table is uncomfortable and most governance decks quietly omit the third column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UPZVXE5NR3FCiAqK" /></figure><p>Built, called, embedded. Only the first column looks like the architecture in Parts I to IV.</p><p>When you build in house, everything the series described applies. When you call an external model through your own gateway, most of it still applies — you pin the prompt, you attach an identity, you refuse what needs refusing, and you accept that the model itself is opaque and that its version may move underneath you.</p><p>When the AI is embedded inside a product you bought, none of it applies. There is no call you make. The prompt is theirs, the retrieval is theirs, the guardrails are theirs, the logs are a feature of their console, and the model can change on their release schedule without a line item on your change calendar.</p><p>The instinctive response is to say the vendor is responsible. That is a reasonable allocation of engineering work and a poor allocation of accountability, because the regulator, the customer and the board will all arrive at your door. The decision was yours. The exposure is yours. What you outsourced was the mechanism, not the answer to why.</p><p>2. AI does not always arrive as a project</p><p>Every organisation I have seen build an AI register built it around projects, because a project is the thing governance knows how to attach to. It has a sponsor, a budget line and a go-live. It also accounts for a shrinking share of the AI actually in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ygksJVklJ7BAKH5H" /></figure><p>Six entry routes, none of which produce a project you could have governed.</p><p>The awkward one is the second box. A renewal with no price change and no new signature is the single most common way a material AI capability enters a company, because updated terms are accepted by continuation and nobody routes a renewal to the people who would have asked the question.</p><p>The remedy is not a better policy. It is changing what triggers the register. Stop sourcing it from the project pipeline and start sourcing it from three places that already exist: the accounts payable list of software vendors, the renewal calendar, and the change notices your vendors already send and nobody reads.</p><p>A register built from projects will find the systems you authorised. A register built from invoices will find the ones you did not.</p><p>Expect the first pass to be embarrassing, and plan for that politically. The number is always larger than the sponsor was told, and the right reaction to that is a tiering exercise rather than a panic, which is the next section.</p><p>3. Procurement is the only chokepoint you have left</p><p>Part II’s central move was to route every model call through one place, because a control that is not in the path is advice. The same logic holds here, and it produces an unglamorous conclusion: for bought AI, the one place everything passes through is procurement. Not the architecture review. Not the AI council. The contract, and the renewal.</p><p>Which means the governance artefact that matters is not a principles document. It is a clause library, and about half of what usually sits in one is decoration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Jp1JIOtWEnmv-To" /></figure><p>The test is the same one from Part II: could this sentence ever be failed, and what happens when it is?</p><p>A clause is a control when it names an event that must happen, a party who must do it, and a consequence if it does not. “Vendor maintains a responsible AI programme” fails all three. “Vendor shall give thirty days’ written notice before any change to the underlying model serving the Service” passes all three, and it is the clause that would have saved you in section 6.</p><p>Two practical notes, because the theory here is easy and the negotiation is not.</p><p>First, the leverage is almost entirely at first signature and almost zero at incident time. A clause you request during procurement costs a week of legal back-and-forth. The same clause requested after a problem is a renegotiation you will lose. Front-load it, and accept that some of your library will be struck out.</p><p>Second, make the clauses the default in your template, so that removing one is a recorded decision with a named approver rather than an omission nobody notices. This is the same idea as Part III’s decision rights, applied to paper: the ability to waive should be narrow and named.</p><p>4. Proportionate diligence, or nobody reads the answers</p><p>The standard failure in vendor AI governance is not laxity. It is a 180-question assessment sent to every vendor, completed by their pre-sales team from a template, filed, and never read by anyone with the expertise to disagree with it. That is Part III’s approval theatre in a spreadsheet: an artefact that looks like scrutiny and contains none.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hFQfugILoMNmoBRm" /></figure><p>Three tiers, sorted by what cannot be undone rather than by contract value.</p><p>Tier by reversibility and data class, not by spend. A cheap tool that can send a customer an email is more dangerous than an expensive one that drafts internal summaries, and procurement’s instinct to scale scrutiny with price gets this exactly backwards.</p><p>For low-risk tools, four fields and a named owner is a complete and honest control. Say so in the policy, so that teams are not tempted to skip the register entirely because the register is exhausting.</p><p>Reserve the real work for the top tier, and make it specific rather than voluminous: your own test cases run before rollout, a version position, an incident clock with a number in it, and an exit plan. Four things you will actually check beat forty you will not.</p><p>5. Four questions that actually discriminate</p><p>Long assessments do not separate serious vendors from confident ones, because both complete them. Four questions do, and each one is answerable in a sentence by a vendor who knows their own system.</p><ul><li>Which model and version is serving this feature today? A vendor who cannot say has no change control of their own, which tells you everything about the notice you will get.</li><li>When did you last change it, and how was I informed? Ask for the last instance, not the policy. The gap between the two is the answer.</li><li>If I disable this feature tomorrow, what happens to the content it has already processed, and to the history? This is the exit test in miniature, asked early enough to matter.</li><li>Can you show me an export of the logs, now, in the format I would keep? Not a screenshot of the console. The export, with the fields visible.</li></ul><p>What you are testing is not compliance posture. It is whether the person in front of you has access to the answer, because a vendor whose own account team cannot reach their model version will not reach it during your incident either.</p><p>Ask them in the room, verbally, and note which ones came back in writing later. That distribution is more informative than the completed questionnaire, and it takes twenty minutes.</p><p>6. Same product name, same price, different model</p><p>Here is the failure mode that is unique to bought AI, and the one conventional third-party risk management has no concept for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uzqvL9GuPHAoucDV" /></figure><p>Nothing in your change management fired, because from your side nothing changed.</p><p>Your vendor improves their product. That is what you pay them for. But a model swap behind an unchanged endpoint is a production change to your system, delivered without a change request, a rollback path or a version number. Your own release calendar was quiet that week.</p><p>The consequence that hurts most is not the immediate output shift. It is the one in the last panel: the record you kept of a decision made last March cannot be reproduced, because the system that produced it no longer exists anywhere. Part I asked whether a decision could be replayed. Part III asked whether someone who was not there could reconstruct why it was reasonable. A vendor model with no version identifier fails both, permanently, and no amount of your own logging fixes it.</p><p>There are exactly two responses, and you need both.</p><ul><li>Contractual: notice before change, a version identifier returned with the output where the interface allows it, and a defined window in which the prior behaviour is still available. Vendors resist the third and often concede the first.</li><li>Operational: a small golden set — twenty cases is plenty — for every vendor feature you genuinely depend on, run on a schedule against the live product. It costs an afternoon to build. It is the only independent signal you will ever have that the product changed.</li></ul><p>A vendor feature with no change notice and no golden set is an unversioned production dependency you have chosen not to monitor.</p><p>7. Shared responsibility, drawn honestly</p><p>Every vendor will show you a shared responsibility diagram, and most of them are accurate about the top and the bottom and silent about the middle. Draw your own, per vendor, and make room for a fourth category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_PZvqJJUDrPhx3T3" /></figure><p>Vendor, you, both — and the row that matters, which is neither.</p><p>The two rows marked neither are where the incidents come from, and they are neither party’s negligence. They are the seam.</p><p>Retrieval is the clearest example. The vendor secures their model; you manage your permissions; nobody owns the question of what happens when the assistant is asked a question whose answer sits in a document the requester should not have seen but technically can. The permissions were already wrong before AI arrived. The assistant simply industrialised the consequence, by making badly-scoped access searchable in natural language for the first time. Part II flagged retrieval integrity and supply chain as the empty cells in the coverage matrix. This is where those empty cells actually land.</p><p>The second row is evidence. Vendor logs are a product feature: retained according to their tier, exportable in their format, and ending when the contract does. That is not a record. If the only copy of why a decision was reasonable lives in a console you might stop paying for, you are renting your own memory, and the retention period was set by someone optimising storage cost rather than your assurance requirement.</p><p>8. When the vendor says no</p><p>Sometimes you ask for notice, export and a version position, and the answer is no — because you are small, or because the product is standardised, or because the feature is bundled inside something you cannot leave. This is the situation the frameworks are least useful about, and it is extremely common.</p><p>There are four honest moves, and only four.</p><ul><li>Narrow the use case. Turn the feature off for the population where the action is irreversible or the data is sensitive, and keep it where it is not. A per-group toggle is a real control and is usually available.</li><li>Compensate around it. You cannot instrument their model, but you can instrument what enters and what leaves: what data the connector can reach, who receives the output, and whether a human step sits between the output and an irreversible action.</li><li>Accept and record. Name the residual risk, name the executive who accepts it, and give the acceptance an expiry date. Part III put this in the board’s small Decides pile. It belongs there, not absorbed silently by whoever owns the ticket.</li><li>Decline. Sometimes the answer is that this capability is not available to you on acceptable terms this year. That is a legitimate outcome and it is almost never offered as one.</li></ul><p>What you should not do is write the requirement into a policy the vendor has already refused, and file it. That produces a document which asserts a control you know does not exist, which is worse than having no document, because someone will later rely on it.</p><p>9. The ending nobody designs</p><p>Ask any vendor relationship one question and you will learn more than a questionnaire would tell you: if we left in thirty days, what could we not reconstruct?</p><p>For ordinary software the answer is manageable — you export the data and rebuild the workflow elsewhere. For AI features the answer is usually worse than people expect, because three things are entangled in the product rather than in your tenancy: the interaction history that constitutes your audit trail, the tuning or grounding built up over two years of use, and the output format a dozen downstream processes quietly depend on.</p><p>Run one exit rehearsal on your highest-tier vendor, on paper, in an afternoon. It is the cheapest diligence available and it consistently finds the dependency nobody had named. It also produces the clause list for your next renewal, which is the only moment you will have leverage again.</p><p>10. Sixty days, and none of it needs engineering capacity</p><p>If you want to act on this rather than file it, the sequence matters more than the ambition — and unlike Part II’s ninety-day plan, none of this depends on a platform team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JozR53liuzqmnZjm" /></figure><p>Register, tier, clause, golden set, owner, rehearsal. In that order.</p><p>The order is deliberate. Building the register first is what gives you the number that unlocks the rest; policies written before the register are always calibrated to the AI leadership believes exists. Tiering comes second so the register does not collapse under its own diligence. The clause library comes third because it only pays out at renewal, and renewals are a queue you have to get ahead of.</p><p>And the exit rehearsal comes last because by then you know which vendor to run it on.</p><p>11. What this costs you to admit</p><p>I want to end where Part II and Part III both ended, which is at the edge of what the evidence supports.</p><p>For AI you build, the limit was that behavioural testing cannot establish the dispositional properties governance frameworks increasingly ask you to assert. For AI you buy, there is a second limit stacked on top of the first: you cannot run the tests at all. Your assurance is a vendor’s description of their own controls, a certification that tells you a management system exists rather than that it is set correctly, and whatever your golden set can see from the outside.</p><p>That is a materially weaker evidence base than the one Parts I to IV were arguing about, and pretending otherwise in a governance file is the specific dishonesty this series has been trying to avoid.</p><p>Which leaves the same two levers Part II arrived at, and they are the only two that survive contact with a vendor who will not negotiate. Narrow what the system is permitted to do. Keep what it does recoverable. Both are entirely within your control, neither requires the vendor’s cooperation, and both work when your evidence runs out.</p><p>You can outsource the model, the infrastructure, the guardrails and the roadmap. The decision stays yours, and so does the sentence that begins “we decided to”.</p><p>Parts I to IV described a company governing what it builds. Most organisations are now governing what they buy, and doing it with instruments designed for a different problem. The register, the tier, the clause and the golden set are not as satisfying as an architecture. They are what is actually in the path.</p><p>#AIGovernance #ResponsibleAI #AIRisk #AICompliance #AIAccountability #AIAssurance #VendorRisk #ThirdPartyRisk #Procurement #SaaSGovernance #ShadowAI #SupplyChainRisk #EnterpriseAI #AIStrategy #AILeadership #CIO #RiskManagement #DigitalTransformation#ModelGovernance #MLOps #DataGovernance #AIRegulation #EUAIAct #TechnologyRisk</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c6189ba0fdd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Borrow The Practice Do Not Rewrite-It]]></title>
            <link>https://medium.com/@eddie.hc.tsui/borrow-the-practice-do-not-rewrite-it-df6065d0fcd4?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/df6065d0fcd4</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 12:20:05 GMT</pubDate>
            <atom:updated>2026-09-19T12:21:38.998Z</atom:updated>
            <content:encoded><![CDATA[<p>Part IV — what public projects already run, and which of those habits your stack is still missing.</p><p>Eddie Tsui · AI governance and decision assurance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eCnWddw3Od1jFUFY" /></figure><p>The practice already has a repository. The slide deck is the copy that goes stale.</p><p>The first three parts of this series were an architecture. Part I said a system has to earn decision authority with evidence, not with a demo. Part II said that evidence is useless unless something in the path can refuse. Part III said the human you put at the end of that path is not a control unless they have time, information, independence, authority, and a consequence.</p><p>People who read them asked a fair next question. Fine. Has anyone already built the unglamorous bits?</p><p>Yes. Not as a single product, and not as a framework you can certify against and forget. As public repositories, maintained by people who got tired of writing the same control into a policy and watching it fail. Singapore’s AI Verify Foundation. Microsoft’s red-team toolkit. NVIDIA’s vulnerability scanner. The UK AI Security Institute’s evaluation framework. Stanford’s multi-metric harness. Meta’s separate safety model. A PII scanner, a fairness library, a signing tool, a trace schema, an incident database, a drift monitor, a proxy that treats an API key as a policy.</p><p>This piece is what I think is worth stealing from them, and what I think you should leave on the shelf. It is less of a design than the earlier parts and more of a field note. The thesis is short.</p><p>If a practice only exists in your policy, the next team will rewrite it, worse. If it exists in a repository, they might run it.</p><p>1. The series built the architecture. It did not name the habits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QV8ieDGSbO_w2ypn" /></figure><p>An architecture can still be faked. A habit with a failing test is harder to fake.</p><p>Nothing in the first three parts is withdrawn. A company that has no gateway, no named decision owner, and no record of what the reviewer saw is not ready for this article. They are ready for the previous ones.</p><p>What those parts left open is the layer underneath the diagram. Which test counts. Who wrote it. What an attack looks like after the Slack thread is gone. Which file has to exist before a model is allowed to move. Whose incidents you are willing to learn from.</p><p>Public projects answered those questions because they had to ship something other people could run. That constraint is the useful part. A practice that cannot be cloned is usually a practice that cannot be audited either.</p><p>2. Four habits. Do not collect twelve logos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SFNOwk7gS6R1pOoi" /></figure><p>Install the habit. The repository is the example, not the destination.</p><p>I grouped the field into four, because a list of tools is how this topic usually dies.</p><p>Judge apart. The system that acts is not the system that decides the act was acceptable. , garak’s detectors, PyRIT’s scorers, and <a href="https://www.google.com/url?q=https://github.com/meta-llama/PurpleLlama&amp;sa=D&amp;source=editors&amp;ust=1789823929577138&amp;usg=AOvVaw1obPHPSsHC0KomwpWqjMmd">Llama Guard</a> all refuse to let the actor mark its own work. They refuse in different ways, which is the point of the later sections.</p><p>Ship an artifact. A release is a file someone else can check. Model cards, a , a <a href="https://www.google.com/url?q=https://cyclonedx.org/capabilities/mlbom/&amp;sa=D&amp;source=editors&amp;ust=1789823929577550&amp;usg=AOvVaw284-7tN5iT6uMLXNUqlSsH">machine-learning bill of materials</a> , a trace in the <a href="https://www.google.com/url?q=https://github.com/sigstore/model-transparency&amp;sa=D&amp;source=editors&amp;ust=1789823929577627&amp;usg=AOvVaw3D9UM0guG8gJ2emDMArcwd">Sigstore signature</a> . If the evidence lives only in a vendor console, you are renting your own memory. <a href="https://www.google.com/url?q=https://github.com/open-telemetry/semantic-conventions-genai&amp;sa=D&amp;source=editors&amp;ust=1789823929577724&amp;usg=AOvVaw2gaHZZ2BR7epy6gonEV_wV">OpenTelemetry GenAI conventions</a></p><p>Remember in public. Your postmortem is not the world’s. The fails where you have not yet. <a href="https://www.google.com/url?q=https://incidentdatabase.ai/&amp;sa=D&amp;source=editors&amp;ust=1789823929577957&amp;usg=AOvVaw0T3WZFPQpxJLieZKO2QQ7V">AI Incident Database</a> watches the distribution after you ship. make spend and model access into something you can revoke on a Tuesday. <a href="https://www.google.com/url?q=https://docs.litellm.ai/docs/proxy/virtual_keys&amp;sa=D&amp;source=editors&amp;ust=1789823929578150&amp;usg=AOvVaw0YSJe6xZX72yiaDW5LerWx">LiteLLM’s virtual keys</a></p><p>A dashboard that shows all four, and cannot refuse anything, is Part II’s failure mode with better branding. The habit includes the moment something fails.</p><p>3. Run a test kit you did not author</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gZRSl1Un_9g5278v" /></figure><p>Your rubric is a conflict of interest. An external cookbook is the cheapest way out of it.</p><p>Read the caveat in their own introduction, because it is the most grown-up sentence in this whole genre. The toolkit does not define ethical standards. A system that passes is not certified safe, unbiased, or finished. That is not modesty. It is the correct scope of a test harness. Part I’s argument was that evidence is not permission. They wrote the same thing into the readme.</p><p>The practice, stripped of the branding, is three rules.</p><p>Pin the recipe. If the team can edit the threshold in the week of the launch, you do not have a test. You have a negotiation that happens to be stored in git.</p><p>Run at least one suite you did not write. It will miss your domain. That is acceptable. A test that only knows your happy path is not independence.</p><p>Do not promote the result into a certificate. “Passed AI Verify” is a sentence your sales team will want. It is not a sentence the project is willing to support, and you should not be more careless than they are.</p><p>4. An attack that is not versioned will return</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-V_yZFpPTAPohlXd" /></figure><p>A jailbreak in a chat thread is an anecdote. A probe with a name is a control.</p><p>The design detail to copy is the split between probe and detector . The thing that generates the nasty input is not the thing that decides the failure occurred. If you let the probe grade itself, you have rebuilt the actor-and-judge problem inside the test.</p><p>A single malicious prompt is a demo. Real misuse is a conversation that wanders until a guardrail gets bored. If your red team cannot remember what it tried last month, it will rediscover the same trick and call it new.</p><p>Put the probes in the release gate from Part II. A model bump that newly fails a named probe is a regression. Treat it like a failing unit test, not like a fascinating research note.</p><p>5. Evaluate the trajectory, not the final sentence</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*feZaD06OEQE5Aoen" /></figure><p>If production can call a tool, the evaluation has to call a tool. Otherwise you tested a different system.</p><p>Most “LLM evals” I see are a prompt in and a paragraph out, scored by another model that has been told to be strict. That tests a chatbot. The system in production is an operator. It reads a file, calls a refund API, writes to a ticket, and only then produces a sentence. If the eval never sees the tool call, you have no evidence about the thing that can hurt someone.</p><p>This is Part I’s evidence requirement, applied to agents, and it is also Part II’s tool authorisation, moved left into the test. Approval that exists only in production is a control you have never watched fail. Approval inside the eval is how you find out the policy is wrong before a customer does.</p><p>Keep the log. Two model versions on the same task should be diffable. “It felt better” is not a release note.</p><p>6. A leaderboard with one column is a sales tool</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V__LLBDTU46IxuQP" /></figure><p>The ranks in the picture are made up. The pattern is not. HELM exists because one number hides the rest.</p><p>, from Stanford’s Center for Research on Foundation Models, evaluates foundation models across scenarios and across metrics that are not accuracy: efficiency, bias, toxicity, and others. The prompts and the responses are inspectable. The is the argument; the repository is so you can reproduce a leaderboard instead of trusting a screenshot of one.</p><p>I am not quoting a ranking. Rankings move, and quoting last year’s winner is how this kind of article goes rotten. The design claim is stable. A model that leads on the metric your vendor put in the subject line is often not the model you would pick if you also looked at harm, cost, and the cases you actually run.</p><p>So the release note gets a scorecard. Accuracy, a fairness slice, a safety probe from section 4, and the cost of a real workload. If you can only bear to publish one column, say which column you refused to publish, and why. Hiding it is the sales move.</p><p>HELM will not know your refund policy. Do not adopt it as a religion. Adopt the refusal to collapse a release into a single green tick.</p><p>7. The model that acts cannot grade the act</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kTZrpH9NLwAOQSkX" /></figure><p>A safety sentence in the same prompt as the task is the actor interviewing itself. It will be generous.</p><p>garak already split probe from detector. PyRIT already split target from scorer. Presidio, next, does not use a model at all for the cases where a model is the wrong instrument. Three projects, one habit.</p><p>The gate from Part II is what makes the judge real. A score that is logged and never able to deny the request is a comment. Wire the judge’s output to a rule: deny, redact, or escalate, and record which judge, which version, which rule. When the judge is wrong, you change the judge. You do not “remind” the actor.</p><p>8. Some failures are a pattern, not a judgement</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dqVOA2WYhgzC2FVp" /></figure><p>If the failure is a card number, do not ask a language model to notice. If the claim is fairness, do not show me an average.</p><p>The alternative is a line in the system prompt. It works until the model is being helpful, or until the identifier is in a table, or until someone asks for a summary of a document that contains one. Part II already said a prompt is guidance. Presidio is what enforcement looks like when the rule is a shape.</p><p>Do not use it as a conscience. It will miss novel personal data and it will flag things that are not. Put a person on the appeals, and keep the recognizer on the path. The point is that the default for a pattern is a pattern.</p><p>Fairness definitions disagree with each other. Fairlearn will not pick your definition. You still have to. What it will not let you do, if you use it honestly, is report one number and go to lunch.</p><p>9. No card, no promotion</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AIP-v13vZP6UzAna" /></figure><p>The card is for a human. The bill of materials is for every tool that has to know what you shipped.</p><p>Model cards are old now, which is why they are trustworthy. The argument is the : intended use, out-of-scope use, evaluation, limitations, the data you are willing to describe. made that the front page of a model rather than an appendix nobody writes. The fields have not needed reinventing. <a href="https://www.google.com/url?q=https://huggingface.co/docs/hub/model-cards&amp;sa=D&amp;source=editors&amp;ust=1789823929588475&amp;usg=AOvVaw0Hob86_wCQuHUmn0CYCJFK">Hugging Face’s model-card convention</a></p><p>The promotion rule is simple. No card, no move to production. A card written by marketing, with no out-of-scope section, does not count. The out-of-scope section is the point. It is where you admit the neighbouring use case that looks the same and is not allowed.</p><p>A card is not an inventory. <a href="https://www.google.com/url?q=https://cyclonedx.org/capabilities/mlbom/&amp;sa=D&amp;source=editors&amp;ust=1789823929588990&amp;usg=AOvVaw0tFfOoWhvOD56ihqoqwcj8">CycloneDX’s ML-BOM</a> is the attempt to make the supply chain machine-readable: the model, the datasets, the parameters, the software around them, and a place for the card itself. A base model you did not train is still in your supply chain. A file that changed under the same name is a different component.</p><p>If it is not in the bill of materials, treat it as shadow AI, even when an engineer can explain it beautifully in a meeting. Part III’s adoption argument was that the governed path has to be the fast path. The BOM is how you notice the other path.</p><p>10. An unsigned model is an unsigned binary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xpwOK9bh6E9r3dhT" /></figure><p>Sign the bytes, publish the event, and refuse to load what does not verify.</p><p><a href="https://www.google.com/url?q=https://github.com/sigstore/model-transparency&amp;sa=D&amp;source=editors&amp;ust=1789823929589785&amp;usg=AOvVaw1vfULWZjrojIdr4haVksks">model-transparency</a> , from Sigstore with OpenSSF and others, signs models the way you should already sign code. The is the one to read. A short-lived certificate is bound to a workload identity or a person. The signature covers a manifest of hashed files, not a filename. The signature goes to a transparency log, so a signing event can be noticed. Verification compares the manifest to the files you actually have. If they differ, the model was changed.</p><p>The practice is the last step, and it is the one teams skip. A signature nobody checks is a sticker. Verify in the pipeline, or in the place the model is loaded, and fail closed. “We sign our models” is a weaker sentence than “an unsigned or mismatched model does not start.”</p><p>This does not make the model good. It makes the model the one you think it is. Part I’s replay requirement dies immediately if the weights can be swapped under you. Signing is how replay stays about the same artifact.</p><p>A long-lived key in a shared folder is not this practice. The log is not optional decoration. It is how you catch the insider who can sign.</p><p>11. If you cannot export the trace, you do not own it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XEcjZIc61iT0aZKO" /></figure><p>Vendor logs are a product. A shared schema is a language. You need the language.</p><p>The contrast is a console that says the chat completed, the user was Dave, the model was “latest,” and the status was ok. That is enough for a status page. It is not enough for Part III’s test: could someone who was not there reconstruct the decision without asking anyone who was? “Latest” will not answer them in eighteen months.</p><p>Own the export. If the only copy of the trace is in a product you might leave, you do not have a record. You have a subscription. Cost belongs in the same trace. Tokens and latency are evidence, not just a finance problem. A runaway agent is often visible in usage before it is visible in a complaint.</p><p>12. Read the failure before it is yours</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CrrSdMlvawmVdQKT" /></figure><p>A landscape slide is not a control. An incident that changed a gate is.</p><p>The <a href="https://www.google.com/url?q=https://incidentdatabase.ai/&amp;sa=D&amp;source=editors&amp;ust=1789823929592487&amp;usg=AOvVaw3eVMpJfFEi3mAkGdjl5Q3N">AI Incident Database</a> , maintained in public at , collects cases where deployed AI systems harmed people. It is not a benchmark and it is not a ranking of vendors. It is a memory the industry did not previously have in one place. <a href="https://www.google.com/url?q=https://github.com/responsible-ai-collaborative/aiid&amp;sa=D&amp;source=editors&amp;ust=1789823929592603&amp;usg=AOvVaw3uAKPuVJS5hqbi76p6UftK">responsible-ai-collaborative/aiid</a></p><p>Part III described your own incident process: grade by what cannot be undone, then ask whether the control was absent or unused. That process is blind to everything that has not happened to you yet. The database is the other input.</p><p>The practice is a standing review, owned, on a calendar. For each incident in your domain, one of three outcomes, written down. It becomes a probe in the suite from section 4. It becomes a rule in the gateway. Or it is explicitly not your shape, and someone signs that. “We monitor the landscape” with none of those three is a news habit.</p><p>Do not treat the database as complete, and do not treat a horrific story as automatically your requirement. The work is the mapping. Skipping the mapping is how teams either panic at every headline or ignore all of them.</p><p>13. Watch the distribution, not the anecdote</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KlrUku92WkNKDOyA" /></figure><p>Part III’s refusal rate can look healthy while the traffic underneath it has left the building.</p><p>The picture is schematic. Input drift climbs. The refusal rate, the metric Part III told you to love, barely moves. Both can be true. Users are asking different questions. The gate is still refusing at the same rate, on a population you no longer recognise. If you only watch one chart, you will congratulate yourself through a silent change of job.</p><p>Output drift matters as much. Length, language, empty answers, toxicity, the rate at which a tool is called. A shift with no ticket is still an incident. Give the threshold an owner, the same person who owns the model, and page them. A chart without a fail line is interior design.</p><p>Do not buy a monitoring product and call it governance. Monitoring that cannot roll back, narrow the tool list, or pull the key is a weather report.</p><p>14. A key is a policy with a budget</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ku0It1OXEE2Qu2I8" /></figure><p>The personal provider key on expenses is a second front door. Close it before you write another principle.</p><p><a href="https://www.google.com/url?q=https://github.com/BerriAI/litellm&amp;sa=D&amp;source=editors&amp;ust=1789823929595206&amp;usg=AOvVaw2A8ziUvRnO8H_1ROlWYQ5x">LiteLLM’s proxy</a> is one public place this already works. <a href="https://www.google.com/url?q=https://docs.litellm.ai/docs/proxy/virtual_keys&amp;sa=D&amp;source=editors&amp;ust=1789823929595316&amp;usg=AOvVaw17I1i59hzosDHfiwPCer8y">Virtual keys</a> carry a model allow-list, a budget, a rate limit, and a team. Spend is tracked against the key. You can revoke it without begging a model provider to rotate a secret that has been copied into four laptops.</p><p>Two lessons, and the second is the one people skip.</p><p>First, unlimited is a decision. A key with no budget and no model list is a provider key with extra steps. Write the ceiling down. A runaway agent is a cost incident and sometimes a safety incident. They often start as the same graph.</p><p>Second, the caller must not choose its own identity. The LiteLLM docs are explicit that a user field supplied by the client can be spoofed, and that you can overwrite it with an identifier tied to the key. Do that. An audit log full of names the client picked is fan fiction.</p><p>This is also Part III’s adoption rule, in the only form engineers will actually feel. The governed key has to be easier to get than the shadow key. Templates, a budget that is enough to work, a quota that is not an insult. If the official path takes two weeks and a corporate card takes an afternoon, you have built a tax. People will avoid it, and they will be right.</p><p>15. Clone the habit. Leave the platform pitch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C2D3MLG2XmJD4sK3" /></figure><p>Part I evidence. Part II enforcement. Part III accountability. Part IV the habits that stop those three from being a diagram.</p><p>What I would actually do, in order, if I were starting from a team that already has the gateway from Part II and the decision owner from Part III:</p><p>Adopt now, almost unchanged. A separate judge on the path, even a boring classifier. A PII recognizer for the patterns you actually see. Virtual keys instead of provider keys, with budgets and an identity the client cannot set. One external recipe pinned as a release blocker, with the caveat that a pass is not a blessing.</p><p>Adapt, because your domain is not their demo. The attack library: take garak or PyRIT and add the probes that match your tools, then keep their detectors. Inspect, or something with the same shape, pointed at your real tools in a sandbox. Fairlearn-style slices on the groups your counsel has already named, not on a tutorial column. A model card template with an out-of-scope section someone is willing to defend. A bill of materials that includes the base model you inherited.</p><p>Sign and trace once the artifact has a reader. Model signing is wasted if nothing verifies it. OpenTelemetry traces are wasted if retention deletes them in ninety days, which is the failure Part III already named. Do these when the gateway and the audit can consume them. Not before, and not never.</p><p>Ignore the pitch that wraps all of this in a single pane and calls it governance. The pane is fine. The refusal is the product. A catalogue of scanners that cannot stop a request is a policy with charts.</p><p>None of these projects will run your company. AI Verify will not pick your risk appetite. HELM will not know your customers. Sigstore will not notice that the signed model is the wrong model for the job, only that it is the model you signed. The incident database will include stories that are not yours. That is not a reason to write your own worse version in a slide. It is a reason to keep Part I’s question in force: what, exactly, is this evidence allowed to prove?</p><p>The series was never a claim that governance can be finished. It was a claim that it can be made specific. Evidence you can fail. A gateway that can deny. A person who can refuse and will be asked why. And, now, habits that already exist in public, so the next team does not have to invent them on a deadline.</p><p>Borrow those. Do not rewrite them in a workshop.</p><p>Parts I to III are the architecture this piece assumes. The links above go to the repositories and papers, not to my summary of them. Several projects move quickly — pin the version you ran, and do not treat a pass as a certificate. If you have replaced one of these with something plainer that still fails closed, I would rather hear about that than about another platform.</p><p>#AIGovernance #ArtificialIntelligence #OpenSource #Cybersecurity #MachineLearning #RedTeaming #LLMSecurity #ResponsibleAI #MLOps #AIAgents #ModelCards #AISupplyChain #Observability #DataPrivacy #AIRisk #PromptInjection #AIEvaluation #SoftwareSupplyChain #AIAudit #MLSecOps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df6065d0fcd4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Human In-The-Loop Is Not A Control]]></title>
            <link>https://medium.com/@eddie.hc.tsui/the-human-in-the-loop-is-not-a-control-7aa1199bb44a?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/7aa1199bb44a</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 12:13:53 GMT</pubDate>
            <atom:updated>2026-09-19T12:22:02.214Z</atom:updated>
            <content:encoded><![CDATA[<p>Part III — decision rights, accountability, and oversight that actually works.</p><p>Eddie Tsui · AI governance and decision assurance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DxF9Og4ZXBumYQ9U" /></figure><p>One hundred and eighty approvals, forty-one seconds each. Somewhere in there was the one that mattered.</p><p>Part I of this series argued that an AI system should not inherit decision authority just because it produces a better-sounding answer than the model it replaces; it has to earn that authority with evidence. Part II argued that evidence is worthless without a chokepoint that can act on it, and described the control plane — gateway, policy-as-code, prompt registry, evaluation gates — that turns a written rule into something a system can actually fail.</p><p>Which is a tidy ending, and also a dodge. Because the honest follow-up question is the one this piece is about.</p><p>Does that actually help?</p><p>Mostly, in my experience, it does not. Not because people are careless, but because the human gate is almost always installed as a shape rather than as a control: a step in a workflow, an approval button, a name in a RACI chart. It satisfies the diagram. It produces a signature. And under any real load, it stops being a decision within about a month, and nobody notices, because the artefact it generates looks identical either way.</p><p>This piece is about the organisational layer — the part you cannot deploy. It is less technical than Part II and, I think, harder, because none of it can be fixed by buying something.</p><p>1. A signature is not a decision</p><p>Start with the artefact everyone trusts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K5-GJZiMDwMzKXuz" /></figure><p>The same request, two realities. The audit trail cannot tell them apart.</p><p>Two organisations run the same control: an agent proposes closing a batch of customer tickets, and a human approves before it executes. Both have the policy. Both have the button. Both produce a complete, timestamped, attributable approval record.</p><p>In the first, the reviewer sees this request as one of a hundred and eighty that day, spends forty-one seconds on it, reads a one-line summary and a confidence score, and clears it. Their refusal rate over the past year is 0.6%, and no refusal they have ever made has been welcomed by anyone.</p><p>In the second, the reviewer sees six requests that day, because the reversible ones were never routed to a person at all. They get seven minutes, the underlying tickets, and — importantly — the two cases the model itself was least certain about. They send two back. Their refusal rate is 18%, it is tracked, and nobody finds it controversial.</p><p>The first organisation has an approval process. The second has oversight. The only reliable way to tell them apart from the outside is to look at everything except the signature.</p><p>If your evidence of human oversight is that the approvals were collected, you have measured compliance with a process, not the exercise of judgement.</p><p>This matters because the signature is exactly what an auditor, a regulator, and your own board will be shown. It is the most legible artefact in the entire system and the least informative.</p><p>2. Four ways a human gate stops being a gate</p><p>The failure is predictable enough to enumerate. Each of these has a measurable tell, which is the useful part — you can check your own organisation against them this week.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fi1P8ub__-Jrx_c7" /></figure><p>None of these are failures of character. All of them are failures of design.</p><p>Automation bias is the reviewer adopting the system’s recommendation as the default and then looking for reasons to accept it rather than reasons to doubt it. This is not stupidity; it is what any sensible person does when the machine is usually right and their day contains other work. The tell is that approval rate correlates with the model’s confidence score rather than with the difficulty of the case. The fix is structural: withhold the recommendation until the reviewer has recorded their own read. It costs a few seconds and it changes what the approval means.</p><p>Approval fatigue is request volume exceeding available attention, at which point the queue gets cleared rather than read. The tell is a median review time under a minute, and approvals bunching at the end of the day. The fix is to cut the number of requests — never the time allowed per request, which is the instinctive move and the wrong one.</p><p>Information asymmetry is holding someone accountable for a judgement they do not have the inputs to make. The tell is quiet: reviewers ask no questions, because they do not know enough to have one. The fix is to ship the evidence with the request, including — especially — what the system was least sure about.</p><p>No power to refuse is the one people are least comfortable naming. If blocking costs the reviewer a difficult conversation, a slipped date and a reputation for being unhelpful, while approving costs nothing today, then approving is the rational act and you have designed a rubber stamp on purpose. The tell is a refusal rate near zero combined with nobody being able to recall a time a block was welcomed.</p><p>3. What a reviewer actually needs</p><p>Invert the failure modes and you get the conditions. I have found five, and the useful property is that they are individually cheap to write into a policy and collectively expensive to actually provide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R1dDJ5a-I8FnRY2t" /></figure><p>Remove any one and the signature still appears in the log.</p><p>Three of these get written down routinely. Time and information are usually acknowledged, even if not resourced. Independence is where it starts getting awkward — the question “does the reviewer report to the person whose delivery date a refusal would slip?” has an embarrassing answer in most organisations, and the honest version of the fix costs money.</p><p>The last two are the ones I would push hardest on.</p><p>Authority asks whether the refusal is binding, or whether it can be overturned by someone senior who was not in the room and did not see the evidence. An override path is legitimate and sometimes necessary — but if it is used routinely and without a written reason, the reviewer is not a control, they are an advisor with a logging obligation.</p><p>Consequence asks what actually happens to the reviewer who blocks correctly, and to the one who waves something through that later goes wrong. If the answer to both is “nothing”, you have not built an incentive, and every other condition on the list is decorative.</p><p>4. Who owns what</p><p>Underneath all of this sits a question most organisations have never answered explicitly: who, by name, owns this decision?</p><p>The common answer — “the AI team owns it” — is not a decision-rights model. It is a single point of failure with a budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xjarJWCQlRTBJ9d9" /></figure><p>Six roles, five rights. The column with four owners is deliberate.</p><p>The distinction that does the most work here is between the model owner and the decision owner . The model owner is engineering: they build it, tune it, operate it, and know how it behaves. The decision owner is the business line whose decision this actually is — whose customers are affected, whose risk appetite is being spent, and who would have made this call by hand last year. In most organisations nobody has been named to this role, which is precisely why the accountability conversation only happens after something has gone wrong, and why it goes badly when it does.</p><p>Two rules fall out of the grid, and both of them are the kind that sound obvious and are widely violated.</p><p>The builder does not clear the release. Not because engineers are untrustworthy, but because self-assessment under a delivery deadline is not assurance, and asking someone to be both the advocate and the judge is a structural unfairness to them as much as a risk to you.</p><p>Anyone can halt; only one role clears. Look at the “can halt in flight” column — four owners. The ability to stop a system should be distributed widely and be uncontroversial to use. The ability to start one should be narrow and named. Organisations routinely get this exactly backwards, making the stop button politically expensive and the start button an informal default.</p><p>5. Three lines of defence, honestly</p><p>Most regulated organisations already have a model for this, and there is no reason to throw it away. Three lines of defence works. It just stops working at exactly the point where understanding the system is the job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*06At62QRh2Gm7r3C" /></figure><p>The model is sound. Two of the three lines are currently unable to perform their part of it.</p><p>The fix is not a new framework. It is two hires and one engineering capability. Give the second line technical challenge capacity — at minimum one engineer who does not report into delivery and whose job is to disagree competently. Give the third line replay . Independence without competence produces a signature; competence without independence produces a self-review. You need both, in different people.</p><p>6. Oversight is a budget, and you are overdrawn</p><p>Here is the arithmetic almost nobody does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KWdXRzA4XYK-cxlX" /></figure><p>Attention is finite, non-transferable, and cannot be bought back by asking people to try harder.</p><p>Reviewer attention is a fixed resource. Somewhere between one and eight requests a day, a person reads properly, asks questions and occasionally refuses. Between nine and thirty, they skim, and still catch the genuinely odd case some of the time. Above thirty, the queue is cleared rather than reviewed, and the gate has silently become a logging step that adds latency.</p><p>A hundred and eighty requests at a genuine seven minutes each is twenty-one hours of attention per day. That queue was never going to be read. The only question that was ever open is who gets blamed for that later.</p><p>Escalate only the irreversible. Part II’s reversibility-and-blast-radius matrix is not just an authority model; it is a routing rule. Reversible, contained actions should never reach a human at all. Spending an approval on something you could undo with one command is spending it on the wrong thing.</p><p>Approve the class, not the instance. Grant a scoped standing authority once, deliberately, with monitoring attached — rather than asking the same question two hundred times and receiving two hundred reflexes. This is a real decision made once instead of a fake decision made constantly.</p><p>Batch with a sampled deep read. Approve in groups, and review a random sample in genuine depth. A 10% sample read properly detects systematic problems far better than 100% skimmed, and it is honest about what it is.</p><p>Expire rather than accumulate. A request nobody could reach inside the window should fail closed, not sit in a queue acquiring urgency until someone clears it unread at 6pm.</p><p>One caveat on standing authority, and it is not a small one: there is <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.17830&amp;sa=D&amp;source=editors&amp;ust=1789796374351563&amp;usg=AOvVaw1oWmUULNWz7LQrO1S059r7">research indicating that agents with persistent memory degrade in safety profile over time</a> even where single-task performance holds steady. An approval granted once is a judgement about a system that will not stay the same. Standing authority needs an expiry date and a re-approval trigger, or you have not reduced the oversight burden — you have deferred it and forgotten about it.</p><p>7. What a review board is actually for</p><p>Most organisations’ answer to AI accountability is a forum. The forum then receives a forty-item agenda, and the one item that mattered gets four minutes and a nod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fDfL5XViS_bjPe_g" /></figure><p>A board that reviews everything decides nothing.</p><p>The discipline is to sort every item into one of three piles, and to be ruthless about it.</p><p>Decides should be small — six to ten items a quarter. New high-risk use cases, and whether to decline them. Changes to a gate threshold, in either direction, because loosening a threshold under delivery pressure is the single most common way a control quietly dies. Exceptions to policy, always with an expiry date attached. Decommissioning something people now depend on. And accepting a named residual risk that the evidence cannot close — which, as Part II argued, is a real and unavoidable category, and belongs here rather than being absorbed silently by whoever happened to be on the ticket.</p><p>Records is everything a control already handled. Requests the gateway denied. Version promotions that passed every gate. Refusal rates by use case, with the trend. Closed incidents and whether remediation actually shipped. These should be read before the meeting, not presented in it. The board’s role is to confirm the control operated, not to re-adjudicate the outcome.</p><p>Delegates is everything that should never have been escalated: low-risk use cases inside an approved pattern, reversible tool grants within a declared scope, prompt edits that pass the evaluation suite. Push these down with a written standard and a reporting line back — and make the delegation explicitly revocable, so that pulling it back is a normal act rather than a crisis.</p><p>8. The one number worth watching</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9KP5C7u90y5_zKbc" /></figure><p>The shape tells you more than the level does.</p><p>Flat and near zero is the most common shape and the easiest to misread as a well-behaved system. It almost never is. Check the median review time before you check the model.</p><p>High and staying there — a third of decisions overturned, month after month — means the oversight is real and engaged, and the automation is not earning its place. That is a good problem, but it is a problem: you have granted authority the system has not justified, and you are paying for it in human attention every day.</p><p>Declining, then settling above zero is the healthy shape. It starts around a quarter, falls as the system genuinely improves, and flattens somewhere meaningfully above zero because reviewers are still finding things. This is what Part I’s idea of progressively earned authority actually looks like in a chart.</p><p>A cliff to zero is the one to treat as an incident. Engaged reviewers who abruptly stopped refusing did not all become convinced in the same week. Something changed — the queue got longer, or somebody senior overruled a refusal in a way everyone heard about. Go and find out which.</p><p>A zero refusal rate is not a passing grade. It means either the system is flawless or the gate is not operating, and only one of those has ever been true.</p><p>Publish the rate per use case. Make an unexplained fall to zero an incident in its own right.</p><p>9. What has to survive eighteen months</p><p>Accountability is retrospective. It arrives when someone asks why a decision was made, typically long after everyone who made it has moved on. So the question is not what you logged. It is whether what you logged can still answer that question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_lkUSbetUUvOBi8X" /></figure><p>Not a log of what happened. A record of why it was reasonable, written while it still was.</p><p>The test is whether someone who was not there could reconstruct why this was reasonable, without asking anyone who was.</p><p>The interesting part is the failure modes, because every one of these was a record that looked complete on the day it was written. The version was never pinned, and the provider has since upgraded the model, so the decision cannot be reproduced and the evidence proves nothing about the system that actually ran. Retention deleted the inputs — the trace survived, but the documents it reasoned over were purged at ninety days by a policy nobody mapped against the assurance requirement. Only the outcome was captured, so you know it was approved but not what the approver saw. Or, most damning, the reason field was a template, and every record reads “reviewed and approved”, which is indistinguishable from nobody having read it.</p><p>There is a subtler problem with retrospective evidence that is worth naming: there is <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.15034&amp;sa=D&amp;source=editors&amp;ust=1789796374364067&amp;usg=AOvVaw0BohPQ1lgSDVZlqqQRSGHJ">evidence that models behave differently when they appear to be under observation</a> . If your assurance sample is drawn from a recognisably different context than production, your record may be an accurate account of a system that only exists during review.</p><p>10. Incidents, and the only question that matters</p><p>Severity for AI incidents should follow reversibility, not embarrassment. Part II’s matrix does double duty here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_uA3i-Edbpu6S5Hr" /></figure><p>Grade by what cannot be undone. Then ask one question.</p><p>S1 is an irreversible action with broad exposure — money moved, data left the tenancy, an external party was told something wrong. S2 is irreversible but contained. S3 is reversible, but a control failed: the gate did not fire, or fired and was bypassed, and nothing broke this time. S4 is a near miss where the control held, which is the most valuable category you have and the one most organisations do not record at all.</p><p>Then, in the postmortem, one question does most of the work. Was the control absent, or was it present and not used?</p><p>This distinction matters more as agents take more actions, because the volume of things to route around goes up. Recent <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.18583&amp;sa=D&amp;source=editors&amp;ust=1789796374368574&amp;usg=AOvVaw1H-lj5zD7whb1qDOFLdAoz">measurement showing that mainstream coding agents take out-of-scope actions on entirely benign tasks</a> is a reminder that the shortcut does not need a malicious user to appear; ordinary work generates it.</p><p>11. Judge the process, not the luck</p><p>Everything above depends on people being willing to make a call and record their reasoning honestly. Which quadrant of this grid you choose to punish determines whether they ever will again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GxorvB7BzRU2XFB-" /></figure><p>Outcomes contain luck. Processes do not, which is why one of them is fair to judge.</p><p>Process followed, outcome good. The system worked. Record it, and ask one question in review: was anything close? A near miss inside a good outcome is the cheapest signal you will ever get, and it is the one most likely to go unspoken because nothing went wrong.</p><p>Process not followed, outcome bad. The only uncontroversial quadrant. The single thing to be careful about is that the consequence lands on whoever held the authority, not on the most junior person who touched the system.</p><p>Process not followed, outcome good. Somebody skipped the gate and got away with it, and this is the most dangerous quadrant in the grid. It feels petty to sanction a success. But if you let it stand, you have taught the entire organisation that the process is optional whenever someone is confident, and every subsequent bypass will cite this one. The luck is not a defence.</p><p>Process followed, outcome bad. Everything was done correctly and it still went wrong, which means the control was insufficient — not the person. How you handle this quadrant is the single biggest determinant of whether people keep bringing you bad news early. Punish it once, even implicitly, even just by asking the decider to explain themselves in a room full of executives, and you will get no more refusals, no more early warnings, and eventually no more honest records, because writing down your real reasoning becomes a liability.</p><p>12. How much oversight should you actually buy?</p><p>There is a question underneath all of this that no governance document will answer for you, and it is the one executives actually ask: how much is enough?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dVJ3_tdim6m2jdg9" /></figure><p>Enough that the next pound of control buys less than a pound of avoided error.</p><p>The shape is familiar from every other risk discipline. The cost of control rises with intensity, and accelerates — each added gate is latency, queue time and attention that has to come from somewhere. The expected cost of error falls, steeply at first and then barely at all. The total is a U, and the bottom of it is the defensible point: not the safest possible position, but the one you can justify to a board and to a regulator with the same explanation.</p><p>Three things make this usable rather than a nice diagram.</p><p>Draw it per use case, not per company. A summarisation tool and a refund agent have their optimum in wildly different places. One global policy applied uniformly puts both of them in the wrong spot, over-controlling the harmless thing and under-controlling the dangerous one — which is, incidentally, how governance functions acquire their reputation.</p><p>Price the control honestly. Include the queue wait, the reviewer’s time at real cost, the added latency, and the features that were never built because the governed path was slow. That last one is invisible in every budget and frequently the largest.</p><p>Price the error at full cost. Remediation, regulatory exposure, the customers who leave without telling you why, and the year you spend rebuilding trust.</p><p>And note the floor. The red curve never reaches zero, because past a certain point the remaining risk is the part your evidence cannot reach — the argument <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.15164&amp;sa=D&amp;source=editors&amp;ust=1789796374378082&amp;usg=AOvVaw3WWP5UzvNAgTy6M_6cHclr">that behavioural evaluation cannot verify the deeper properties governance frameworks are starting to demand</a> . Beyond that point, buying more testing is buying reassurance rather than safety. The only controls that still work are narrowing what the system is permitted to do, or not doing it.</p><p>13. The frameworks already ask for this</p><p>It would be convenient to treat the organisational layer as good practice that can wait until after the platform work is done. It cannot, because three separate regimes already ask for these artefacts by name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KM_5mG9JV-sWgg4t" /></figure><p>Different vocabularies, substantially the same six things.</p><p><a href="https://www.google.com/url?q=https://www.nist.gov/itl/ai-risk-management-framework&amp;sa=D&amp;source=editors&amp;ust=1789796374379621&amp;usg=AOvVaw0mplIK71aQjhpITzhwevo1">NIST’s AI Risk Management Framework</a> puts an entire function — Govern — around exactly this material: documented roles, traceability sufficient for third-party review, oversight described with the authority and competence to intervene, and incident response planned in advance rather than improvised. <a href="https://www.google.com/url?q=https://www.iso.org/artificial-intelligence/ai-management-systems&amp;sa=D&amp;source=editors&amp;ust=1789796374380334&amp;usg=AOvVaw1-3pX9BTEaYzjV2RVi2QJo">ISO/IEC 42001</a> turns much of it into certifiable management-system obligations: top-management accountability, documented information under change control, competence requirements, management review at planned intervals, and corrective action. The makes a subset of it binding for high-risk systems in scope — defined provider and deployer obligations, technical documentation and automatic logging, human oversight measures that let a person actually intervene or stop the system, conformity assessment before market, and post-market monitoring with reporting duties for serious incidents.</p><p>The practical value of the mapping is not compliance theatre. It is that the accountability work you were treating as optional is already required of you in three places, which is usually the argument that unlocks the budget.</p><p>14. Governance has to be the fast path</p><p>All of which is worthless if people go around it, so it is worth being precise about why they do. It is almost never disagreement. It is the wait.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1U0zMSffO9yPdJE9" /></figure><p>Engineers do not route around controls because they disagree with them.</p><p>Picture the two paths honestly. On the governed path, a developer registers the use case in twenty minutes, gets a scoped gateway key immediately, forks a pre-evaluated prompt template, inherits evaluation gates as a single CI block, and ships on day two. Eighteen months later there is a rollback in one command, the evidence has already been collected, and an incident is reconstructed in twenty minutes.</p><p>On the shadow path, they put a personal API key on expenses, write the prompt as a string literal, test it by hand once because it looked fine, and ship the same afternoon. Eighteen months later it is business-critical, nobody knows it exists, there is no rollback and no evidence, and reconstructing what it ever did takes three weeks and ends in a rebuild.</p><p>The second path wins every time the first one is slower, and no amount of policy changes that. Which leads to the conclusion I would most like people to take from this section:</p><p>Every shadow deployment is a review of your developer experience, not of your policy.</p><p>If the governed path is slower, you have not built a control. You have built a tax, and people will avoid it in proportion to how much it costs them. The governed path has to be the one with the templates, the keys already provisioned, the quotas, the working examples and the responsive support — so that going around it is not rebellion but simply more work. That is an investment in platform engineering, not in governance, and it is usually the highest-return governance spend available.</p><p>15. Where this leaves the series</p><p>Three pieces, three claims, and they only work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uSMgBkQGSca0egMC" /></figure><p>Part I: the evidence. Part II: the enforcement. Part III: the accountability.</p><p>Part I said a system must earn decision authority with evidence, rather than inheriting it because it demos well. Part II said evidence without a chokepoint is a preference with good documentation, and described the machinery that can refuse. Part III says that machinery still resolves, at its hardest edges, to a person — and that if that person has no time, no information, no independence, no authority and no consequence, then the whole stack terminates in a signature that means nothing.</p><p>Three things I would hold onto.</p><p>Name the owner, not the committee. A forum can advise, record and challenge, and those are real contributions. It cannot be accountable, because accountability that is shared among eleven people in a quarterly meeting is not held by anyone. Put a person’s name against the decision, and give them the authority that makes the name mean something.</p><p>Spend oversight where it cannot be undone. Attention is finite and non-transferable. Every approval you request on a reversible action is an approval you will not get on the one that matters. Routing is a safety control, not an efficiency measure.</p><p>Judge the process, not the luck. Punish a correct decision that ended badly and you will never be told about the next one early. Reward a bypass that ended well and you have repealed your own policy, whatever the document still says.</p><p>None of this is a solved problem, including for me. The uncomfortable part of Part II — that there is a gap between what governance frameworks ask you to assert and what testing can actually establish — does not close in Part III. It just stops being an engineering problem and becomes a question about who is willing to put their name on the residual, having understood what it is.</p><p>That is not a satisfying place to end a series. I think it is the honest one.</p><p>If you have run any of this in a real organisation — particularly the refusal-rate metric or the decision-rights split — I would genuinely like to hear where it broke. The failure reports are more useful than the success stories.</p><p>Part I and Part II of this series are worth reading first if you have not — this piece assumes both. Research references throughout point to the source papers; several are recent preprints, and are worth reading directly rather than taking my summary of them.</p><p>#AIGovernance #ArtificialIntelligence #OpenSource #Cybersecurity #MachineLearning #RedTeaming #LLMSecurity #ResponsibleAI #MLOps #AIAgents #ModelCards #AISupplyChain #Observability #DataPrivacy #AIRisk #PromptInjection #AIEvaluation #SoftwareSupplyChain #AIAudit #MLSecOps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7aa1199bb44a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompts Are Guidance Gateways Are Enforcement]]></title>
            <link>https://medium.com/@eddie.hc.tsui/prompts-are-guidance-gateways-are-enforcement-9fee8680144d?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fee8680144d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[risk-management]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 07:28:08 GMT</pubDate>
            <atom:updated>2026-09-18T07:29:08.648Z</atom:updated>
            <content:encoded><![CDATA[<p>Part II — building AI governance that can actually say no.</p><p>Eddie Tsui · AI governance and decision assurance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jc1rc9Zc6E6pdAQA" /></figure><p>The distance between an AI policy and an AI control is the distance between a sentence and a system.</p><p>In the first part of this series I argued that AI should not inherit decision authority just because it produces a better-sounding answer than the model it replaces. It has to earn that authority by producing evidence: a measurable objective, a fair comparison, a stable boundary, a replayable record.</p><p>Several people who read it agreed with the argument and then asked the same question, which is the right question. Fine. What actually enforces any of this?</p><p>It is a fair challenge, because most AI governance fails at exactly that point. Not at the level of principle, where almost everyone agrees, but at the level of mechanism. Organisations write the policy, circulate the policy, train people on the policy, and then discover eighteen months later that nothing in their architecture was ever capable of noticing when the policy was broken.</p><p>This piece is about the mechanism. It is more concrete and more opinionated than Part I, it names specific open-source tools, and it ends somewhere uncomfortable: with the limits of what any of this can prove.</p><p>1. Governance that cannot refuse is not governance</p><p>Here is a test you can run on your own organisation this week. Take your most important AI rule — for most companies, some version of no customer data goes to an external model — and ask a simple question about it. If an engineer broke that rule this morning, what in your architecture would have stopped them?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Thm2c5dOkPWdsYEf" /></figure><p>The same rule, expressed two ways. Only the second one has authority.</p><p>If the honest answer is “nothing, but we would find out eventually”, then what you have is not a control. It is a preference with good documentation. The rule is real, the intent is real, and the enforcement is absent.</p><p>A policy you cannot fail is a preference.</p><p>This distinction matters more than it used to, for a reason that has nothing to do with principle and everything to do with volume. When three people were using AI carefully on a pilot, informal governance worked, because you could hold the whole thing in your head. When four hundred engineers are using coding agents, internal copilots, and a dozen model providers on their own initiative, informal governance does not scale down gracefully. It simply stops existing, quietly, while the documentation continues to describe a company you no longer are.</p><p>2. Be honest about where you actually are</p><p>It helps to have a ladder, because it makes self-assessment harder to fudge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kg-mYZ3g9ZwQ8wok" /></figure><p>Six levels of governance maturity. The jump from Level 1 to Level 3 is cultural. The jump from Level 3 to Level 4 is architectural.</p><p>The important thing about this ladder is that the two hard jumps are hard in completely different ways. Getting from Level 1 to Level 3 is mostly cultural: you are persuading teams to treat prompts as versioned assets and to accept a test suite that can block their release. Getting from Level 3 to Level 4 is architectural: you are building a chokepoint and moving enforcement into it.</p><p>Level 4 is the realistic destination for most organisations , and I would rather see a company reach a genuine Level 4 than describe an aspirational Level 5 it has not staffed. Level 5 — continuous red-teaming, formal model risk management, full audit automation — is expensive, and as we will see at the end, it is still bounded by something no amount of spending removes.</p><p>3. Work out what you are actually governing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7x75EETSseTE_lYu" /></figure><p>Six artefacts that silently change production behaviour. None of them look like code. All of them need an owner.</p><p>Two of these deserve particular attention because they are the ones teams almost always miss.</p><p>The second is agent tool definitions . A new tool does not change what the system says ; it changes what the system can do , which is a categorically larger change. As organisations start assembling internal MCP servers and agent skill libraries, this becomes its own governance surface — one that recent work on frames, correctly I think, as a software asset management problem rather than a novel one. <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.18401&amp;sa=D&amp;source=editors&amp;ust=1789719909871448&amp;usg=AOvVaw0lQOhiRLCj5q9PFl6cTr8u">lifecycle governance for agent skill libraries</a></p><p>4. Seven layers, and only four of them can refuse</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nyVTk-q1qX1Gnxx1" /></figure><p>Seven layers, grouped by what they can actually do: declare, enforce, or prove.</p><p>Grouped honestly, the seven layers do three different jobs. Policy and prompt standards declare — they are necessary, and they enforce nothing on their own. The AI Gateway , CI evaluation , runtime guardrails , and human approval enforce — each of them can stop something from happening, at a different granularity: per request, per version, per message, per deployment. Observability proves — it cannot prevent anything, only reconstruct it afterwards.</p><p>This is worth being pedantic about, because a governance programme built only from the declaring and proving layers is extremely common and looks impressive from a distance. It produces policies, dashboards, and quarterly reports. It has no ability to refuse.</p><p>Prompts are guidance. Policies and gateways are enforcement.</p><p>5. Put the control in the call path</p><p>The single highest-leverage move in this whole architecture is unglamorous: route every model call through one place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JeW8rCuKeYTOZpxg" /></figure><p>The gateway request lifecycle, with the latency each stage costs and the four points at which a request can be refused.</p><p>6. A rule is not a check</p><p>Inside the gateway, the thing that makes policy real is expressing it as code rather than prose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fEWHRheDxecvOEZB" /></figure><p>Same sentence, two levels of authority. Only the second one is a control you could show an auditor.</p><p>The sentence has not changed. The authority has. And the second version has a property the first one never will: it produces evidence continuously, as a side effect of operating, rather than as an annual exercise in reconstruction.</p><p>For high-risk use cases it is worth knowing that this style of constraint can get more expressive than simple allow-or-deny. Rules with a temporal shape — whenever the agent reads customer data, it must redact before any outbound action — can be expressed as formal temporal logic and checked at runtime; there is for this, at much lower cost. If you go there, treat the constraint set as a versioned governance artefact, exactly like your policies and prompts. <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.16198&amp;sa=D&amp;source=editors&amp;ust=1789719909878265&amp;usg=AOvVaw1UVxYAe1snkcALK9sniNPa">work showing small specialised monitors outperform large model-as-judge setups</a></p><p>7. A prompt edit is a model change</p><p>If a sentence can alter production behaviour, then editing that sentence is a production deploy. Most organisations do not treat it that way. Someone adjusts a system prompt in a console, the behaviour of a customer-facing feature changes, and there is no diff, no test, no approver, and no way back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7yAM0eNO2k5hi_QE" /></figure><p>A prompt promotion path, and a release history in which the cost-saving edit was caught before it reached anyone.</p><p>A prompt registry — is the usual open-source answer — fixes this by making prompts versioned, tested, approved, and immutable once released. Callers then ask the gateway for a use case , never for a raw prompt string, which means rollback is one step rather than an archaeology project.</p><p>The release history in that diagram is the pattern I have seen most often in practice, and it is worth spelling out. Version 1.5 was a sensible engineering change: shorten the system prompt, cut token cost. It also quietly dropped a required legal disclaimer, because the disclaimer was prose inside a long instruction rather than a structured block. A disclosure check in CI caught it at 61% compliance. Version 1.6 achieved the same cost saving with the disclaimer restored as a fixed block that cannot be edited away.</p><p>Nobody was careless. The system was simply capable of noticing.</p><p>8. Every gate needs a number that can close it</p><p>Automated evaluation is where governance becomes a release decision instead of an opinion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j1MwZGTKKxvQxBxr" /></figure><p>Five gates, each with a threshold, a measurement, and an owner. A partial pass is a fail.</p><p>The tooling here is mature. handles behavioural and regression testing and emits JUnit XML, so results appear in your existing CI dashboards next to unit tests rather than in a separate portal nobody opens. handles adversarial probing, including probes aimed specifically at agent tool surfaces and system prompt extraction.</p><p>Two details separate a gate that works from a gate that performs working.</p><p>Express thresholds statistically. “The red team found some issues” is an anecdote. “Attack success rate is 3.1%, with a 95% confidence interval of 1.9 to 4.6, against a threshold of 5%” is a measurement you can defend in an audit and track across releases. Modern red-team tooling reports intervals; use them.</p><p>Test in the languages and states you actually ship. Single-language evaluation is not sufficient — guardrails are , and attackers can simply switch. Neither is single-state evaluation: there is a <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.18239&amp;sa=D&amp;source=editors&amp;ust=1789719909882304&amp;usg=AOvVaw1-UYyH6TgLpe0C7wbjLhdO">measurably weaker in low-resource languages</a> , which means the “safe” behaviour of asking a follow-up question opens a channel your tests probably never exercise. <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.17324&amp;sa=D&amp;source=editors&amp;ust=1789719909882650&amp;usg=AOvVaw2y_0mIC--thyyAZOI4UTeC">counter-intuitive finding that an agent’s clarification-seeking state is more vulnerable than its execution state</a></p><p>9. Injection is an authorisation problem wearing a text costume</p><p>Now we get to the part where agents change the shape of the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0nsFmDlT6mSLkTIn" /></figure><p>Input filtering has a ceiling. Action authorisation does not depend on winning that argument.</p><p>The reason is that there is no reliable boundary between content and command. A <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.17634&amp;sa=D&amp;source=editors&amp;ust=1789719909884333&amp;usg=AOvVaw0oQd-5PFrChVGYqQ9vQokF">contextual-integrity analysis of injection defences</a> makes the theoretical case: ambiguity between data and instruction is irreducible, so some share of attacks will always arrive looking legitimate. The empirical picture agrees. Attacks that defeat instruction-versus-data separation directly, and <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.18133&amp;sa=D&amp;source=editors&amp;ust=1789719909884864&amp;usg=AOvVaw0RK-qFrp2_R65HPdGLmbU4">reframe hostile instructions as innocuous worked examples</a> reach high success rates against well-aligned frontier models, which disposes of “we use a safe model” as a defence. <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.17971&amp;sa=D&amp;source=editors&amp;ust=1789719909885143&amp;usg=AOvVaw3ILiyvol51NC-QRmN_jC17">query-efficient obfuscation attacks</a></p><p>This is not just theoretically tidier; it is what measurably works. The most production-realistic injection benchmark I am aware of <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.17986&amp;sa=D&amp;source=editors&amp;ust=1789719909886488&amp;usg=AOvVaw2H-mweBDDr312i_axRXT5t">found that a two-layer defence — a prompt filter plus pre-execution tool-call authorisation — intercepted every attack tested</a> . Keep the input filter; is a cheap first pass. But put your real investment on the action side.</p><p>The attack can succeed against the model and still fail against the system. That is the whole design goal.</p><p>10. Every tool call is a request for permission</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BK2I7vRtBZ-tWTRQ" /></figure><p>The model decides what it would like to do. The policy engine decides whether it may.</p><p>The example in that diagram is deliberately mundane, because the mundane cases are the common ones. An agent was asked to tidy a billing module. Somewhere in its reasoning it decided to modify a production infrastructure file. No attacker was involved. Nothing in the task mentioned production.</p><p>This is not hypothetical. There is <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.18583&amp;sa=D&amp;source=editors&amp;ust=1789719909888141&amp;usg=AOvVaw3d5u2eRJNzTduwit10ZdLy">measurement showing that mainstream coding agents take out-of-scope actions on entirely benign tasks</a> — deleting unrelated files, rewriting configuration — and, importantly, that the framework’s permission model matters more than the model’s alignment . Which is good news, because the permission model is the part you control. If you deploy coding agents across an organisation without consent gates, you should expect occasional unintended destruction, and you should prefer approval-gated agent frameworks in any repository with real blast radius.</p><p>Note what the denial in that diagram produces besides safety: an out-of-scope event in the audit log, and a decrement to that agent’s trust score. Which brings us to what you do with that telemetry.</p><p>11. Reversibility, not accuracy, sets the ceiling</p><p>The most useful reframing I have found for agent authority is to stop asking how accurate the agent is and start asking what happens when it is wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5T6oXdF8131UHgu9" /></figure><p>Accuracy decides whether an agent is useful. Reversibility decides how much authority it may hold.</p><p>A model that is right 97% of the time is excellent for drafting a reply and unacceptable for issuing refunds, and the difference has nothing to do with the 97%. It is that one of those mistakes is a deleted draft and the other is money that left.</p><p>That last quadrant is the one worth being stubborn about. The right answer to “the agent needs production deploy access to be really useful” is usually not a better guardrail. It is a narrower tool.</p><p>12. The failure mode that only time reveals</p><p>Here is a risk that is nearly invisible to conventional evaluation, and that is becoming more relevant as agents acquire persistent memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_As3SlUexa4hvCbz" /></figure><p>Both lines are real. Only one of them is being measured.</p><p>13. Map your controls to named risks, then look at the holes</p><p>At some point you have to answer the question “are we covered”, and the only non-delusional way to do that is against an external taxonomy rather than your own list of things you happen to have built.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t8FtcqZebOJRqBfy" /></figure><p>The matrix does not tell you that you are safe. It tells you where you are not looking.</p><p>For the governance programme wrapped around all this, <a href="https://www.google.com/url?q=https://www.nist.gov/itl/ai-risk-management-framework&amp;sa=D&amp;source=editors&amp;ust=1789719909895107&amp;usg=AOvVaw27L_Fb4TU50Zz-_Lu-FnLB">NIST’s AI Risk Management Framework</a> and its Govern-Map-Measure-Manage functions are the most useful organising scaffold, and they map cleanly onto the technical layers: your evaluation suite is Measure, your observability and response is Manage. If you operate in or sell to the EU, run your use case registry against the risk tiers — anything touching biometrics, employment, credit, or law enforcement triggers full high-risk obligations — and treat Article 50 disclosure as a non-negotiable block in your prompt registry rather than a per-team decision.</p><p>14. The audit gap, and why I am telling you about it</p><p>If this article stopped at the previous section it would be a decent engineering blueprint and a slightly dishonest one. So here is the part that governance vendors do not put on a slide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YLIpN75dMu1jO4Jx" /></figure><p>What you are asked to assert, what your evidence establishes, and the distance between them.</p><p>Those are genuinely valuable. They are also not the same kind of claim, and no volume of passing tests closes the distance. Recent work names this fragile assurance : the evidential structure does not support the safety claim being asserted on top of it, and . <a href="https://www.google.com/url?q=https://arxiv.org/abs/2605.15164&amp;sa=D&amp;source=editors&amp;ust=1789719909897444&amp;usg=AOvVaw2cRtcpJndgSt8XDiBAZh0R">argues that behavioural evaluation cannot verify the deeper properties governance frameworks are starting to demand</a></p><p>For low and medium-risk use cases this is fine, and I want to be clear about that. Behavioural testing is proportionate to most of what organisations actually deploy, and you should not let philosophical humility talk you out of shipping a summarisation feature.</p><p>For genuinely high-risk use cases — autonomous agents with access to critical systems, irreversible decisions about people — there are only two honest responses. Decline the autonomy , because if the claim you cannot evidence is the claim that matters, the engineering answer is a narrower system rather than a broader disclaimer. Or put the authority somewhere your evidence does not have to reach : a human decision that does not rest on test results, and an action constrained so that a wrong answer stays recoverable.</p><p>Be precise about what your evidence proves, and explicit about what it does not. A governance file that implies more than it can carry is worse than one that admits the limit.</p><p>15. Ninety days to a working Level 4</p><p>If you want to act on this rather than file it, the sequence matters more than the ambition. Build the chokepoint before the sophistication.</p><ol><li>Weeks 1–2. Inventory. Find every model call in the organisation. This will take longer than you think and the number will be larger than you expect. Register each one as a use case with an owner, a data class, and a risk tier.</li><li>Weeks 3–4. Stand up the gateway in log-only mode. No policy, no blocking. Just route traffic through it and observe. You now have the audit layer, and you have not broken anything.</li><li>Weeks 5–6. Move prompts into the registry. Version and pin them. Serve them from the gateway, so callers request a use case rather than pass a string.</li><li>Weeks 7–8. Add CI evaluation. Golden-set regression and required-disclosure checks first. These are boring, high-value, and almost never contested.</li><li>Weeks 9–10. Enable your first blocking policy. Pick one, make it the unambiguous one — PII to external models — and let it fail closed. This is the moment the programme changes from advisory to real.</li><li>Weeks 11–12. Add tool-call authorisation for agents , scoped with the reversibility matrix. Then run your coverage matrix and publish the gaps honestly, including the ones you are not going to fix this year.</li></ol><p>The order is deliberate. Every step produces something useful on its own, nothing depends on a big-bang migration, and the politically difficult step — blocking a real request — comes after you have spent six weeks demonstrating that the gateway does not break anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BIbM3dFtMdzWMtTk" /></figure><p>Part II of a series on AI decision assurance.</p><p>Governance is not a document, and it is not a committee. It is the set of places in your architecture where a request can be told no, plus an honest account of what those places can and cannot prove.</p><p>Build the chokepoint. Authorise actions rather than adjudicate sentences. Scope authority by what is recoverable rather than by what is accurate. And write down the limit of your own evidence before somebody else has to find it for you.</p><p>Author’s note: the architecture described here is a working position, not a finished standard. The tool choices reflect an open-source-first preference and a constraint that confidential data must not reach external models; your constraints may reasonably lead you elsewhere. I have tried to be explicit about which claims are engineering judgement and which rest on published work, and the limits described in section 14 apply to my own conclusions as much as to anyone else’s.</p><p>#AIGovernance #AIAgents #AISafety #LLM #LLMOps #PromptEngineering #AISecurity #PolicyAsCode #OPA #OWASP #NISTAIRMF #EUAIAct #ResponsibleAI #AIRiskManagement #PromptInjection #AgenticAI #MLOps #PlatformEngineering #Compliance #ArtificialIntelligence</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fee8680144d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Can Replace the Model]]></title>
            <link>https://medium.com/@eddie.hc.tsui/ai-can-replace-the-model-medium-article-266004605e4b?source=rss-95401a211927------2</link>
            <guid isPermaLink="false">https://medium.com/p/266004605e4b</guid>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Never a Paragraph]]></dc:creator>
            <pubDate>Fri, 18 Sep 2026 06:50:06 GMT</pubDate>
            <atom:updated>2026-09-18T07:30:08.362Z</atom:updated>
            <content:encoded><![CDATA[<p>A practical governance framework for deciding when AI should augment, challenge, or replace traditional data science</p><p>By Eddie Tsui · 14 min read</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z-XJHFQnS5v3hRCY" /></figure><p>Every enterprise AI conversation eventually arrives at the same moment. Someone has seen a demonstration that was genuinely impressive, and someone else has to decide whether the organization will now act on it.</p><p>That is the moment the conversation usually goes wrong — because the question on the table is almost always the easier one.</p><p>The important question is not whether AI can produce an answer. It is whether an organization can prove when that answer should be trusted, acted upon, or rejected.</p><p>The argument in one line: producing an answer is a technical capability; receiving authority to act is a governance decision. The two are routinely confused, and the confusion is expensive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g5dHOUnZRRqb6W3s" /></figure><p>Producing an answer is a technical capability. Receiving authority to act is a governance decision.</p><p>1. The wrong question, asked confidently</p><p>Imagine an organization that already has a data science model supporting a recurring business decision — pricing, credit, inventory, churn, maintenance, it does not matter which.</p><p>The model is not perfect. It needs structured data, specialist maintenance, statistical validation, and periodic recalibration. Everyone involved knows its weaknesses intimately, which is itself a form of control.</p><p>Then a new AI system arrives. It is faster. It understands natural language. It combines structured data with documents, emails, and user instructions. It explains itself fluently. In a demonstration, its recommendations look intelligent.</p><p>So the question becomes: can AI replace our existing data science model?</p><p>I believe that is the wrong starting point. The better question is: can AI carry the same, or a stronger, burden of evidence?</p><p>Replacing a model is easy. Replacing the controls around the model is much harder. When an enterprise moves from established rules or data science to AI, it can unintentionally discard reproducibility, transparent calculations, stable performance measures, version control, uncertainty estimates, and clear accountability — none of which appeared in the demonstration, because none of them are visible when things go well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tx_XJXLOEqndW8KI" /></figure><p>Replacing a model without replacing its assurance mechanisms creates an assurance gap.</p><p>A more powerful model inside a less governable system is not progress. It is the same decision, made faster, with fewer ways to find out that it was wrong.</p><p>2. The model is not the product</p><p>A model creates no business value on its own. Value comes from a complete decision system: data is observed, a prediction is generated, business constraints are applied, an action is selected, the action is executed, and the outcome is measured.</p><p>Every one of those stages can fail independently of the model. A prediction can be statistically accurate while the resulting decision is commercially poor. A recommendation can be sound but arrive too late to matter. A model can look effective because it was only ever evaluated on easy cases. And an AI-generated explanation can be entirely persuasive while citing the wrong number.</p><p>Governance therefore cannot stop at model accuracy. It has to follow the whole chain from observation to action to outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b7nCE4e4J2IrAmvI" /></figure><p>The model is only one component. Governance must cover the complete path from measurement to action and learning.</p><p>3. AI versus data science is a false choice</p><p>Public discussion tends to present traditional data science and generative AI as rival camps. In practice, an organization has at least four viable ways to position AI inside a decision system, and each one carries a different validation burden.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MRVI8p0snsMgp3I6" /></figure><p>AI can sit on top of, alongside, inside, or in place of data science. Each position creates a different validation burden.</p><p>The useful questions are not about presence, but about position and power:</p><ul><li>Where does AI sit in the decision path?</li><li>What information does it control?</li><li>Can it change the governed answer?</li><li>Can it initiate or execute an action?</li><li>What happens when it disagrees with the existing model?</li><li>How can its decision be reconstructed afterwards?</li></ul><p>4. The Decision Assurance Stack</p><p>To decide what authority an AI system should receive, I propose a seven-layer Decision Assurance Stack . This is a practical synthesis, not an official regulatory standard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3nEg8y_azcC8JFHV" /></figure><p>Decision authority should rest on accumulated assurance, not on the apparent sophistication of the technology.</p><ol><li>Purpose and decision rights — what may this system do?</li><li>Data and measurement validity — are we measuring the right thing?</li><li>Reproducibility and deterministic replay — can we reconstruct the decision?</li><li>Baseline and challenger validation — did complexity earn its place?</li><li>Decision and causal validation — did the action improve outcomes?</li><li>Uncertainty and failure boundaries — when must the system abstain?</li><li>Executable governance — can a failed control stop a release?</li></ol><p>Each layer answers a different question, which is why the stack is not a score. Strength at one layer does not compensate for failure at another. A system with excellent accuracy and no replay capability is not seven-tenths governed; it is ungoverned in a specific and predictable way.</p><p>5. Measurement before modelling</p><p>A sophisticated model cannot repair a badly defined target.</p><p>Before comparing AI and data science at all, an organization should validate identity resolution, timestamps and cut-offs, target construction, missing observations, data leakage, source provenance, and consistency between training and production.</p><p>This matters because observed business outcomes are not the same thing as the underlying phenomenon you want to predict. Observed sales, for instance, do not represent unconstrained demand when the product was out of stock. Treat the absence of a sale as proof of zero demand, and the model will faithfully learn the wrong relationship — whichever technology you used to build it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uX93pUyzbjSd429G" /></figure><p>Model validity begins with measurement validity. Errors introduced before modelling can be amplified by both AI and DS.</p><p>Many apparent model failures are measurement failures wearing an algorithmic costume.</p><p>6. Mirror before replacement</p><p>Before replacing an incumbent process, reproduce it.</p><p>A deterministic mirror is a coded implementation of the current decision logic, run on the same historical inputs. Its first purpose is not improvement. Its first purpose is control.</p><p>Building one is usually uncomfortable, because it reveals what the incumbent process actually does as opposed to what the policy document says it does: where undocumented exceptions live, where human intervention quietly enters, and whether the old-versus-new comparison everyone has been arguing about is even fair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_H8clDZlULqVC_dc" /></figure><p>A deterministic mirror establishes fidelity to the existing process. It does not establish that the existing process was correct.</p><p>Fidelity proves you copied the incumbent correctly. It does not prove the incumbent was correct. The same warning applies when AI is trained to reproduce historical human decisions: behavioural similarity is not evidence that those decisions were appropriate or optimal.</p><p>7. Complexity must earn its right to operate</p><p>Every sophisticated method should compete against the simplest credible alternative. A serious challenger set usually includes doing nothing, a simple business rule, a naive statistical baseline, the current production model, an improved DS or ML challenger, and the AI challenger.</p><p>The comparison only means something if every candidate runs on the same track: the same data, target, cut-off dates, constraints, decision units, and business measures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PlgQAwCF7v_29muE" /></figure><p>A fair challenger test requires every method to run on the same analytical and commercial track.</p><p>A new method should not receive decision authority because it is technically more advanced, and certainly not because it produced a more impressive demonstration.</p><p>8. Prediction is not proof of business value</p><p>Prediction quality, decision quality, and causal business value are three separate levels of evidence, and they are regularly reported as one.</p><p>A system can predict accurately and recommend an infeasible action. A policy can select a feasible action and then fail in execution. An outcome can improve after deployment for reasons that have nothing to do with the system.</p><p>These distinctions become material the moment an organization claims that AI increased sales, reduced inventory, improved compliance, or generated financial value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bk-er0aB92X1gUNR" /></figure><p>Prediction is the first level of evidence, not the final proof of business value.</p><p>A better prediction is not automatically a better decision, and a better decision is not automatically proof of causal value. An outcome claim should always state what was measured, against which comparator, over which population, at what decision level, and with which limitations.</p><p>9. A governed system must be allowed not to answer</p><p>Enterprise AI should not only know how to answer. It should know when to answer, when to escalate, and when to abstain.</p><p>A confidence label is not enough, because a label changes only the wording. Uncertainty should change what the system is permitted to do .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FPcptA2KMjv-8mTC" /></figure><p>The operational response to uncertainty should be answer, escalate, or abstain, depending on evidence and exposure.</p><p>So it is worth asking plainly: does your AI system have permission not to answer? A system without an abstention condition is structurally obliged to express certainty, including when the evidence is missing or the situation sits outside its validated boundary.</p><p>10. Governance must be executable</p><p>Frameworks and review committees are necessary, but the high-value controls should also exist as automated gates in the release path wherever that is practical.</p><p>A control becomes materially stronger when failing it can block deployment, restrict authority, or trigger an investigation — rather than generate a documented concern that everyone acknowledges and then works around.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VWe2KfJl73cAUWmh" /></figure><p>Governance becomes operational when controls can prevent an invalid release, not merely document concerns afterward.</p><p>A committee can record a concern. Only a gate in the release path can prevent one.</p><p>11. The AI Substitution Test</p><p>Before AI replaces an existing data science component, I propose five independent governance gates. This is an author-proposed framework, not an official regulatory test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TRVxgwlMUW-JoPGP" /></figure><p>Replacement should pass five independent governance gates. The framework is not intended as a simple average score.</p><p>Gate 1 · Outcome. Does AI improve the actual decision outcome, rather than an intermediate technical measure?</p><p>Gate 2 · Evidence. Has AI been compared fairly against the incumbent, a naive baseline, and a credible DS challenger?</p><p>Gate 3 · Replay. Can the organization reconstruct the input, versions, output, constraints, human intervention, and final action?</p><p>Gate 4 · Failure containment. When AI is wrong, will the error be detected before it creates unacceptable exposure?</p><p>Gate 5 · Governance equivalence. Does the deployment preserve or improve the controls available under the component it replaces?</p><p>Five gates, not five points. A serious failure in one gate should not be hidden by strength in another.</p><p>12. Replacement should be progressive and reversible</p><p>Authority should be earned in steps: exploration, explanation, extraction, challenge, recommendation for approval, bounded execution, and only then component-level replacement. Different components can sit on different rungs at the same time, and most should.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ttxVXDJ_D3Jc3kro" /></figure><p>Decision authority should increase only as the evidence and control environment mature.</p><p>The ladder must also work downwards. New evidence, data drift, upstream system changes, or a failed control should be able to reduce authority without requiring a crisis, a reorganization, or anyone’s permission to be embarrassed.</p><p>13. The stronger future is a hybrid architecture</p><p>Each capability in this space is good at something the others are not.</p><p>AI is strong in language, context, interaction, perception, and unstructured information. Data science is strong in structured measurement, statistical testing, calibration, and prediction. Deterministic code is strong in arithmetic, rules, constraints, replay, and auditability. Optimization selects well among feasible actions. And humans remain essential for accountability, material exceptions, risk acceptance, and any decision that sits outside the validated operating boundary.</p><p>The objective is not to crown a universal winner. It is to give every component a role that can be validated, monitored, and governed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1g2cPcccCXlgTAuB" /></figure><p>The stronger future is a governed architecture in which AI, DS, rules, optimization, and human authority perform distinct and auditable roles.</p><p>Conclusion: AI must earn decision authority</p><p>AI can often produce a forecast, classification, recommendation, explanation, or action faster than a conventional analytical workflow. That is a real advance, and it is not the same as being entitled to act.</p><p>To replace data science responsibly, an AI system should be able to demonstrate nine things:</p><ol><li>Valid measurement</li><li>Improvement over credible baselines</li><li>Reproducibility at the level of the individual decision record</li><li>Uncertainty control, including the ability to abstain</li><li>Failure containment</li><li>Better feasible decisions, not merely better predictions</li><li>Real outcome evidence wherever outcome claims are made</li><li>Continuing monitoring after release</li><li>Governance at least equivalent to what it replaces</li></ol><p>If a replacement proposal cannot produce those nine, it is not ready to own a decision — however good the demonstration was.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zvr9xM5gzBx6oR7O" /></figure><p>Innovation should not require trust to move faster than proof.</p><p>The final question, then, is not can AI make this decision?</p><p>It is: what evidence must AI produce before we allow it to make this decision, and what will stop it when that evidence is no longer valid?</p><p>That second question is where AI governance turns into decision assurance. And it is how an organization can innovate without asking trust to move faster than proof.</p><p>Author’s note</p><p>This article presents a generalized governance framework informed by experience with enterprise decision systems. Project identities, operational details, and confidential information have been removed or abstracted.</p><p>The Decision Assurance Stack, the AI Substitution Test, the authority ladder, and the conceptual decision architecture are author-proposed frameworks. They should be adapted to each organization’s legal, technical, operational, and risk context.</p><p>If this was useful, the most valuable thing you can do is ask one question in your next AI review: what would stop this system, and who is allowed to press it?</p><p>#AIGovernance #DecisionAssurance #ResponsibleAI #DataScience #ArtificialIntelligence #MachineLearning #ModelRiskManagement #AIRiskManagement #EnterpriseAI #MLOps #TrustworthyAI #AIAudit #DataGovernance #DataQuality #Explainability #CausalInference #DecisionIntelligence #AIStrategy #AIEthics #AIAdoption</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=266004605e4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>