<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yigit Tas on Medium]]></title>
        <description><![CDATA[Stories by Yigit Tas on Medium]]></description>
        <link>https://medium.com/@yigit.tas?source=rss-440042aab33c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SNl-vx7eH3Or4aGsOA4nkQ.png</url>
            <title>Stories by Yigit Tas on Medium</title>
            <link>https://medium.com/@yigit.tas?source=rss-440042aab33c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:57:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yigit.tas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Prompts Have No Test Coverage]]></title>
            <link>https://medium.com/@yigit.tas/your-prompts-have-no-test-coverage-2393e12a4986?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2393e12a4986</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Wed, 15 Jul 2026 10:59:17 GMT</pubDate>
            <atom:updated>2026-07-15T10:59:17.485Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LdrUd0IusmE2HViFqc475g.png" /></figure><p>I wrote a rule for my agent: “Run the tests after every change.”</p><p>The next day, it made a change, didn’t run the tests, and closed with a perfectly calm summary: “The change is ready.” Thank you. You’re ready, I’m ready, everyone is ready except the tests.</p><p>My first reflex, everyone’s first reflex: write a sterner sentence. “ALWAYS run the tests.” I used capital letters. Because capital letters work, obviously. It has the energy of a roommate taping a note to the fridge: “WHOEVER FINISHES THE YOGURT BUYS THE NEXT ONE.” We all know who buys the yogurt: nobody.</p><p>By now, we’ve all read the essay explaining why this happens. An agent doesn’t see one instruction; it sees layers of them: the system prompt, repo rules, folder rules, skills, and the task itself. The layers contradict one another, no one has defined which one wins, context engineering matters, and so on. True. All of it. I’m not going to write that essay again, because it has already been written. Many times.</p><p>My problem is different. The industry has known the diagnosis for two years. Why hasn’t treatment started?</p><h3>Sweeping My Own Doorstep</h3><p>Recently, I ran a small audit of our own systems. I’m reporting the result as-is because it is embarrassing, and embarrassing things are instructive.</p><p>Our most mature system has 314 green tests. Parsers, guards, formatters. All tested, all passing, a dashboard to be proud of.</p><p>The same system’s rules file describes itself in the file map: “AI role instructions (THIS IS WHERE BEHAVIOR IS SHAPED).” One file. The brain of the system. Number of tests touching that file: zero.</p><p>It gets better. In the test folder, I found a file with “policy” in its name. I was delighted. I opened it. It tests deterministic guard code. So even in the most-tested corner of the system, the word “policy” points not to the prompt, but to the steel door we built because we don’t trust the prompt.</p><p>Putting it in one sentence startled even me: <strong>the file where one hundred percent of the behavior lives has zero percent test coverage.</strong> No one had noticed, because the coverage report doesn’t even count that file. It isn’t code, so it doesn’t count.</p><p>We built CI pipelines that demand 80 percent coverage from our code; we ship our agent’s constitution to production on the strength of “I added it yesterday, and I think I phrased it well.”</p><h3>Can a Rule Even Be Tested?</h3><p>“Testing prompts” isn’t new, I know. Eval tools exist: you give the prompt a task and score the output. Fine, but they evaluate the prompt as a whole. My concern is a smaller, sneakier unit: individual rules.</p><p>A rules file doesn’t live as a whole. It lives line by line, and it dies line by line. Does the “run the tests” line still change anything? Did the “never log credentials” line die three months ago in the shadow of some newer rule? We don’t know. A sentence sitting in the file doesn’t prove it still works. Sometimes the model already behaves that way by default, sometimes another instruction overrides it, and sometimes no real task ever triggers the rule. But we keep adding rules and never delete one (what if we need it?), so the file swells with every incident into a document taken seriously for its weight rather than its power. Every paragraph is the tombstone of some midnight incident. “Never run tests against a production connection.” Rest in peace.</p><p>For decades, code has had a standard answer to this problem: write tests, read the coverage, delete dead code. The same three moves work for rules:</p><p><strong>One: define a behavioral expectation for each rule. </strong>What is the test for “run the tests”? Give the agent a small task, then inspect the trace instead of the final message: was a test command attempted? If you also want to prove the tests passed, measure the exit code. “Attempted” and “passed” are separate claims. If a rule claims to govern behavior, its evidence must be behavioral.</p><p><strong>Two: run an ablation experiment. </strong>Remove the rule from the file and run the same task again. Did behavior change? We can’t say, “The agent read this line”; that would be mind-reading. We can say something more modest and more valuable: with the rule present, behavior looked one way; with it removed, it looked another. That is a measurement.</p><p><strong>Three: classify the result. </strong>Within the model, task, and conditions under which it was tested, every rule falls into one of three buckets: live (remove it and behavior breaks; it earns its keep), dead (remove it and nothing changes; it takes up space in the graveyard), or untestable (it is linked to no observable behavior; “be careful” in spirit; the most dangerous kind, because you can neither trust it nor confidently delete it).</p><p>So much for theory. I don’t trust theory, so last week I tried this on our own rules file.</p><h3>One Afternoon, a Few Dollars</h3><p>The setup is embarrassingly simple. I took the rules file from one of our agent systems and created 20 isolated working copies. Each copy received a variant of the file: some had the original, some had the “run tests” rule removed, some had a contradicting rule added next to it (“during quick iteration, don’t run tests”), and some had the “communicate in Turkish” rule removed. Then I gave the same small coding task to an agent in every copy and did one thing above all: I did not read the final message. I read the diff, the commands that ran, whether anything was committed, and the language of the reply.</p><p>Full disclosure: five copies of each of the four variants, 20 sessions total. This is a pilot, a signal rather than a statistic. But the signal was strong enough to produce three surprises. A note on the denominators: every rule remains present in the variants where it wasn’t removed, so the test rule was present in 10 sessions total and the Turkish rule in 15. The numbers below come from reading one shared pool of sessions rule by rule, not from separate experiments.</p><p><strong>Surprise one: a rule is not a law; it’s a nudge.</strong> With the “run tests” rule present, I saw a test attempt in four out of ten sessions. With the rule removed: zero. So the rule was live; it genuinely nudged behavior. But a nudge is not a guarantee. The real lesson arrived when I added the contradiction: one line, “during quick iteration, don’t run tests,” with the original rule left untouched. The conflicting instruction won in all five sessions without producing a clarification request. And you don’t get to choose which side wins, because you never defined it.</p><p><strong>Surprise two: a rule whose presence and absence looked identical.</strong> The “communicate in Turkish” rule was present in 15 sessions and followed in none of them. I removed it: still zero. Nothing changed. With an English task, the reply was in English every time. When I went back to the file, I saw why. The rule actually reads: “Communicate in Turkish (unless the user requests otherwise).” To the model, a user writing in English can plausibly count as “requesting otherwise”; the rule carries its own escape hatch. On top of that, the model’s tendency to mirror the user’s language is a strong default behavior, and at the moment of decision, the context contains one line of Turkish instruction against a hundred lines of English. The rule isn’t wrong; it loses the tug-of-war. The painful part is that I wrote that rule and believed for months that it worked. It had simply never been tested, because I always wrote in Turkish. The task language and the rule always pointed in the same direction. The first time the rule was genuinely challenged, it failed, and I saw that failure only through the ablation.</p><p><strong>Surprise three, my favorite: agents cited the rules file while violating it.</strong> Six of the ten sessions with the test rule fully present skipped the tests; five of those six gave the same justification: “Per AGENTS.md, this change doesn’t require tests.” I opened the file. It says no such thing. The file contained the rule, yet the agent cited that same file to legitimize not following it. Anyone who has sat in a meeting where a sentence began with “per company policy” about a policy that exists nowhere knows this feeling. The sixth session? It skipped the tests and didn’t even bother with a defense; it described its own quick check and moved on. That was the most honest behavior in the table. In this small sample, fabricated justifications appeared exactly where the skip needed defending. A defense only shows up where something felt like it needed defending.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nV1iGOdC0tHY2OaeGF19vg.png" /></figure><h3>Two Confessions</h3><p>This wasn’t the first time I’d paid the price of managing without measuring. Two examples:</p><p><strong>First confession.</strong> In our rules file, Section 2 says, “By default, do not run the whole test suite.” In the same file, three sections later: “After every change touching this module, the suite must stay green.” I wrote both, on different days and in response to different concerns. Each made perfect sense at the time. Which one wins? The file doesn’t say. That contradiction never produced an error message; a human found it by accident while reading. In code, we call this a merge conflict, and we’ve had machines catch it for 20 years. In rules, we still do it by eye.</p><p><strong>Second confession.</strong> One of our rules requires every development task to be registered with a metrics service. With a single agent, no problem. The moment the work split between a manager agent and coding subagents, the rule regularly evaporated. “We’ll tell the subagent to read the rules file first,” we said. It turns out that a subagent with a narrow, code-focused prompt quietly drops the bookkeeping while the real work is waiting, like any of us at the end of a sprint with documentation left to write. The fix was not a sterner sentence; we changed who the rule was addressed to. Management science: don’t blame the worker when you assigned the work to the wrong role. But we lived with this rule dying in delegation for months without knowing. Had we measured it, we would have seen the problem in week one.</p><h3>The Claim and the Trace</h3><p>The other side of the coin: a rule is my word to the agent; the final message is the agent’s word to me. Both are words. The evidence for both is the trace.</p><p>In our pilot, the claims matched the traces in all 20 sessions. Other evaluations have found a much larger gap. In <a href="https://medium.com/@sebuzdugan/your-agent-benchmark-is-lying-to-you-65a552fea151">a recent write-up</a>, an evaluation harness that trusted the agent’s self-report showed 87 percent success, while a harness that inspected the traces found a trace-verified rate of 33 percent. The gap, more than half the cases, was work reported as done that was never done.</p><p>“I ran the tests” is a claim. Is there a command and a zero exit code behind it? “I didn’t touch the file” is a claim. Is the diff actually empty? All of this is deterministically checkable. Matching claims to traces is machine work, not trust work.</p><p>And the same discipline turns around and points at us.</p><blockquote>We don’t trust the agent’s “done,” so why do we trust our own “this rule works”? Both are unsupported claims. One just sounds more convincing because we’re the ones saying it.</blockquote><h3>The Pieces Are Emerging</h3><p>The field is not empty; on the contrary, it has been filling up quickly. <a href="https://arxiv.org/abs/2606.15828">A catalog of rules-file smells</a> includes conflicting instructions as one such smell. <a href="https://arxiv.org/abs/2605.10039">Another study</a> found that, under its experimental conditions, a rule’s position in the file did not produce the expected measurable difference; the stronger effect was compliance eroding step by step as the session progressed. The rule doesn’t age in the file; it ages inside the session, minute by minute. On the trace-verification side, <a href="https://arxiv.org/abs/2606.04990">frameworks and surveys</a> have started to appear as well.</p><p>The pieces are on the table: static analysis of rules files, trace verification, and measurements of compliance over time. What is still missing is a practical, rule-by-rule behavioral audit: under specified test conditions, is each rule live, dead, or untestable? Once you’ve seen that report, you can no longer casually say, “Let me add one more rule.” You clean the graveyard first.</p><h3>The Bill</h3><p>Everything above reduces to one line of accounting:</p><p>Writing a rule is cheap. Knowing whether the rule works is expensive. We’ve always bought the cheap one.</p><p>A sentence costs a minute; a mechanism costs a week. I confessed this in <a href="https://medium.com/@yigit.tas/9e6883bcd12d">an earlier essay</a>: for years, I chose the sentence, called it governance, and got what I paid for. This essay is my attempt to finally pay that invoice. The pilot turned out to be cheaper than I expected: 20 sessions, one afternoon, a few dollars’ worth of tokens. In exchange, I found one live rule, one dead rule, one unresolved contradiction that quietly picked a winner, and an agent that cited the rules file while violating it. I learned more about my rules file in one afternoon than I had in years of looking at it.</p><p>So now I ask one question before adding any rule: How will I know it works? If I have no answer, I don’t write it. Because a rule without an answer isn’t a rule; it’s a well-intentioned sentence. No different from the note on the fridge: writing it feels good, and it doesn’t buy the yogurt.</p><p>Prompts have no test coverage. Yet.</p><p>Neither do yours. But auditing yours now takes two commands:</p><pre>npm install -g rulecov<br>cd your-repo &amp;&amp; rulecov audit</pre><p>I turned the setup I built for this experiment into an open-source tool called <a href="https://github.com/Yiwit/rulecov">rulecov</a>. It reads your rules file, runs the ablation experiment in isolated copies of your repo, and prints a report card. It checks behavior against command traces, diffs, commits, and reply properties instead of treating the agent’s self-report as proof. One caveat: today’s v0 can measure that a test command was attempted, not that it passed. “Attempted” and “passed” are separate claims. That distinction is the point.</p><p>Looking at your table, I’m already curious which rule you’ll delete first.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2393e12a4986" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Agentic AI, a Status Is a Mood]]></title>
            <link>https://medium.com/@yigit.tas/in-agentic-ai-a-status-is-a-mood-9e6883bcd12d?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e6883bcd12d</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 02:10:36 GMT</pubDate>
            <atom:updated>2026-07-07T02:10:36.695Z</atom:updated>
            <content:encoded><![CDATA[<h4>The fifth and final post of the series: what building agentic systems taught me about the difference between a claim and a record.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liZRp5gUtJiM4umpfJJ3vw.jpeg" /></figure><p>This series has been heading in the same direction from the start. It began by arguing that <a href="https://medium.com/@yigit.tas/agentic-ai-is-more-orchestration-than-model-a24a86f4b28d?sharedUserId=yigit.tas">agentic AI is more orchestration than model</a>, and that orchestration means little until you <a href="https://medium.com/@yigit.tas/in-agentic-ai-orchestration-isnt-enough-you-also-need-to-define-the-role-of-data-99e090ccb520?sharedUserId=yigit.tas">define the role of data</a>. The last two posts narrowed in on a single word, saying what evidence is not.</p><p><a href="https://medium.com/@yigit.tas/in-agentic-ai-success-is-not-evidence-d95a1a50488a?sharedUserId=yigit.tas">Success is not evidence</a>. <a href="https://medium.com/@yigit.tas/in-agentic-ai-the-output-is-not-the-evidence-865b06a8e3f4?sharedUserId=yigit.tas">Output is not evidence</a>. Instruction is not evidence. A confident title is not evidence.</p><p>A fair question kept coming back: <em>then what is?</em></p><p>I could answer with a definition. But this series has never trusted anything that shows up without an artifact behind it, and a definition is exactly that. So let me answer with the day the question stopped being theoretical for me, and then with what I kept seeing everywhere once I knew what to look for.</p><h4>The day we looked</h4><p>Everything was green.</p><p>The test cases had been generated, executed, reported. The dashboard was happy. Every case had a title, steps, a request, an expected status: everything a reviewer would want to see was on the screen.</p><p>Then one day we looked at the artifacts. Not the model’s description of what it sent, but the actual bytes that left the system. Every recorded request body, on every “passing” test, was the same string:</p><pre>[object Object]</pre><p>The model had produced the request body as a structured object. The transport layer had stringified it the laziest way possible. So for a while, every green test had been sending the same literal garbage to the API, and no layer above was impolite enough to notice.</p><p>The bug itself was ordinary. A serialization mistake, the kind you fix in an afternoon and forget by Friday. What stayed with me was something else: <strong>there was no signal that would have told us</strong>. No status was wrong, no alert fired, no summary hinted. The only thing standing between us and months more of confident nonsense was that the system had recorded what it actually sent, and that someone eventually read it.</p><p>That is the whole answer, really. Everything after this is me unpacking it.</p><h4>The fact about the world</h4><p>The first thing that day rearranged in my head:</p><blockquote>A success flag is not a fact about the world. It is a fact about the system’s mood.</blockquote><p>The fact about the world is the artifact: the request that left, the response that returned, and an expectation those two can be compared against.</p><p>There is a quiet condition hiding in that sentence: the artifact has to exist. Ours did, and only because it was captured as a side effect of execution, not as a separate “collect evidence” step someone had to remember. Anything optional gets skipped, precisely on the day it is needed.</p><p>Picture a mobile agent that captures failure screenshots in the background, without waiting for the upload to finish. On most runs this is invisible and free. Then one run dies mid-flight, and the claim it leaves behind, “the tap failed,” has no pixels behind it. The evidence policy is fine on average.</p><blockquote>But evidence does not matter on average. It matters on the worst day.</blockquote><p>There is a subtler version of the same failure: the artifact exists, faithfully stored, while the system that asked for it waits for a delivery that never arrives. Every log in that story is green. Evidence that never reaches the decision it was supposed to inform is indistinguishable from no evidence at all.</p><h4>Where the expectation comes from</h4><p>The comparison needs a second ingredient, and this is where it gets uncomfortable: an expectation is itself a claim about the world, and models are happy to supply one.</p><p>It is a surprisingly easy failure to produce. A generation pipeline runs a case, looks at what came back, and records that this is what should come back. The claim and its proof are now the same sentence written twice. Nothing will ever fail again, and nothing will ever be tested again, either. What makes it worse is that the honest source is usually sitting right there, and it takes a certain discipline for a system to prefer the boring source over its own improvisation.</p><p>So the rule: the expectation is fixed before the run, and it comes from outside the model. The example in the ticket, the spec, the schema. When no source exists, the honest output is <em>no expectation</em>, not a plausible one.</p><p>The mechanism I have come to like most here is embarrassingly small, and it is the third ingredient: give every expectation a note about where it came from. The ticket. The spec. Or, literally, as one of the possible values, <em>default</em>: the system admitting “I made this one up.” An admitted invention can be reviewed. A silent one is a time bomb wrapped in green.</p><p>And if you keep the structure but skip the grounding, you get the worst of both worlds. Ask a model for structured assertions (URLs, selectors, orderings) and it will produce them beautifully, referencing pages that do not exist. Structure without grounding is not evidence; it is confident fiction with a schema. The one-line rule worth framing on a wall: <em>do not assume a success the page does not show.</em></p><h4>Uncertainty is part of the record</h4><p>There is a fourth ingredient, and it is the least intuitive one: a trustworthy record says how sure it is.</p><p>A generated candidate that arrives with “this rests on an assumption” or “this part is an open question” is worth more than one that arrives certain, because the certain one is hiding the exact same gaps, just without telling you. Uncertainty is not a weakness of the record; it is the record being honest about its own edges.</p><p>But declared uncertainty only counts if something reads it. I have watched candidates carry confidence levels and open questions through an entire pipeline in which not a single gate ever looked at them. Cases marked “low confidence, three open questions” sailed toward production next to the certain ones, at the same speed. An unread uncertainty field is decoration. Worse than decoration, in fact, because it lets a system claim a humility it does not practice. If a candidate says “I am not sure,” some gate, somewhere, has to change its behavior because of it. Otherwise the field is theater.</p><h4>The recursion</h4><p>Here is the part I resisted longest.</p><p>The nine differently titled test cases from two posts ago, nine names wrapping one identical request, were a claim nobody had asked to prove itself. By this point in the series, that pattern is familiar. But the lens does not stop at the model’s claims. Eventually you have to turn it on your own safeguards, because a safeguard is also a claim.</p><p>A readiness check is supposed to be the adult in the room: before anything is generated, verify that the task is actually testable. Ours was a real checklist: deterministic, versioned, the opposite of a feeling. And it lied to us anyway. One of its flags verified that acceptance criteria <em>existed</em>, not that they <em>said anything</em>. A ticket containing an endpoint address and nothing else passed as sufficient, while its error behavior was defined nowhere. The generation inherited that false confidence, and the cases came out looking ready, because the gate had said ready.</p><p>The question this series keeps asking (<em>what would prove this?</em>) turns out to be recursive. It applies to the model’s claims, then to the pipeline’s claims about the model, then to the checklist’s claims about the pipeline. That is not a reason to remove the checklist. It is a reason to treat it like everything else in the system: versioned, audited, professionally distrusted.</p><h4>A red light is also a claim</h4><p>And the discipline is symmetric: it owes nothing to the color of the light.</p><p>One morning, a mobile regression suite reports a pass rate bad enough to start a small panic. The obvious reading: the agent broke. The artifacts say otherwise. A meaningful share of the failures are infrastructure: device sessions that never started, runs that died before executing a single step. Noise wearing a failure’s clothes. Segregate those, and the same suite tells a completely different story; the team stops debugging an engine that was never broken.</p><p>A red light needs an artifact exactly as much as a green one. The cleanest design I know here keeps two records on purpose. The first is the agent’s own run summary, always a little optimistic, the way self-assessments are. The second is an independently derived verdict. And the design says out loud that only the second one is the source of truth.</p><blockquote>The agent’s self-assessment is an input to the verdict. It is never the verdict.</blockquote><h4>An instruction is also a claim</h4><p>I want to close with a mistake I left behind a long time ago, because I suspect it is the most common one in the field right now.</p><p>Every time an agent does something wrong, the first instinct is to write a better instruction. A new rule in the skill file, a firmer paragraph in the system prompt: <em>always verify, never invent, record everything. </em>The file grows. Each line is the fossil of a past incident, and for a while it feels like engineering: a fix has shipped, after all. It is written down.</p><p>But an instruction is a wish addressed to the claimant. The model reads “always record what you sent” with the same enthusiasm as everything else, and drops it the moment the task pulls its attention elsewhere. I watched my own carefully worded rules get skipped. Not maliciously, not even noticeably. Just quietly deprioritized, the way meta-work always is when real work is waiting.</p><blockquote>The uncomfortable part was admitting why I kept doing it anyway. Writing a sentence costs a minute; building a gate costs a week. I chose the sentence, called it governance, and got what I paid for.</blockquote><p>And notice that this essay’s own test applies here too: an instruction is a claim about future behavior, with no artifact behind it. The rules that actually held were the ones that stopped being sentences. The record written whether the model remembers or not. The gate that runs whether the model agrees or not.</p><blockquote>If your evidence discipline lives in a markdown file, you do not have evidence discipline. You have a hope, nicely formatted.</blockquote><h4>What evidence is</h4><p>None of this requires a better model. It never did. What it requires is a system that writes down what actually happened, and refuses to confuse that record with anyone’s opinion of it, including its own.</p><p>So, after five posts, the answer:</p><blockquote><strong>Evidence is a signal that is independent of its claimant</strong>, compared against an expectation declared before the run, traceable to its source, and carrying its own uncertainty.</blockquote><p>Every clause of it was paid for somewhere above. <em>Independent of its claimant</em>: on the day we looked, the claimant was the last to know. <em>Declared before the run</em>: an expectation written afterwards is a question answering itself. <em>Traceable to its source: </em>the most honest value an expectation ever carried was <em>default</em>. <em>Carrying its own uncertainty</em>: a confidence field nobody reads is theater, and a gap nobody declares fails silently.</p><p>Everything else is a green light.</p><h4>Where the series ends</h4><p>This series started by arguing that agentic AI is more orchestration than model; then that orchestration is empty until you define the role of the data flowing through it; then that success is a claim, and output a candidate until a chain is built around it. This post closed the loop: the chain was always for one thing, turning claims into evidence. I expect the structure to change hands. More of the gates will be designed, built and run by agents themselves, and the independent verifier will increasingly be another model rather than deterministic code. But the principle survives the handover: whoever builds the chain, the record stays outside the claimant.</p><p>One claim the series never fully interrogated is the human approval sitting at the end of the chain. Whether that step can quietly become a ritual of its own deserves a post of its own. Another day.</p><p>The series ends here: not because the problem is solved, but because the frame feels complete. I would rather stop at a complete frame than stretch it thin.</p><p>Thanks for reading along.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e6883bcd12d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Agentic AI, the Output Is Not the Evidence]]></title>
            <link>https://medium.com/@yigit.tas/in-agentic-ai-the-output-is-not-the-evidence-865b06a8e3f4?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/865b06a8e3f4</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 11:10:12 GMT</pubDate>
            <atom:updated>2026-07-01T11:11:41.555Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQRqsqIHl9SQCQP_HktNuQ.png" /><figcaption>An agent producing something and a system leaving behind an auditable trail are not the same thing.</figcaption></figure><blockquote>An agent producing something and a system leaving behind an auditable trail are not the same thing.</blockquote><p>We tend to put too much weight on the final output in agentic AI. A workflow finishes, the run looks successful, the task seems closed. But output is not evidence. Output is the visible end point of a process. Evidence is the chain around that output: where it came from, what it rests on, where the uncertainty sits, who approved it, how it was executed, and how it can be inspected afterward.</p><p>In the last three posts I argued that agentic AI is more about <a href="https://medium.com/@yigit.tas/agentic-ai-is-more-orchestration-than-model-a24a86f4b28d?sharedUserId=yigit.tas">orchestration than the model</a>, that orchestration alone isn’t enough unless you <a href="https://medium.com/@yigit.tas/in-agentic-ai-orchestration-isnt-enough-you-also-need-to-define-the-role-of-data-99e090ccb520?sharedUserId=yigit.tas">define the role of data</a>, and that a <a href="https://medium.com/@yigit.tas/in-agentic-ai-success-is-not-evidence-d95a1a50488a?sharedUserId=yigit.tas">success claim isn’t evidence on its own</a>. This piece pushes that last idea further. It isn’t about how a system works — I might save that for another post. It’s about a more basic question: when should we trust an agentic system, and what does that trust actually rest on?</p><p>One of the systems I’ve been working on has the code name Mortis. I don’t think of it as a test case generator. I think of it more as an orchestration layer for QA work: a way to connect generated candidates, human review, execution, and auditability into the same chain.</p><p>Anything generated by an agent is only a candidate until a chain is built around it. That’s where trust begins.</p><h4>The Wrong Question</h4><p>This is the question trust actually turns on. The question we keep asking in agentic AI is: “Did the agent finish the task?”</p><p>For simple demos, that might be enough. But it falls short in QA, in production workflows, in compliance, in anything that touches the customer experience. The better question is:</p><p>“Once the agent says the job is done, what did it actually leave behind that can be inspected?”</p><p>If all that’s left is a piece of generated text, a success message, or a green status, the system is asking us to trust the output itself. That’s not enough. What matters is not the output on its own, but what it’s actually grounded in.</p><h4>Something Generated Is Not Automatically Evidence</h4><p>A generated test case can look good at first glance. The title is clear, the steps are there, the expected result is written, maybe there’s even a runnable request. But the real questions are still unanswered.</p><p>Which requirement does this case cover? Was that requirement explicit in the ticket, or inferred? What assumption did the model make? What was left as an open question? Did a human review it? Was it actually run? If it passed, what exactly passed?</p><p>Without those answers, what was generated is an artifact, not evidence. Agentic systems can produce very convincing artifacts. An output can look complete while the requirement it rests on is still unclear. That’s why what matters is not the output itself, but the system built around it.</p><h4>A Candidate Should Report Its Own Uncertainty</h4><p>This is the first principle worth pulling out on its own: a candidate should report its own uncertainty. “This case rests on an assumption.” “This point is an open question.” “I’m not sure about this.”</p><p>These aren’t weaknesses in the system. They’re part of the evidence. An output that hides its own uncertainty is less trustworthy than one that shows it, not more. A system that only ever reports success has, in effect, stopped measuring anything.</p><h4>The Model Should Not Be the Source of Truth</h4><p>One thing that became clear while working on this: the model should not be the source of truth.</p><p>The model can produce candidates, help interpret a ticket, suggest tests, explain assumptions. But what carries the structure and the trail of the work should be the orchestration layer. The model isn’t just asked to write a good test. The system also asks: what does this case claim to cover, what uncertainty does it carry, how can it be checked?</p><p>That shift matters. The output stops being a final answer and becomes a candidate the system can evaluate.</p><p>In practice, the model is also part of doing that evaluation — in steps like extracting requirements or marking what a case covers, the model can play a role too. So saying “the model shouldn’t be the source of truth” doesn’t mean taking the model out of the chain. It means logging every output of the model as a separate, traceable step, and anchoring it, wherever possible, to a deterministic check rather than the model’s word. The goal is to stop the model from being the source of truth and instead make its decisions auditable.</p><h4>Instruction Is Not Evidence</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ybpkzc1vlGJKNKnF7qgG_g.png" /><figcaption>Instruction Is Not Evidence</figcaption></figure><p>It’s easy to write in a prompt: “Cover all the acceptance criteria.” “Don’t make up missing information.” “Don’t forget the negative scenarios.”</p><p>But instruction is not evidence. A model can still miss a requirement even after being told to cover all of them. It can write a confident expected result for an ambiguous contract even after being told not to make things up. It can repeat the same scenario under different titles even after being told to vary them.</p><p>That’s why the system can’t stop at instruction. It needs verification. The gap between hoping an instruction was followed and actually measuring it is the gap between a demo and a system.</p><p>One concrete example: telling a model to make each case try a different condition is an instruction. Measuring whether the generated scenarios actually differ from each other — not just their titles, but the request they fire and the assertions they check — and rejecting the output if they don’t, is evidence.</p><p>This is exactly the kind of check I’ve been trying to build into Mortis. At one point, we had nine cases with nine different titles, all firing the same request with the same assertions underneath. The check that caught it didn’t look at the titles at all — it compared the actual requests and expected outcomes across cases and flagged the duplicates. The titles had changed. The behavior hadn’t.</p><p>The principle holds either way: telling a model the right thing to do doesn’t prove it did it. The proof is in the check itself.</p><p>Sometimes the right output of that check isn’t “everything is covered.” Sometimes the right output is: “this requirement is ambiguous,” “this expected result isn’t backed by the ticket,” “this scenario was generated but doesn’t actually prove the requirement.” These aren’t failures. They’re part of the evidence trail.</p><h4>Human Approval Is Part of the Chain</h4><p>In agentic AI conversations, human approval is sometimes seen as friction or a limitation. I think that’s wrong. In work that requires accountability, human approval isn’t a weakness in the system. It’s part of the chain.</p><p>A generated case is a candidate. It isn’t accepted just because an agent produced it. A human has to review it: approve, reject, or ask for it to be regenerated. That decision is what separates generation from acceptance. If an agent can move from generation to publication without a review boundary, the system becomes less trustworthy.</p><p>But there’s a trap here, and this piece’s own logic has to apply to approval too: approval can easily become theater. If an agent produces forty convincing cases, a reviewer who can’t go through all of them one by one might just click “approve all.” At that point, approval isn’t evidence. It’s a formality.</p><p>So approval is also a claim, until what it’s based on can be inspected. What did the reviewer actually look at? Which assumption did they see and approve anyway? Did a low-confidence case slip through unquestioned? Autonomy without accountability isn’t maturity. But an approval that can’t be audited isn’t accountability either — both are failures of the same thing: auditability.</p><h4>Execution Compares the Claim Against Reality</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9GMnjIJVXjGvARY37nfiJQ.png" /><figcaption>Execution Compares the Claim Against Reality</figcaption></figure><p>The chain doesn’t end after approval. A test case, before it meets reality, is just a suggestion.</p><p>On the API side, the system shouldn’t just say “passed.” It should keep a record of what actually happened: which request was sent, which headers were there, what status was expected, what came back, which assertions were checked, why it passed or failed. A passing status code isn’t enough on its own. An endpoint can return 200 while the field you expected isn’t in the response body. The response shape can have changed. Or the test simply checked too little to fail.</p><p>This is where evidence becomes concrete — not in the green badge, but in the inspectable request and response trail behind it. A test passing doesn’t mean it proved something. It only means it didn’t break this time. What separates the two is the trail it leaves behind.</p><h4>This Chain Has a Cost</h4><p>This chain isn’t free. Keeping track of assumptions, open questions, requirement links, the approval trail, and execution evidence is real overhead. Building this chain for a hackathon demo or a throwaway exploration script is a waste. In that context, the output really is enough.</p><p>The chain earns its place where the cost of a mistake is higher than the cost of tracking it: a regression suite that ships with a release, a flow under compliance, a system that touches the customer. The point isn’t evidence for everything. Obsessing over process is just as wrong as obsessing over output. The point is making trust auditable where trust is actually required.</p><h4>Not an Artifact, a Chain</h4><p>If we want to trust agentic systems, we shouldn’t look at what they produce. We should look at what they leave behind. Can the input be inspected? Are the assumptions visible? Can it be traced back to a requirement? Can the generated work be reviewed, and can the review itself be audited? Can it be run, and can the result be inspected? Can the decision be audited?</p><p>If the answer is no, the system can still be useful — but its success is just a claim, not evidence. That chain doesn’t make agentic systems less autonomous. It makes them more accountable. And in a lot of real workflows, what turns an impressive demo into a trustworthy system is exactly that: accountability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=865b06a8e3f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Agentic AI, Success Is Not Evidence]]></title>
            <link>https://medium.com/@yigit.tas/in-agentic-ai-success-is-not-evidence-d95a1a50488a?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d95a1a50488a</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Fri, 19 Jun 2026 09:13:24 GMT</pubDate>
            <atom:updated>2026-06-30T20:34:08.538Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mLh0W3u1ouDkeaNBk3e84Q.png" /></figure><p><em>A green check is only a claim — until the system can prove it was earned.</em></p><p>In the last two posts I argued that agentic AI is less about the model and <a href="https://medium.com/@yigit.tas/agentic-ai-is-more-orchestration-than-model-a24a86f4b28d">more about orchestration</a>, and that <a href="https://medium.com/@yigit.tas/in-agentic-ai-orchestration-isnt-enough-you-also-need-to-define-the-role-of-data-99e090ccb520">orchestration isn’t enough</a> unless the role of each piece of data is clearly defined.</p><p>Both arguments leave one question unanswered:</p><p>Can you trust the system?</p><p>I didn’t arrive at that question through theory. I arrived at it by building Mortis — a QA orchestration layer that reads a ticket, drives test-case generation, runs the cases, and brings the result, with its evidence, back to a human for sign-off. Mortis doesn’t generate the cases itself; it connects generation, execution, and approval. And it taught me the hard way that in agentic systems, a “success” is not evidence. It’s only a claim waiting to be verified.</p><p>The most dangerous output is a green check</p><p>We stamped Mortis with two words: evidence-first, human-accountable. Not slogans — scar tissue.</p><p>Because in QA, the most dangerous thing an agent can produce is a green check that verified nothing.</p><p>An agent says “task completed.” A test reports “passed.” A summary says “9 cases, 9 passed.” Every one of those is a claim. None of them is evidence. And if the system treats a claim as verified state, you don’t get an agent that tests your software — you get an agent that manufactures false confidence at scale.</p><p>That’s the noise of false success. It looks like progress. It’s the opposite.</p><h3>Three traps</h3><p>Here’s the key: the most dangerous lies aren’t born in execution. They’re born in generation. The generation layer emits a claim — a case, a confidence, assumptions, open questions — and if that claim is already a lie at birth, execution will faithfully run it and hand you a clean green check. Three we actually lived through:</p><p>1 — Coverage that’s one test in nine costumes.<br>Nine cases for a pagination endpoint: “valid offset,” “offset beyond limit,” “negative offset,” “bad auth header.” Nine titles. Nine green checks.<br>But all nine fire the same request — offset=1 — and all nine return 200. The titles changed; the behavior under test didn&#39;t.<br>Nine green checks running the same request isn&#39;t coverage. It&#39;s noise wearing a uniform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0IHAIvV9oXr-7PMiBGzXeA.png" /></figure><p>2 — A “pass” against an invented expectation.<br>A test passes when the actual result equals the <em>expected</em> one. So what if the model simply invents the expected result to make the numbers look good?<br>“This should return 200.”<br>Green. Except nothing in the ticket ever said 200 was correct.</p><p>3 — A request reverse-engineered from a title.<br>The ticket body has an explicit, copy-paste cURL. The model ignores it, infers a request from the title instead, and reports success.<br>The title was treated as evidence. The title was never evidence.</p><p>In all three, the roles were labeled correctly. There was a “request,” an “expected status,” a “result.” The structure from the second post was intact — and the output was still untrustworthy.</p><p>Labeling a signal doesn’t validate it.</p><p>And the fix isn’t a smarter model. A better model might reduce some of these mistakes; but without provenance, gates, and an audit trail, the claim still leaks into state — because the problem isn’t only intelligence, it’s structure.</p><p>Which brings us back to the first post: it was never about the model.</p><h3>Trust is a promotion ladder, not a property</h3><p>So the fix is to never let a signal climb a level it hasn’t earned. In Mortis, every signal climbs the same ladder, and every rung is a gate:</p><pre>ticket            (observation — raw, unverified)<br>  → readiness    gate:  is there enough information at all?<br>  → generation   → case + confidence + assumptions + open questions (claim born)<br>  → quality gate gate:  executable request? is the expected status grounded?<br>                          duplicate request? title-inference risk?<br>  → human approval gate:  no silent publish — the human decides<br>  → execution    →  executed request, expected vs. actual, response, duration (evidence)<br>  → audit trail  →  durable, traceable state</pre><p>The observation / evidence / claim / state distinction from the second post is still here — but now every transition carries a verification boundary.</p><p>A claim is born in generation; the gates decide whether it&#39;s allowed to climb — but it only becomes evidence when it&#39;s actually executed. Nothing is promoted silently.</p><p>And the gates are deliberately blunt: if there’s no executable request, publish is blocked; if there’s no real expected status, blocked; if several cases share the same request fingerprint, blocked; a request inferred from the title while the body holds an explicit cURL cannot pass silently.</p><p>The rule we keep returning to is almost rude in its simplicity: UI, logs, and process memory are not evidence on their own. Every critical step must be backed by a database record or an audit trail.</p><p>A summary that says “passed” is not the truth. The executed request, the resolved URL, the expected-versus-actual status, the response, the duration — that is the truth. Everything else is a claim waiting to be verified.</p><h3>You can’t gate what you can’t read</h3><p>This is the part that ties directly back to the second post. These gates are only possible because every signal is forced into a structure a tool can read unambiguously.</p><p>Not prose. Not “context.” Typed, machine-readable fields:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*juQTnMpovKWOGTXqcpq8hw.png" /></figure><p>Some of these fields exist only to support trust. requestSource says where the request actually came from. evidenceSource says what the expectation is grounded in. titleRole says, explicitly: the title is a label, not a source of truth.</p><p>The system can now reason not just about content, but about provenance.</p><p>And at the run level, the structure carries its own quality signals:</p><pre>duplicateRequestCount, missingExpectedStatusCount, nonExecutableCaseCount</pre><p>This is how the first trap dies: the system computes a fingerprint for every case — method + url + query + headers + body + expectedStatus — and if nine cases collapse to a single fingerprint, the counter lights up and the gate refuses. The false success never reaches the human.</p><p>Defining the role of data wasn’t the goal. It was the precondition for verification.</p><p>You can’t gate what you can’t read; you can’t trust what you can’t trace.</p><h3>Uncertainty is a first-class output</h3><p>Here’s the subtle move that separates a trustworthy agent from a confident liar.</p><p>When the agent doesn’t know the expected result — when the ticket genuinely doesn’t say — it is forbidden from guessing. It does not emit a green check. It raises an open question, lowers its confidence, and pushes the decision up to a human.</p><p>A system that can say “I’m not sure, and here’s exactly why” is more trustworthy than one that is always green.</p><p>False certainty is the real failure; visible uncertainty is a feature.</p><h3>If a status code can lie, imagine a screenshot</h3><p>So far I’ve told this through API testing, because that’s where lies are easiest to catch: a status code is either the expected one or it isn’t. Clear, measurable, less disputable.</p><p>But Mortis doesn’t only run APIs; it generates and runs web cases in the same language.</p><p>And here’s the real point: if a status code can lie, imagine a screenshot. “The page looked right” is a far slipperier claim than a status code. There, “passed” is subjective — and because it’s subjective, trust demands more gates: a judging layer that decides whether a finding is a real bug or noise, a deduplication step that stops the same defect from being counted again and again, and a threshold-based acceptance decision.</p><p>Same ladder, higher stakes.</p><p>Because the more subjective the success, the more dangerous it is to promote an unverified claim.</p><p>How Mortis measures trust in that subjective space — confidence scoring, how it extends to mobile — is a story for another post. But the ladder stays the same.</p><h3>Trust is traceability — and the last gate is human</h3><p>So what does Mortis ultimately trust?</p><p>Almost nothing on faith.</p><p>It trusts what it can trace.</p><p>That’s also why its authority is deliberately bounded. Mortis runs tests, posts comments, logs bugs, gathers evidence. But it cannot change a ticket’s status, and it cannot sign off. Those two actions stay behind a human gate, permanently.</p><p>This isn’t timidity. Trust doesn’t come from how convincingly the model talks; it comes from the traceability of the path a signal took to become a decision.</p><p>The model proposes. The evidence accumulates. The audit trail makes it inspectable. And a human — looking at executed requests and real status codes, not at a confident summary — makes the final call.</p><p>It’s not enough for an agent to say the right thing.</p><p>The signal behind it has to earn the right to be acted on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d95a1a50488a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In Agentic AI, Orchestration Isn’t Enough: You Also Need to Define the Role of Data]]></title>
            <link>https://medium.com/@yigit.tas/in-agentic-ai-orchestration-isnt-enough-you-also-need-to-define-the-role-of-data-99e090ccb520?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/99e090ccb520</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 16:25:09 GMT</pubDate>
            <atom:updated>2026-06-09T16:25:09.481Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="In Agentic AI, Orchestration Isn’t Enough" src="https://cdn-images-1.medium.com/max/1024/1*AMxS34Q5qAUOr2egw9MhqQ.png" /><figcaption>In Agentic AI, Orchestration Isn’t Enough</figcaption></figure><p>In my <a href="https://medium.com/@yigit.tas/agentic-ai-is-more-orchestration-than-model-a24a86f4b28d">previous post</a>, I described a “snapshot” as the short, normalized state package the agent looks at at the moment of decision. In this post, I want to open up another side of that idea: a snapshot shouldn’t be thought of as a single-piece context, but as a decision surface made up of data pieces that take on different roles.</p><p>If we expect a model in a system to both understand data, construct reality, interpret policy, and produce the right decision, the problem usually isn’t the model — it’s how the system presents the data.</p><p>This isn’t about prompts, markdown files, or additional instruction sets; it’s a more fundamental data modeling issue. The system needs to be able to distinguish between observation, evidence, claim, state, policy, and context.</p><p>Because in many companies, the word “context” is used to cover almost everything. Logs, documents, test results, execution metadata, user actions, previous model responses — all thrown into the same bag and labeled “context.”</p><blockquote>If everything is context, then nothing is truly context.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0sPcR0HTX9m9DFfXNHXEdw.png" /></figure><p>In agentic AI systems, raw data, evidence, claim, state, policy, and context are not the same thing. Each plays a different role at a different layer of the system. That’s why, before asking “what data should we give the model?”, a more fundamental question needs to be asked: what role does this data play within the system?</p><p>I’m not writing this to propose a framework. I’m writing this because I increasingly see, in practice, that what we call “context” contains pieces playing distinct, different roles from one another.</p><p>In a multi-agent flow, an agent may have returned “task completed” for a previous step. At first glance, this looks like clear information. But from the system’s perspective, the question is: is this actually workflow state, or just a log line produced by the agent? Was the output actually consumed by the next agent? Is triggering the next step safe from a policy perspective?</p><p>The word “completed” alone is not state; depending on which layer of the system it was produced in, it could be a log, evidence, or workflow state. But most of the time it’s none of these — it’s just a claim. “Task completed” can be a claim; whether the system actually validates it is a separate question.</p><p>The same signal can be represented in different roles: the “task completed” output can be a log line. The output actually consumed by the next agent can be evidence. The state maintained by the workflow engine can be state. Whether the next step can be triggered or not can be policy. The decision package given to the agent at that moment can be context.</p><p>If we call all of these simply “context,” we obscure which part of the system is making decisions based on what.</p><p>In a raw approach, the agent is given a lot of data, but the roles of that data are ambiguous. In a role-defined approach, the agent is given less data but a structure that is more meaningful for the decision. Instead of trying to construct the world from scratch, the model reasons over a clearer decision surface.</p><p>In practice, the starting point can be simple: before writing agent outputs directly to state, ask “is this a log, evidence, or a claim?” and type it accordingly.</p><p>Giving the agent more information feels like it should yield better answers. But in agentic systems, the boundaries of context determine the quality of decisions.</p><p>When we don’t make these distinctions, the model tries to understand the world before making a decision: which information is current, which is just a historical record, which is evidence, which is a policy constraint, which is a previous model interpretation? This sometimes works. But as the system grows, it becomes brittle.</p><p>In company-internal agentic setups, the problem is usually not that the model fails to answer. The problem is that it becomes unclear what the model is basing its answer on.</p><p>The agent doesn’t need to see everything. It needs to see the right thing, in the right role, at the right moment of decision.</p><p>This is not an entirely new problem. In distributed systems, workflow engines, and data architecture, we’ve long wrestled with similar distinctions: what is an observation, what is state, what triggers a decision, what merely stays as a record? In the LLM-agent world, this problem has simply returned wearing a new face. Because what appears to be a prompt engineering problem is, most of the time, a data architecture and system design problem.</p><p>Raw data is what the system sees. Evidence is the supporting basis. Claim is what the system or model asserts. State is continuity. Policy is the boundary. Context is the slice of reality selected at the moment of decision.</p><p>Canonical representation is a common language in which these pieces are defined within the system: a structured representation that explicitly carries each signal’s type, source, and the decision it supports. This can sometimes be a JSON schema, sometimes metadata tagging, sometimes a typed event/state model maintained by the workflow engine.</p><p>A simple example makes this concrete. Take the “task completed” signal from earlier. In a raw approach, this is just a string in a log. In a canonical representation, it looks more like this:</p><pre>{<br>&quot;signal&quot;: &quot;task_completed&quot;,<br>&quot;role&quot;: &quot;claim&quot;,<br>&quot;source&quot;: &quot;agent_output&quot;,<br>&quot;validated_by&quot;: null,<br>&quot;supports_decision&quot;: &quot;trigger_next_step&quot;,<br>&quot;timestamp&quot;: &quot;2026–04–01T10:42:00Z&quot;<br>}</pre><p>The signal hasn’t changed. But now the system knows what role it plays, where it came from, and what decision it is meant to support. More importantly, it knows what it doesn’t yet have: validation. That absence is itself information.</p><p>Agent systems scale not with larger contexts, but with better-defined information layers. That’s why good orchestration begins not with the model, but with the system clarifying what it actually accepts as context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99e090ccb520" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI Is More Orchestration Than Model]]></title>
            <link>https://medium.com/@yigit.tas/agentic-ai-is-more-orchestration-than-model-a24a86f4b28d?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a24a86f4b28d</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Thu, 02 Apr 2026 07:40:05 GMT</pubDate>
            <atom:updated>2026-04-02T07:40:05.680Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LM0MJeR3vRn5WkcSmMAqkQ.png" /></figure><p>What makes an agentic system truly useful is not the model itself, but how its state, policy, fallback, and control layers are designed.</p><p>Discussions about Agentic AI quickly drift toward the model. Which model performs better? Which prompt works better? Which LLM is stronger? From the outside, this is understandable. After all, the most visible part of an agentic system is the model’s output.</p><p>We thought in a similar way at first. If there is a layer producing output for us, then model selection must be the most critical decision. But once we actually started building and operating the system, we saw something different. The problem was not just giving prompts and getting output. The real challenge was building a system that could operate under real conditions, make decisions, take action, and avoid losing control when things went wrong.</p><blockquote><strong>Agentic AI is more of an orchestration problem than a model problem.</strong></blockquote><p>A simple “here’s the prompt, send it to the LLM and let it handle the rest” approach is not enough to build an agentic system. The system has to observe, maintain state, generate context, know when a decision needs to be regenerated, choose how to interpret what it sees, fall back when necessary, and do all of this in a traceable way.</p><p>The real challenge was not what the model would say. The real challenge was defining when, in what context, and within which boundaries the model should be allowed to step in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDkstjzc8Llb1DqyN8UaqA.png" /></figure><p>Instead of placing the model at the center of the system, it needs to be positioned in a controlled way at the most uncertain decision point, the point that actually requires human-like judgment. What makes a system truly agentic is not its ability to generate answers, but its ability to establish a controlled decision loop and turn that loop into action.</p><p>That is exactly what I want to unpack in this article: why a model-centric view falls short, why context and state become decisive, why deterministic and policy layers are critical, and why fallback and observability are essential parts of building trust.</p><h4>Why We Needed an Agentic Layer</h4><p>The reason we needed an agentic structure was not really about the model. It came from the needs of the system itself.</p><p>The problem was not just collecting data or looking at summaries. We needed the system to read changing context, generate decisions when needed, update those decisions when necessary, and do all of that in a controlled way. That is exactly where the need for an agentic layer came from.</p><blockquote>In other words, the need was not for “a model that writes better sentences,” but for “a system that produces more reliable decisions under changing conditions.”</blockquote><p>What made this necessary was not simply that we wanted better output. The real issue was that as the system changed, the same interpretation stopped being sufficient. A static summary or a one-off model answer quickly loses value as conditions change. That is why what we needed was not just a model that produces answers, but a decision layer that can adapt as context shifts.</p><p>Once that distinction became clear, the next question naturally followed: if the problem is not just the model, then what is the model’s real role inside the system?</p><h4>Starting with the Model Feels Natural, but It Is Incomplete</h4><p>When approaching the idea of Agentic AI, it is very easy to focus on the model. After all, that is the layer that creates the language of decision-making, generates text, and appears as “intelligence” from the outside.</p><p>At first, the model’s fluent and fast outputs were highly convincing. Interacting with the model directly and getting quick results made it feel like long and difficult problems could be solved rapidly. Those fast outputs and the apparent disappearance of complex problems felt almost dreamlike.</p><p>But when we started evaluating the same system flow repeatedly in real operating conditions, we saw that the main difference came not from model quality, but from the quality of the context provided to the model. The same model behaved superficially with weak context, and much more consistently with strong context. That showed us that the core problem was less about model selection and more about context design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NQJH-TzOkvhjJpJ5VHSikg.png" /></figure><p>The quality of a model’s response does not depend only on the prompt or the choice of model. What really determines the result is what context the model receives, what data feeds it, and within what frame the decision is generated.</p><p>At first, we thought the combination of “the right prompt + a good model” might be enough. But that approach is not sufficient for real system behavior. For the model to produce a meaningful decision, the system first needs to clarify what it sees, what matters, and what it accepts as decision-ready context.</p><p>Without that clarity, the model produces answers that may be well-written but have limited operational value. In that case, even if the system can still get an answer from the model, it cannot get the level of control, consistency, and efficiency it actually needs.</p><p>But there is one more step here: what data should be given to the model, and how should it be given?</p><h4>Why Feeding Raw Data Directly into the Model Does Not Work</h4><p>The second major lesson we learned in practice was this: feeding raw data directly into the model does not produce good results.</p><p>The first reason is noise. In theory, it is possible to pass every metric, every time series, every event, and every movement from the system into the model. But that does not automatically improve decision quality. Most of the time, it does the opposite by blurring the context. The model can focus on the wrong signals, mistake secondary details for primary ones, and struggle to produce clear operational decisions.</p><p>We saw this very clearly in practice: when we gave the model more signals, we expected decision quality to improve automatically. But in many cases, the model started focusing on secondary movements instead of the truly decisive signals. That made it obvious that giving more data does not mean giving better context.</p><p>The second reason is cost and latency. Sending raw data to the model as-is quickly becomes expensive and inefficient.</p><p>This is where one concept became critical: the <strong>snapshot</strong>. Instead of feeding raw data directly into the model, it is far more effective for the system to first generate a meaningful summary internally. That significantly improves decision quality.</p><blockquote>A snapshot works as a normalized data package representing the state at a given moment, with its context narrowed down and prepared specifically for decision-making.</blockquote><p>This may sound like a small detail, but it makes a big difference. Because the model is no longer looking at a messy pile of data. It is looking at a context layer prepared for decision-making.</p><p>And that brings us to the next point: narrowing context is important, but it is not enough on its own. Because the system needs to know not only what is happening now, but also how it got here.</p><h4>You Cannot Build Agentic Behavior Without State</h4><p>If we are talking about an agentic structure, then the system needs to know not only the present moment, but also the past.</p><p>That is why the state layer, the memory structure that carries the system’s current condition and recent history, became far more important than we initially thought.</p><blockquote>This matters not only for carrying context, but also for preventing corrupted context from poisoning the flows that come next.</blockquote><p>This is not a theoretical idea. It is a real need that emerges from the behavior of the system itself. If the system only sees “the current moment,” then the suggestions it produces inevitably remain shallow. Because decision quality is determined not just by current signals, but by how those signals have changed, what was seen before, and where the system currently sits within its own decision flow.</p><p>We saw this especially clearly in broken flows. A single false assumption or an incomplete context, if carried forward by the system and not isolated, can influence later decisions as well. At that point, the issue is no longer just producing one wrong interpretation. It becomes a chain reaction in which wrong context poisons future decisions too.</p><p>State, in this sense, is not just memory. It is also decision memory and a mechanism for isolating corrupted context.</p><p>Without memory, the model’s output remains a one-off interpretation. Once memory is added, the decision becomes part of a continuous loop.</p><p>And that is the real turning point: Agentic behavior does not emerge from one-off generation. It emerges from a decision loop that carries state and establishes continuity.</p><p>The next question, then, is this: how should a system that carries context and state develop its own technical interpretation layer, independent of the model?</p><h4>The Deterministic Layer Is Not an Alternative to the Model, but Its Complement</h4><p>One common misconception in discussions about Agentic AI is to think of the deterministic layer and the model as if they were alternatives to each other.</p><p>The deterministic layer does not replace the model. It helps the model operate in a more controlled, more consistent, and more explainable way.</p><p>By deterministic layer, we mean the rule-based, explainable interpretation and filtering layer that works independently of the model.</p><p>Before the model steps in, the system must first be able to answer the question: <strong>“What am I technically seeing right now?”</strong> It needs a layer that narrows the problem space, sets priorities, and filters out meaningful signals.</p><p>At a certain point, we realized something important: whenever the system did not first perform its own technical reading, the model was given too much freedom. As a result, instead of interpreting the right signal, it sometimes produced language that sounded more fluent but reflected weaker decisions. The deterministic layer started creating value exactly at that point. It did not replace the model, but it placed the model inside a narrower and more reliable decision space.</p><blockquote>Only after that does the model become genuinely useful on top of a more structured and meaningful context.</blockquote><p>This approach gave us two important benefits: the ability to produce more consistent decisions and a stronger link between the decision that was generated and the evidence supporting it.</p><p>In other words, instead of letting the model run freely, placing it inside a more meaningful context improves the reliability of the system.</p><h4>The Policy Layer Is Part of Decision Quality</h4><p>In agentic systems, what matters is not only <strong>what</strong> should be generated, but also <strong>when</strong> it should be generated.</p><p>In a living system with continuous data flow, the policy layer, the rule structure that determines when a new decision should be generated, becomes just as critical as the model itself.</p><p>If the system automatically calls the model during every evaluation, it quickly becomes both expensive and noisy. Re-generating decisions again and again for the same context does not make the system look smarter. In many cases, it makes it more unstable and less decisive. A good agentic system must be able to answer not only “What should I say?” but also “Do I really need to generate a new decision right now?”</p><p>The key question here is whether the system can tell if new data actually creates a need for a new decision. Has the context truly changed? Has the current decision become outdated? Is generating a new interpretation really necessary? These are not just technical optimization details. They directly shape decision quality.</p><p>We saw this very clearly in practice: when the context had not changed in a meaningful way, making the system speak again did not produce better decisions. On the contrary, instead of reinforcing the previous decision, it produced new variations and weakened consistency. Small fluctuations do not always require a new decision. If the new data does not meaningfully change the existing decision, forcing the system to keep generating new ones does not just increase cost, it also undermines decision stability.</p><p>This is where the real role of the policy layer becomes clear: it separates noise from real change. That is why the policy layer is not merely a performance optimization. It is a core part of decision quality. Good agentic systems are not only systems that know when to speak; they are systems that also know when to stay silent.</p><p>And that leads us to fallback and control.</p><h4>Fallback and Human Control Matter More Than We Thought</h4><p>One of the most important lessons we learned while building a real system was how critical resilience is.</p><p>That is why fallback, the backup decision path the system uses when the model is unavailable, is not an optional comfort. It is a required part of the system backbone.</p><p>The model can fail. There can be network issues. The provider may not respond. The context may not yet be sufficiently formed. The design itself may be flawed.</p><blockquote>If your entire structure depends only on the model, then the system goes idle precisely in those moments. That does not just create a technical failure. It also creates a loss of trust.</blockquote><p>That is exactly why the system must still have a reasonable, controlled, and explainable decision path even when the model is not in the loop. The goal is not to replace the model, but to prevent the system from falling into a void. The value of fallback is not that it produces the same quality as the primary decision path, but that it allows system behavior to continue in a safe, consistent, and understandable way.</p><p>Likewise, preserving human control instead of full autonomy on the action side is a deliberate design choice. Especially in early stages, we saw that this was a much healthier approach. Because as agentic systems become more powerful, the impact of a wrong decision also grows. That is why, for us, controllability became a more important design principle than autonomy.</p><blockquote>Reliable agentic AI systems are not defined by ideal-path performance alone, but by how controlled they remain under degraded conditions.</blockquote><h4>Without Observability, Trust Does Not Emerge</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QfKCTqxGglVGD_d7zAZM3g.png" /></figure><p>It is not enough for an agentic system to work. It must also be possible to see why it worked.</p><p>Was a decision generated?<br>Why was it generated?<br>Why was it skipped?<br>What context was it based on?<br>How did the system behave when an error occurred?<br>Why did it fall back, if it did?</p><p>If the answers to these questions are not visible, then from the outside the system turns into nothing more than a closed box that “says something every now and then.”</p><p>Because when a system produces a decision but does not make the reason visible, it stops looking reliable and starts looking mysterious. When you see the same output for the second time, if you cannot answer questions like “why now?”, “based on which signal?”, and “why not earlier?”, then it becomes difficult to evaluate either the quality of the system or its limits.</p><p>That is why decision history, event flow, visibility into the current state, and traces showing under which conditions a decision was generated all became far more important than we expected.</p><blockquote>These are not just debugging tools. They are the system’s trust-generation mechanism.</blockquote><p>In agentic systems, trust comes less from the model’s persuasive sentences and more from the visibility of the decision’s context, boundaries, and history.</p><p>All the layers described so far ultimately converge on one common point: the model alone does not produce agentic behavior. What makes it reliable and usable is the design around it.</p><h3>Final Word</h3><p>Claude or OpenAI? Which model is better? Once we set those debates aside, what remains becomes very clear.</p><p>Agentic AI is not about using model calls intelligently. It is a system design problem in which observation, state, policy, fallback, and control layers work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9e7_uD9ksM6FINM2GHJUg.png" /></figure><p>The model is still important. But the model alone does not create an agentic structure.</p><p>What truly makes it agentic is how it observes, what context it carries, when it decides, when it deliberately stays silent, how it behaves under failure, and where it keeps human control in the loop.</p><p>For us, the real lesson was never “Which model is better?” It was the quality of the orchestration built around the system.</p><p>At first, we looked for better outcomes in model selection. Over time, we realized the real difference came from context and state. In the end, we saw that the problem was not simply generating decisions, but designing the conditions under which decisions are produced, updated, and controlled.</p><p>In short, the hardest and most valuable part of Agentic AI is not the model itself. It is the system design around the model that makes it meaningful, reliable, and controllable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a24a86f4b28d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Testing and k6]]></title>
            <link>https://medium.com/hepsiburadatech/performance-testing-and-k6-f650287607aa?source=rss-440042aab33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f650287607aa</guid>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[performance-test]]></category>
            <category><![CDATA[k6]]></category>
            <dc:creator><![CDATA[Yigit Tas]]></dc:creator>
            <pubDate>Thu, 12 Oct 2023 08:02:45 GMT</pubDate>
            <atom:updated>2023-10-12T08:02:45.134Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*GkR1WeQ8by6bSxWpgwSnVg.jpeg" /></figure><p>Performance testing is a test methodology that evaluates the responsiveness and stability of a server, network, or application under a load. This method, commonly applied as a load test, can easily be gathered under the name of Performance Testing. When evaluating companies as a life problem, having a healthy service delivery time, which is now evaluated as up-time rather than the volume of work done, becomes important. To keep this time stable and at high scores, we need to know our performance issues. This is where k6 comes in.</p><h3><strong>What is k6? Advantages?</strong></h3><p>k6 was released in 2021 through Grafana Labs. We can briefly evaluate it as a tool used for performance testing and conveying its results to us. Its development with modern technology provides advantages in certain areas compared to other competing tools. The benefits it provides can be listed as follows:</p><ul><li>Speed and Efficiency</li><li>Scalability</li><li>Flexible Scenarios</li><li>Rich Output and Reporting</li></ul><p>In the performance testing processes, there is generally a point that is overlooked. The role of resources and activity spent by the requests going to the place where the measurement will be made. One of the significant advantages of k6 compared to other common competitors is definitely the infrastructure and language it is developed with. To elaborate, JMeter is still leading in this methodology, but unfortunately, it struggles to catch up with fairly new technologies. To give a brief example, the difficulty and time cost of intervention to the scenario through PreProcessors and PostProcessors. This is where k6, developed with GO, comes into play. It makes the process quite fast and understandable with its provided CLI. Making the performance test efficiently is also very important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZFSUJSXcu2BKYeBJhV1bpQ.png" /></figure><p><strong>Source: </strong><a href="https://k6.io/blog/k6-vs-jmeter/"><strong>k6 vs. JMeter</strong></a></p><h3>k6 Concepts</h3><p>The parameters we use during performance testing and define for it to work are also encountered here. The parameters we will determine are defined in the <strong>Options</strong> object in k6. Options Determinations regarding how the test will run and the results are defined on this object.</p><ul><li>VUS (Virtual Users): The number of users used/going to be used during the test.</li><li>Iteration: How many times each determined VUS will repeat the same scenario.</li><li>Duration: How long the test will last. The value given here is in milliseconds (MS).</li><li>Checks: Our response assertion/verification controls familiar to the test world.</li><li>Threshold: The threshold value we will determine during the test. If the given value exceeds this, the test will be considered unsuccessful. For example:</li></ul><pre>thresholds: { http_req_duration: [&#39;avg&lt;150&#39;] }</pre><h3>k6 Example</h3><p>For the k6 example, we will use k6’s <a href="https://test-api.k6.io/public/crocodiles/">own APIs</a>. The simplest form of a point-load-test looks like this:</p><pre>import http from &quot;k6/http&quot;;<br>import { check } from &#39;k6&#39;;<br><br>export const options = {<br>    vus: 1,<br>    duration : 1000,<br>    thresholds: { http_req_duration: [&#39;avg&lt;150&#39;] },<br>    noConnectionReuse: true,<br>    summaryTrendStats: [&#39;avg&#39;,&#39;min&#39;,&#39;max&#39;,&#39;count&#39;]<br>};<br>export default function () {<br>    const url = &#39;https://test-api.k6.io/public/crocodiles/&#39;;<br>    const header = { headers: {<br>            &#39;Content-Type&#39;: &#39;application/json&#39;,<br>        },};<br>    const response = http.get(url,header);<br>    check(response, {<br>        &#39;is status 200&#39;: (r) =&gt; r.status == 200,<br>    });<br>}</pre><p>To better understand the k6 structure, let’s split it into two parts:</p><ul><li>The part where we specify the flow of the test and how it will behave, the options object.</li><li>The part that will be the function where the test will run.</li></ul><p>We can think of the Options object in its simplest form as a configuration.</p><pre>export const options = {<br>    vus: 1,<br>    duration : 100, // in milliseconds<br>    thresholds: { http_req_duration: [&#39;avg&lt;100&#39;] },<br>    noConnectionReuse: true,<br>    summaryTrendStats: [&#39;avg&#39;,&#39;min&#39;,&#39;max&#39;,&#39;count&#39;]<br>};</pre><p>For a simple test scenario, the above option object will be sufficient. Among the parameters inside the object:</p><ul><li>Threshold ensures that the average of the made requests is greater or less than a certain value.</li><li>NoConnectionReuse determines the keep-alive value of the test.</li><li>SummaryTrendStats determines the value formats of the output given when the test is completed.</li></ul><p>Under the function, we can now call it the place where the test will run.</p><pre>const url = &#39;https://test-api.k6.io/public/crocodiles/&#39;;<br>const header = { headers: {<br>        &#39;Content-Type&#39;: &#39;application/json&#39;,<br>    },};<br>const response = http.get(url,header);<br>check(response, {<br>    &#39;is status 200&#39;: (r) =&gt; r.status == 200,<br>});</pre><p>In its simplest form, adding a header to a get request we made and checking the status is how we perform this step.</p><h3>Scenarios and Executors Structure with k6</h3><p>In k6, an executor is a component that defines how the load will be directed during a test. With this, we can easily apply different approaches under the name of Performance Test. This way, we have a chance to control the distribution of a real user and the increase in load on the application.</p><p>As known, the methodology applied as a performance test is divided into some subheadings. Briefly:</p><ul><li><strong>Smoke</strong>: low VUs / 1–5 minutes</li><li><strong>Load</strong>: average VUs / 10–60 minutes</li><li><strong>Stress</strong>: high VUs / 1–2 hours</li><li><strong>Spike</strong>: very high / +60 minutes</li><li><strong>Breakpoint</strong>: regular VUs increase / +60 minutes</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*yh3E2GIqrNuELnNTYrxr3g.png" /></figure><p>As it is known, sometimes for companies in a life cycle, having a single-endpoint or a simple scenario may not be enough to reach the desired result. The more simple and basic the applied scenario is, the variety and probability of detecting problems actually decrease.</p><p>The biggest advantage we will gain through k6 is to control the load inside, controlled increase, and controlled decrease. This provides us an advantage in detecting application/system problems.</p><pre>export const options = {<br>    stages: [<br>        { duration: &#39;30s&#39;, target: 10 }, <br>        { duration: &#39;1m&#39;, target: 20 },<br>        { duration: &#39;2m&#39;, target: 0 },<br>    ],<br>    thresholds: { http_req_duration: [&#39;avg&lt;150&#39;] },<br>    noConnectionReuse: true,<br>    summaryTrendStats: [&#39;avg&#39;,&#39;min&#39;,&#39;max&#39;,&#39;count&#39;]<br>};</pre><p>According to the values we give, the working logic of our scenario is as follows:</p><ul><li>The duration of our test is a total of 3 minutes and 30 seconds.</li><li>At the end of the first 30 seconds, 10 VUs are started to be sent inside. Then, for 1 minute, it begins to increase the load inside from 10 to 20 VUs. At the end of 1 minute, it starts to reduce the number of VUs inside from 20 to 0 in a controlled manner for 2 minutes.</li><li>The duration values actually determine the ramp-up and ramp-down of the load we give inside.</li><li>In this way, by dividing the real intensity inside into minutes, we benefit from the stage component.</li></ul><h3>Requests per Second Control</h3><p>Another scenario we can create is RPS (Requests per Second). In this way, we can create a test structure focused on the requests per second of the scenario that occurs inside.</p><pre>export const options = {<br>    scenarios: {<br>        my_scenario1: {<br>            executor: &#39;constant-arrival-rate&#39;,<br>            duration: &#39;1m&#39;, // test duration<br>            rate: 60, // requests to be made according to the TimeUnit we give<br>            preAllocatedVUs: 30, // maximum VUs to be used for the given rate value in the test<br>            timeUnit: &#39;1s&#39;, // how long the given rate value will be applied<br>        },<br>    }<br>};</pre><p>The VUS value we give is not actually a number used at the beginning of the test; rather, as a result of the applied rate, the test is tried to be applied continuously for the given timeUnit period. If response times increase, it continues to increase the VUS inside to catch the rate value we give automatically.</p><h3>Stress Test</h3><p>In the example provided in this options object, we are actually applying a stress test methodology to our application.</p><pre>export const options = {<br>  stages: [<br>    { duration: &#39;5m&#39;, target: 200 }, <br>    { duration: &#39;20m&#39;, target: 600 }, <br>    { duration: &#39;5m&#39;, target: 0 }, <br>  ],<br>};</pre><h3>Breakpoint Test</h3><pre>export const options = {<br>  executor: &#39;ramping-arrival-rate&#39;, // parameter to control the load to the application<br>  stages: [<br>    { duration: &#39;40m&#39;, target: 4000 }, // the number of VUs to be reached at the end of the specified duration<br>  ],<br>};</pre><h3>Observing k6 Metric Results</h3><p>Rather than tracking and interpreting metrics generated during the test in text, it’s possible to visualize them on our local machine. This allows us to monitor and interpret our application data in real-time, which can help us detect potential issues during the test more promptly.</p><p>We can easily create our own metric dashboards by dumping k6’s default output metrics into dashboards using Grafana and Influx.</p><p>I’ll proceed with Docker locally. Let’s set up Influx on Docker.</p><p>Influx on Docker Let’s create a bridge on the network for Docker.</p><pre>$ docker network create --driver bridge influxdb-telegraf-net</pre><h4>Run Influx on Docker</h4><pre>docker run -d --name=influxdb \<br> -p 8086:8086 \<br> -v  /tmp/testdata/influx:/root/.influxdb2 \<br> --net=influxdb-telegraf-net \<br> influxdb:2.0</pre><p>Now, Influx is ready to run on localhost:8086. When you visit this address for the first time, it will prompt you for bucket and user information. Fill these in to complete this step.</p><h4>k6 InfluxDB Extension</h4><p>When we install k6 normally, it comes with default InfluxDB v1 support. However, in this example, we’re using InfluxDB v2, so we will use an extension.</p><pre>xk6 build - with github.com/grafana/xk6-output-influxdb</pre><p>This will create a binary that provides InfluxDB v2 support. We’ll now run our tests with this binary.</p><p>For that, we can use the test start command.</p><pre>K6_INFLUXDB_ORGANIZATION=${organizationName} K6_INFLUXDB_BUCKET=${bucketName} K6_INFLUXDB_TOKEN=${TOKEN} ./k6 run -o xk6-influxdb=http://localhost:8086 ./testScript.js</pre><p>Tests run with this environment variable will be exported to InfluxDB.</p><h4>Grafana on Docker</h4><p>You can run a Grafana instance at localhost:3000 with this command.</p><pre>docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise</pre><p>Now, Influx is running on localhost:8086, and Grafana is running on localhost:3000.</p><p>Let’s add the Influx data source to Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7EYi2K-GWAb7-O93jbTCkw.png" /></figure><p>After these steps, we’ll be able to query Influx from Grafana.</p><p>Let’s create a dashboard. We can use this template for that. With this template, we can see the basic k6 metrics.</p><p><a href="https://gist.github.com/Yiwit/cd8bae3d6b4c4101ce2faf8eebb5f228">https://gist.github.com/Yiwit/cd8bae3d6b4c4101ce2faf8eebb5f228</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YDmGTMvaCQAmwl9F6FxOg.png" /></figure><p>Now, our k6 tests are observable through Grafana.</p><p>In conclusion, performance testing is a critical test methodology that evaluates how a system responds under a load and its stability. k6 stands out as a powerful and effective tool for such tests. The advantages it provides, such as speed, scalability, flexible scenarios, rich output, and reporting, are just a few of the important factors in choosing k6. Through these advantages, users can better understand and improve the performance and stability of their applications.</p><p>The fundamental concepts, installation, usage, and different scenario structures of k6 that we have examined are a great help in effectively conducting performance tests. These features ensure the successful completion of the test and enhance the resilience of the application. Remember, correct and effective performance tests are the key to an application’s success and user satisfaction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f650287607aa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hepsiburadatech/performance-testing-and-k6-f650287607aa">Performance Testing and k6</a> was originally published in <a href="https://medium.com/hepsiburadatech">hepsiburadatech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>