<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gurudatt S A on Medium]]></title>
        <description><![CDATA[Stories by Gurudatt S A on Medium]]></description>
        <link>https://medium.com/@gurudatt.sa26?source=rss-b0c4aa941b7c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sP4mC3af6tHuX3ZLMmJd7Q@2x.jpeg</url>
            <title>Stories by Gurudatt S A on Medium</title>
            <link>https://medium.com/@gurudatt.sa26?source=rss-b0c4aa941b7c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:58:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gurudatt.sa26/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Token Budgets: The Hidden Lever Every LLM Developer Needs to Master]]></title>
            <link>https://medium.com/@gurudatt.sa26/token-budgets-the-hidden-lever-every-llm-developer-needs-to-master-cbf8ae9397b6?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf8ae9397b6</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Sun, 07 Jun 2026 12:07:05 GMT</pubDate>
            <atom:updated>2026-06-07T12:07:05.230Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A provider-agnostic guide to understanding, setting, and optimising token budgets across OpenAI, Anthropic, Google, and beyond.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzTUOd0FfGTQ15dhyYB6Eg.png" /></figure><p>You’ve integrated an LLM into your product. The demo went great. Then the first real month’s API bill arrives — and it’s three times what you projected. Or a user reports that responses are getting cut off halfway through. Or your RAG pipeline is mysteriously slow.</p><p>Nine times out of ten, the root cause is the same: <strong>you haven’t thought carefully about token budgets.</strong></p><p>This is the guide that fixes that. We’ll start from first principles and work through everything you need to run LLMs efficiently in production — regardless of which provider you use.</p><h3>01 — What Is a Token?</h3><p>A <strong>token</strong> is the atomic unit of text that a language model processes. Not a word, not a character — a sub-word chunk produced by a tokenisation algorithm (typically Byte-Pair Encoding, or BPE).</p><pre>&quot;Token budgets are powerful&quot;<br>→  [Token] [ budgets] [ are] [ power] [ful]   = 5 tokens</pre><p>The rule of thumb that holds across most models:</p><blockquote><strong><em>1 token ≈ 4 characters ≈ ¾ of an English word</em></strong></blockquote><p>But that average hides a lot of variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44whUJ-SOXEvGTiNMTxP5Q.png" /></figure><p>This matters because a 10,000-word technical document in English is ~13,000 tokens, but the same content in Japanese could be 30,000+ tokens — which changes whether it fits in your context window at all.</p><h3>How to count tokens yourself</h3><p>Never guess — measure. Every major provider offers a tokeniser:</p><pre># OpenAI / tiktoken<br>import tiktoken<br>enc = tiktoken.encoding_for_model(&quot;gpt-4o&quot;)<br>tokens = enc.encode(&quot;Your text here&quot;)<br>print(len(tokens))<br><br># Anthropic<br>import anthropic<br>client = anthropic.Anthropic()<br>response = client.messages.count_tokens(<br>    model=&quot;claude-opus-4-5&quot;,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Your text here&quot;}]<br>)<br>print(response.input_tokens)<br># Google (Gemini)<br>import google.generativeai as genai<br>model = genai.GenerativeModel(&quot;gemini-1.5-pro&quot;)<br>result = model.count_tokens(&quot;Your text here&quot;)<br>print(result.total_tokens)</pre><h3>02 — The Four Types of Token Budget</h3><p>Token budget is not one thing. There are four distinct concepts, and confusing them is a frequent source of bugs and unexpected costs.</p><h3>Type 1 — Context Window (Hard Ceiling)</h3><p>The context window is the maximum number of tokens the model can process in a single request — input <em>plus</em> output combined. It is a hard architectural limit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sx2QCgrxb-6xHCHcH7QjQg.png" /></figure><p>What eats into this window:</p><pre>[System prompt] + [Chat history] + [Retrieved docs] + [Tools/functions] + [Output] = Context window</pre><p>Exceed the limit and you get an error — or the provider silently drops the oldest messages (sliding window), which can cause the model to forget earlier instructions.</p><h3>Type 2 — max_tokens (Output Cap)</h3><p>max_tokens (or max_completion_tokens in newer OpenAI APIs) caps how long the model&#39;s <em>response</em> can be. It has no effect on input. Setting it too low is the #1 cause of truncated responses.</p><p><strong>Important:</strong> unused max_tokens cost nothing. A response of 400 tokens when max_tokens=1000 bills you for 400 tokens, not 1,000. Always give yourself headroom.</p><h3>Type 3 — Reasoning / Thinking Budget</h3><p>Several frontier models now have a dedicated reasoning phase that happens before the visible response:</p><ul><li><strong>OpenAI o1 / o3 / o4-mini</strong>: max_completion_tokens covers both reasoning and output. You can use reasoning_effort (low, medium, high) to control depth.</li><li><strong>Claude (extended thinking)</strong>: Separate budget_tokens parameter for the internal reasoning pass.</li><li><strong>Gemini 2.0 Flash Thinking</strong>: Thinking tokens are controlled via thinkingConfig.thinkingBudget.</li></ul><p>Reasoning tokens are billed as output tokens and can easily dwarf your visible response in cost. A “high effort” o1 request might spend 20,000 tokens thinking before producing a 500-token answer.</p><h3>Type 4 — Cost Budget</h3><p>At the business level, a token budget is a spend cap. Since pricing is per-million tokens (input and output billed separately), token count is your primary cost lever.</p><p><strong>Current pricing reference (as of mid-2026):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFr0qsXBnQilwV4dAjZ8zQ.png" /></figure><blockquote><em>⚠️ Always verify pricing at your provider’s official pricing page — rates change frequently.</em></blockquote><h3>03 — Setting Token Budgets: API Reference</h3><h3>OpenAI</h3><pre>from openai import OpenAI<br>client = OpenAI()<br><br># Standard completion<br>response = client.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    max_tokens=1024,<br>    messages=[<br>        {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a concise assistant.&quot;},<br>        {&quot;role&quot;: &quot;user&quot;,   &quot;content&quot;: &quot;Explain attention mechanisms.&quot;}<br>    ]<br>)<br># Reasoning model (o4-mini) with effort control<br>response = client.chat.completions.create(<br>    model=&quot;o4-mini&quot;,<br>    max_completion_tokens=8000,<br>    reasoning_effort=&quot;medium&quot;,   # low | medium | high<br>    messages=[<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Solve this multi-step optimisation problem.&quot;}<br>    ]<br>)<br>print(f&quot;Prompt tokens:     {response.usage.prompt_tokens}&quot;)<br>print(f&quot;Completion tokens: {response.usage.completion_tokens}&quot;)<br>print(f&quot;Reasoning tokens:  {response.usage.completion_tokens_details.reasoning_tokens}&quot;)</pre><h3>Anthropic (Claude)</h3><pre>import anthropic<br>client = anthropic.Anthropic()<br><br># Standard completion<br>response = client.messages.create(<br>    model=&quot;claude-sonnet-4-6&quot;,<br>    max_tokens=1024,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Summarise this report.&quot;}]<br>)<br># Extended thinking<br>response = client.messages.create(<br>    model=&quot;claude-opus-4-5&quot;,<br>    max_tokens=16000,        # must exceed budget_tokens<br>    thinking={<br>        &quot;type&quot;: &quot;enabled&quot;,<br>        &quot;budget_tokens&quot;: 8000<br>    },<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Design a distributed caching strategy.&quot;}]<br>)<br>print(f&quot;Input:  {response.usage.input_tokens}&quot;)<br>print(f&quot;Output: {response.usage.output_tokens}&quot;)</pre><h3>Google (Gemini)</h3><pre>import google.generativeai as genai<br><br>model = genai.GenerativeModel(<br>    &quot;gemini-1.5-pro&quot;,<br>    generation_config=genai.GenerationConfig(<br>        max_output_tokens=1024,<br>        temperature=0.7<br>    )<br>)<br>response = model.generate_content(&quot;Explain transformer architecture.&quot;)<br>print(f&quot;Input tokens:  {response.usage_metadata.prompt_token_count}&quot;)<br>print(f&quot;Output tokens: {response.usage_metadata.candidates_token_count}&quot;)<br>print(f&quot;Total tokens:  {response.usage_metadata.total_token_count}&quot;)</pre><h3>Mistral</h3><pre>from mistralai import Mistral<br>client = Mistral(api_key=&quot;your-key&quot;)<br><br>response = client.chat.complete(<br>    model=&quot;mistral-large-latest&quot;,<br>    max_tokens=1024,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Review this code for security issues.&quot;}]<br>)<br>print(f&quot;Prompt tokens:     {response.usage.prompt_tokens}&quot;)<br>print(f&quot;Completion tokens: {response.usage.completion_tokens}&quot;)</pre><h3>04 — Five Strategies to Use Your Budget Effectively</h3><h3>Strategy 1 — Trim the system prompt ruthlessly</h3><p>Your system prompt runs on <strong>every single request</strong>. It’s a fixed per-call tax. A 2,000-token system prompt on 50,000 daily calls costs 100 million input tokens per day — just in overhead.</p><p><strong>Audit checklist:</strong></p><ul><li>Remove instructions that repeat what the model already knows by default</li><li>Eliminate redundant examples — keep one, not five</li><li>Use placeholders for content that changes per-request (user name, date, context)</li><li>Target under 500 tokens for most use cases; under 200 for high-volume APIs</li></ul><pre># Before: 1,800 tokens<br>system_prompt = &quot;&quot;&quot;<br>You are a helpful customer service assistant for Acme Corp.<br>You should always be polite, professional, and helpful.<br>You should never be rude or dismissive.<br>You should always greet the user warmly.<br>[...200 more lines of redundant instructions...]<br>&quot;&quot;&quot;<br><br># After: 180 tokens<br>system_prompt = &quot;&quot;&quot;<br>You are Acme Corp&#39;s customer service assistant.<br>Be concise, friendly, and solution-focused.<br>Escalate billing disputes to human support.<br>&quot;&quot;&quot;</pre><h3>Strategy 2 — Match max_tokens to the task</h3><p>Right-sizing your output budget prevents both truncation (too low) and wasted budget (too high on expensive models):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-qOoeGU87XuNqRRppvwL4A.png" /></figure><p>Also tell the model what you expect in the prompt itself — “respond in 2–3 sentences” is a surprisingly effective length constraint.</p><h3>Strategy 3 — Tier your reasoning budget</h3><p>For models with reasoning modes, don’t default to maximum effort on every request. Build a tiered routing layer:</p><pre>def get_model_config(task_type: str) -&gt; dict:<br>    &quot;&quot;&quot;Route tasks to the right model and reasoning budget.&quot;&quot;&quot;<br>    configs = {<br>        &quot;classification&quot;: {<br>            &quot;model&quot;: &quot;gpt-4o-mini&quot;,<br>            &quot;max_tokens&quot;: 50,<br>            &quot;reasoning_effort&quot;: None<br>        },<br>        &quot;code_review&quot;: {<br>            &quot;model&quot;: &quot;gpt-4o&quot;,<br>            &quot;max_tokens&quot;: 2000,<br>            &quot;reasoning_effort&quot;: None<br>        },<br>        &quot;architecture_design&quot;: {<br>            &quot;model&quot;: &quot;o4-mini&quot;,<br>            &quot;max_tokens&quot;: 8000,<br>            &quot;reasoning_effort&quot;: &quot;medium&quot;<br>        },<br>        &quot;complex_reasoning&quot;: {<br>            &quot;model&quot;: &quot;o3&quot;,<br>            &quot;max_tokens&quot;: 16000,<br>            &quot;reasoning_effort&quot;: &quot;high&quot;<br>        }<br>    }<br>    return configs.get(task_type, configs[&quot;code_review&quot;])</pre><p>A classification task on GPT-4o-mini at $0.15/million tokens costs ~100x less than the same task routed to o3 with high reasoning effort. The quality difference for simple tasks is negligible.</p><h3>Strategy 4 — Compress conversation history</h3><p>In multi-turn applications, chat history is the fastest-growing cost driver. Every message you include in the next request costs you input tokens again.</p><pre>import anthropic<br><br>client = anthropic.Anthropic()<br>def compress_history(<br>    messages: list[dict],<br>    model: str,<br>    keep_recent: int = 6,<br>    threshold: int = 20<br>) -&gt; list[dict]:<br>    &quot;&quot;&quot;Summarise old turns when history gets long.&quot;&quot;&quot;<br>    if len(messages) &lt; threshold:<br>        return messages<br>    old    = messages[:-keep_recent]<br>    recent = messages[-keep_recent:]<br>    summary_prompt = (<br>        &quot;Summarise this conversation in under 150 words. &quot;<br>        &quot;Preserve: key decisions, facts established, user goals.\n\n&quot;<br>        + &quot;\n&quot;.join(f&quot;{m[&#39;role&#39;].upper()}: {m[&#39;content&#39;]}&quot; for m in old)<br>    )<br>    summary_resp = client.messages.create(<br>        model=model,<br>        max_tokens=200,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: summary_prompt}]<br>    )<br>    summary = summary_resp.content[0].text<br>    return [<br>        {&quot;role&quot;: &quot;user&quot;,      &quot;content&quot;: f&quot;[Earlier conversation summary]: {summary}&quot;},<br>        {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Understood. Continuing from where we left off.&quot;},<br>        *recent<br>    ]</pre><p>This can reduce history tokens by 60–80% in long sessions with minimal context loss.</p><h3>Strategy 5 — Use prompt caching</h3><p>All major providers offer <strong>prompt caching</strong> — a mechanism that lets you avoid re-encoding static content (system prompts, large documents) on every request.</p><pre># Anthropic — cache_control on large static blocks<br>response = client.messages.create(<br>    model=&quot;claude-sonnet-4-6&quot;,<br>    max_tokens=1024,<br>    system=[<br>        {<br>            &quot;type&quot;: &quot;text&quot;,<br>            &quot;text&quot;: large_static_document,<br>            &quot;cache_control&quot;: {&quot;type&quot;: &quot;ephemeral&quot;}  # cache this block<br>        }<br>    ],<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_question}]<br>)<br># Cache hit: ~90% discount on the cached input tokens<br><br># OpenAI - automatic prompt caching for prompts &gt; 1,024 tokens<br># No extra parameters needed; caching happens automatically<br># Cached tokens billed at 50% of normal input rate</pre><p>Prompt caching is one of the highest-ROI optimisations for production systems. A system prompt or knowledge base document that gets re-sent on every request becomes nearly free after the first call.</p><h3>05 — Monitoring Token Usage in Production</h3><p>You can’t optimise what you don’t measure. Here’s a provider-agnostic tracker:</p><pre>import time<br>from dataclasses import dataclass, field<br>from collections import defaultdict<br><br>@dataclass<br>class TokenUsageTracker:<br>    daily_budget: int = 2_000_000<br>    _log: list = field(default_factory=list)<br>    _daily: dict = field(default_factory=lambda: defaultdict(int))<br>    def record(<br>        self,<br>        session_id: str,<br>        model: str,<br>        task_type: str,<br>        input_tokens: int,<br>        output_tokens: int,<br>        reasoning_tokens: int = 0,<br>        cached_tokens: int = 0<br>    ):<br>        today = time.strftime(&quot;%Y-%m-%d&quot;)<br>        total = input_tokens + output_tokens + reasoning_tokens<br>        self._daily[today] += total<br>        entry = {<br>            &quot;ts&quot;: time.time(), &quot;session&quot;: session_id,<br>            &quot;model&quot;: model, &quot;task&quot;: task_type,<br>            &quot;input&quot;: input_tokens, &quot;output&quot;: output_tokens,<br>            &quot;reasoning&quot;: reasoning_tokens, &quot;cached&quot;: cached_tokens,<br>            &quot;total&quot;: total<br>        }<br>        self._log.append(entry)<br>        utilisation = self._daily[today] / self.daily_budget<br>        print(<br>            f&quot;[{task_type}] {model} | &quot;<br>            f&quot;{input_tokens}in + {output_tokens}out + {reasoning_tokens}think &quot;<br>            f&quot;| Daily: {self._daily[today]:,} / {self.daily_budget:,} &quot;<br>            f&quot;({utilisation:.1%})&quot;<br>        )<br>    def top_consumers(self, n: int = 5) -&gt; list:<br>        from collections import Counter<br>        counts = Counter(e[&quot;task&quot;] for e in self._log)<br>        tokens = defaultdict(int)<br>        for e in self._log:<br>            tokens[e[&quot;task&quot;]] += e[&quot;total&quot;]<br>        return sorted(tokens.items(), key=lambda x: x[1], reverse=True)[:n]<br>tracker = TokenUsageTracker(daily_budget=1_000_000)</pre><p><strong>Key metrics to track per call:</strong></p><ul><li>Input tokens, output tokens, reasoning tokens (separately)</li><li>Cached vs uncached input tokens</li><li>Model name and task type</li><li>Latency (tokens/second correlates to cost)</li><li>Session or user ID for attribution</li></ul><h3>06 — Provider Comparison at a Glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8X07Yp7X_C0kVre0MM5mLg.png" /></figure><h3>07 — Quick Reference Cheat Sheet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R00sNqeerVknhdhKs8ZOCA.png" /></figure><p>Situation Action Responses getting cut off Increase max_tokens Context window overflow Compress history, chunk documents, use RAG Costs too high Audit system prompt, use smaller model for simple tasks, enable caching Slow responses Reduce reasoning budget, use streaming Reasoning model overkill Switch to reasoning_effort: low or standard model Repetitive large inputs Enable prompt caching Multi-turn conversations Summarise history after N turns Batch, non-urgent jobs Use batch API (50% cheaper, async) Counting tokens before calling Use provider tokeniser library</p><h3>Closing Thoughts</h3><p>Token budgets are one of those subjects that separates developers who <em>use</em> LLMs from those who <em>engineer</em> production systems with them. The gap isn’t about prompt cleverness — it’s about understanding the economics of the context window and building systems that respect them.</p><p>Three changes will cover 80% of your wins:</p><ol><li><strong>Measure first.</strong> Log input + output + reasoning tokens on every call. You’ll immediately see where the money goes.</li><li><strong>Right-size the tool.</strong> Use smaller, cheaper models for tasks that don’t need frontier capability. Most classification, formatting, and retrieval tasks don’t need o3 or Opus.</li><li><strong>Cache aggressively.</strong> If any content repeats across requests — a system prompt, a knowledge base, a schema — cache it.</li></ol><p>Start there. The numbers will tell you what to fix next.</p><p><em>Written June 2026 · Covers OpenAI, Anthropic, Google Gemini, Mistral · Pricing and model details subject to change — always verify at your provider’s official documentation.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf8ae9397b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From LangGraph to Living Test Suites: Building an AI Test Generator That Learns Your Team’s Style]]></title>
            <link>https://medium.com/@gurudatt.sa26/from-langgraph-to-living-test-suites-building-an-ai-test-generator-that-learns-your-teams-style-f4abd20b86d8?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4abd20b86d8</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[playwright-test]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Sun, 17 May 2026 07:23:26 GMT</pubDate>
            <atom:updated>2026-05-17T07:23:26.596Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A walkthrough of LangGraph — what it is, why it exists, how it differs from a regular script — and then how to use it to build a Playwright test generator that actually writes tests the way your team does.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nStIVY3CXbW7w0rnzMghow.png" /></figure><h3>Part 1: What LangGraph Is (And Why It Exists)</h3><p>If you’ve used LangChain, you’ve probably built something like this:</p><blockquote>prompt → LLM → parser → done</blockquote><p>A linear pipeline. Input goes in one end, output comes out the other. Clean and simple. It’s also a poor fit for almost every real application.</p><p>Real workflows branch. They loop. They make decisions based on intermediate results. They sometimes call tools, sometimes don’t. They occasionally need to pause and wait for a human. A straight line can’t express any of that gracefully.</p><p><strong>LangGraph is the framework for building those non-linear LLM workflows.</strong> It models your application as a <em>graph</em> — a set of steps connected by edges, with a shared piece of data flowing through them.</p><h3>The Three Concepts You Need</h3><p>LangGraph is built on three primitives. Everything else is variations on these.</p><p><strong>1. State</strong> is the data that flows through your application. It’s a dictionary with a schema — for example, “every state has a question, a category, and an answer.&quot; As the workflow runs, each step reads from and writes to this shared state.</p><p>Think of it as a clipboard that gets passed from station to station. Each station can read what’s on it and add new entries.</p><p><strong>2. Nodes</strong> are the steps. Each node is a function that takes the current state, does something (often calls an LLM, but it could be anything — a database query, an HTTP call, a calculation), and returns updates to the state.</p><p>A crucial detail: a node doesn’t have to call an LLM. A node that runs a SQL query is just as valid as a node that talks to Claude. This is what makes LangGraph useful for real systems — you can mix LLM and non-LLM logic naturally.</p><p><strong>3. Edges</strong> connect the nodes. Most are simple (“after node A, go to node B”). Some are <em>conditional</em> — they run a small routing function that looks at the state and decides where to go next.</p><p>That’s it. State, nodes, edges. With those three things, you can build anything from a chatbot to a multi-agent research system.</p><h3>A Tiny Example</h3><p>The simplest non-trivial LangGraph application is a question router. The user asks something, an LLM classifies it as “math” or “general,” and the workflow routes to the right handler:</p><pre>┌──────────┐<br>        │ classify │<br>        └────┬─────┘<br>             │<br>       ┌─────┴─────┐<br>       ▼           ▼<br> ┌──────────┐ ┌──────────────┐<br> │   math   │ │   general    │<br> │ handler  │ │   handler    │<br> └──────────┘ └──────────────┘</pre><p>In code, that’s three node functions, one routing function, four edges, and you’re done. The whole thing fits in about 40 lines.</p><p>What matters isn’t the line count. It’s that the <em>shape</em> of the application — branch here, converge there — is visible at a glance. Compare that to the same logic written as nested if statements in a script. Same behavior; far less readable.</p><h3>How LangGraph Differs From a Plain Script</h3><p>A reasonable question: why use a framework at all? You could write the same logic in plain Python:</p><pre>if classify(question) == &quot;math&quot;:<br>    answer = math_handler(question)<br>else:<br>    answer = general_handler(question)</pre><p>For two nodes and one branch, the framework is overkill. The benefits compound as workflows grow:</p><ul><li><strong>State is shared cleanly</strong> — no passing twelve arguments through every function call</li><li><strong>Conditional logic is structural, not buried</strong> — you can <em>see</em> the routing as a graph</li><li><strong>Each node is independently runnable</strong> — you can test math_handler without running the whole pipeline</li><li><strong>Streaming, persistence, and human-in-the-loop are free</strong> — they’re built into the framework</li><li><strong>Adding a new branch is local</strong> — you don’t rewrite the surrounding code, just add a node and an edge</li></ul><p>For anything with five or more steps, or any branching logic, the framework starts paying for itself. For anything you might want to extend later, it pays for itself immediately.</p><h3>How LangGraph Differs From “An Agent”</h3><p>This trips people up, so worth being explicit about it.</p><p>An <strong>agent</strong> is an LLM that can call tools in a loop: think, call a tool, observe, think again, call another tool, eventually finish. The model decides what to do next at each step.</p><p>A <strong>LangGraph workflow</strong> is a structure <em>you</em> design: the model can only do what the graph says. Some nodes might be agents internally, but the graph as a whole is deterministic in its structure.</p><p>The trade-off:</p><ul><li><strong>Agents are flexible</strong> but unpredictable, slower, and harder to debug</li><li><strong>Workflows are predictable</strong> but only handle paths you anticipated</li></ul><p>Most production systems are a hybrid: a structured graph that contains one or two agent nodes in the places where dynamic decision-making is actually needed. We’ll see exactly this pattern in Part 2.</p><h3>Part 2: Putting LangGraph to Work on a Real Problem</h3><p>With the concepts in hand, let’s apply them to a problem most teams have: <strong>tests get written too late, or not at all.</strong></p><p>A feature ships. Someone says “we should add tests for that.” A week later, the context is gone and the tests still don’t exist. By the next sprint, the missing tests have become a habit, and the bugs that follow are baked in.</p><p>The question this article tackles: <strong>can an AI watch your commits, draft Playwright tests in your team’s style, and post the results to Slack — so the team is more likely to actually review and merge them?</strong></p><h3>The Pipeline as a Graph</h3><p>Here’s the workflow expressed as a graph:</p><pre>   ┌──────────────────┐<br>   │  fetch_commits   │  Pull recent diffs from git<br>   └────────┬─────────┘<br>            ▼<br>   ┌──────────────────┐<br>   │  analyze_changes │  What changed, from a user&#39;s perspective?<br>   └────────┬─────────┘<br>            ▼<br>   ┌──────────────────┐<br>   │ create_test_plan │  What scenarios should we test?<br>   └────────┬─────────┘<br>            ▼<br>   ┌──────────────────┐<br>   │  needs_tests?    │  Skip if no user-facing changes<br>   └────────┬─────────┘<br>       ┌────┴────┐<br>       ▼         ▼<br>   ┌────────┐  ┌─────┐<br>   │generate│  │skip │<br>   │ _tests │  │     │<br>   └────┬───┘  └──┬──┘<br>        ▼         ▼<br>   ┌──────────────┐<br>   │  summarize   │  Condense for human consumption<br>   └──────┬───────┘<br>          ▼<br>   ┌──────────┐<br>   │post_slack│<br>   └──────────┘</pre><p>Six nodes, one conditional edge, one accumulating state object. Notice how the <em>shape</em> of the application is the diagram — no nested if statements, no hidden control flow. That’s the LangGraph payoff.</p><h3>Where the Agent Lives</h3><p>This pipeline has one agent and five regular nodes. That ratio is deliberate.</p><ul><li><strong>fetch_commits</strong> is just git commands. No LLM.</li><li><strong>analyze_changes</strong>, <strong>create_test_plan</strong>, and <strong>summarize</strong> are single LLM calls with fixed inputs and predictable outputs.</li><li><strong>post_slack</strong> is an HTTP call. No LLM.</li><li><strong>generate_tests</strong> is the one node that needs to <em>gather information dynamically</em> — read the actual component code, check what selectors really exist, consult example tests for style. That&#39;s the job an agent is good at.</li></ul><p><strong>The rule of thumb: use an agent only where dynamic information-gathering is the actual task.</strong> Everywhere else, a plain LLM call is faster, cheaper, and more predictable.</p><h3>The Generic-AI Problem</h3><p>Before we get to the agent, let’s name the problem it solves.</p><p>If you ask any LLM to write a Playwright test for “user can log in with Remember Me checked,” you’ll get something that looks like this:</p><pre>test(&#39;remember me works&#39;, async ({ page, context }) =&gt; {<br>  await page.goto(&#39;http://localhost:3000/login&#39;);<br>  await page.locator(&#39;input.email-field&#39;).fill(&#39;test@example.com&#39;);<br>  await page.locator(&#39;input.password-field&#39;).fill(&#39;pass123&#39;);<br>  await page.locator(&#39;input.remember-me-checkbox&#39;).check();<br>  await page.locator(&#39;button.submit-btn&#39;).click();<br>  // ... etc<br>});</pre><p>Plausible. Compiles. Probably even passes — for now.</p><p>But it’s wrong in the ways that matter: those CSS classes don’t exist in your codebase. Your team uses getByRole and getByLabel. You have a LoginPage Page Object. Your assertion style is different.</p><p>Someone now has to rewrite the test before it can merge. You traded “writing a test” for “rewriting an AI test.” Sometimes that’s still a win; often it’s a wash.</p><h3>The Insight: Reference Materials</h3><p>Here’s the move that changes everything: <strong>give the agent a folder of reference examples, and tools to read them.</strong></p><pre>references/<br>├── example-tests/<br>│   ├── login.spec.ts            ← canonical login pattern<br>│   ├── form-submission.spec.ts<br>│   └── search-and-filter.spec.ts<br>├── conventions.md               ← &quot;how we write tests&quot;<br>├── page-objects/<br>│   └── how-we-use-them.md<br>└── anti-patterns.md             ← &quot;do not do this&quot;</pre><p>Then tell the agent: <em>“Before writing a test, look at the references. Find the example most similar to what you’re writing. Follow its style.”</em></p><p>When the same “Remember me” task runs through this agent, it now:</p><ol><li>Lists what references exist</li><li>Reads login.spec.ts for the style</li><li>Reads the actual LoginForm.tsx to see real selectors</li><li>Reads conventions.md to confirm rules</li><li>Writes a test that matches your codebase</li></ol><p>The output transforms:</p><pre>test(&#39;Remember me persists session across browser restarts&#39;, async ({ browser }) =&gt; {<br>  const ctx = await browser.newContext();<br>  const page = await ctx.newPage();<br>  const login = new LoginPage(page);<br>  await login.signIn(&#39;test@example.com&#39;, &#39;pass123&#39;, { rememberMe: true });<br>  await expect(page).toHaveURL(&#39;/dashboard&#39;);<br>  const storage = await ctx.storageState();<br>  const sessionCookie = storage.cookies.find(c =&gt; c.name === &#39;session&#39;);<br>  expect(sessionCookie?.expires).toBeGreaterThan(Date.now() / 1000 + 7 * 86400);<br>});</pre><p>Same model. Same task. The only difference is the reference material it consulted before writing.</p><p>That’s roughly the lift between “AI tests we throw away” and “AI tests we actually merge.”</p><h3>The Four Tools the Agent Needs</h3><p>Concretely, the generate_tests agent has access to four tools. Their names are self-explanatory:</p><ul><li><strong>list_references()</strong> — see what reference materials exist</li><li><strong>read_reference(name)</strong> — read a specific reference file</li><li><strong>read_component(path)</strong> — read the actual source code of the component being tested (so it can see real selectors, not invent them)</li><li><strong>write_test(filename, code)</strong> — save the generated test to disk</li></ul><p>Four tools. The agent figures out how to combine them. A typical run is five to ten tool calls and takes about ten seconds.</p><h3>Why Tools Beat Stuffing the Prompt</h3><p>A reasonable objection: “Why not just paste all the example tests into the prompt?”</p><p>You can. It’s called few-shot prompting and it works fine for tiny reference sets. It breaks down quickly:</p><ul><li><strong>Token bloat.</strong> You pay for every example on every call, whether the agent needed it or not.</li><li><strong>No selection.</strong> The agent receives every example even when only one is relevant — models can get confused by irrelevant context.</li><li><strong>Doesn’t scale.</strong> Twenty examples, two hundred examples — you can’t fit them all.</li><li><strong>No verification.</strong> The agent can’t check that a selector actually exists in your code; it can only pattern-match against the examples.</li></ul><p>Tools solve all four. The agent reads only what it needs, and can verify against the live codebase before writing anything.</p><p>The mental-model shift is the takeaway here: <strong>stop thinking of context as something you push into the prompt. Start thinking of it as something the agent pulls in as needed.</strong></p><h3>What You Put in the Reference Folder Matters More Than the Code</h3><p>Here’s the part that surprised me most. Once the agent and tools are set up, <strong>improving output is no longer a prompt-engineering problem. It’s a reference-curation problem.</strong></p><p>Better tests come from better references. If the generated tests aren’t using your Page Object Model, the fix isn’t a smarter prompt — it’s adding a clearer Page Object example to the references folder. If tests don’t have good error-case coverage, add an anti-patterns doc with examples of what’s missing.</p><p>This is great news, because it means non-engineers can improve the agent. Your QA lead doesn’t need to learn LangGraph to make the tests better. They maintain the reference folder — which is something they’re already qualified to do, and which improves human onboarding for new testers as a side benefit.</p><p>A useful reference folder typically includes:</p><ul><li>Two to four <strong>canonical test examples</strong> covering common patterns (form, navigation, list/filter, auth)</li><li>A short <strong>conventions doc</strong> — “we use getByRole over CSS, files are kebab-case, group with describe blocks”</li><li><strong>Page Object examples</strong> if you use them</li><li>An <strong>anti-patterns doc</strong> — “do not do X, here’s why,” often more useful than “do Y”</li></ul><p>Notice this folder is also useful for human developers learning to write tests on your team. The agent and the new hire benefit from the same artifacts. That’s a nice property.</p><h3>Things That Go Wrong (Honestly)</h3><p>Five real failure modes, roughly in order of how often they show up:</p><p><strong>The agent skips reading references.</strong> Despite being told to, the model sometimes barrels ahead and writes from scratch. Make “read at least one reference before writing” a hard rule in the system prompt. If it still happens, validate after the run and re-invoke.</p><p><strong>The agent hallucinates selectors.</strong> Even with read_component available, the model occasionally guesses. Add an extract_test_ids tool that returns <em>only</em> the test IDs that actually exist in a file, and require the agent to call it before writing assertions.</p><p><strong>The agent reads too much.</strong> Sometimes it reads every reference file before writing a single line. Set a max-iterations limit and add a system-prompt nudge — “be efficient, don’t re-read files.”</p><p><strong>Tests pass but assert on the wrong thing.</strong> Hardest one. The agent writes a syntactically valid test that checks the wrong indicator (a .success class) when the real signal is something else (the URL change). Include a &quot;common assertion mistakes&quot; section in the references. Human review remains non-negotiable.</p><p><strong>Cost creeps up.</strong> Each agent run can be 5–15× the tokens of a single LLM call. On a busy repo, it adds up. Run on PR-merge rather than every commit; cache reference reads within a single run; set a token budget per node.</p><p>The pattern across all of these: <strong>the agent is a draft generator, not a senior engineer.</strong> Treat its output like a pull request from a new team member — review before merging.</p><h3>What This Buys You (and What It Doesn’t)</h3><p>Honest expectations:</p><p><strong>It buys you</strong> tests that match your team’s style without anyone hand-rewriting them, a shorter loop from “code changed” to “test exists,” better consistency than humans naturally produce (the agent always follows the references; humans get tired), and a forcing function for documenting your testing conventions — because now the references have to actually exist.</p><p><strong>It does not buy you</strong> tests you don’t need to review, a replacement for thinking about what to test, coverage of weird edge cases the LLM didn’t notice, or anything resembling exploratory testing.</p><p>The right framing: this is autocomplete for the <em>boring</em> part of testing — translating “we should test X” into “here’s a draft test for X in our team’s style.” The interesting parts (deciding what to test, judging if it’s enough, finding what the test missed) remain a human’s job.</p><h3>Bringing It Back Together</h3><p>We started with three concepts: state, nodes, edges. Six nodes and one conditional edge later, we had a pipeline that watches git, drafts Playwright tests in your team’s style, and posts the results to Slack.</p><p>The architecture is small. The leverage is real. And the part that does the heavy lifting — the agent reading reference materials — is something a team can keep improving without ever touching the framework code.</p><p>That’s what makes LangGraph worth learning. Not because graphs are intrinsically cool, but because the right abstractions let you build systems where the interesting work (curating examples, defining conventions) happens at the level where humans actually have judgment — and the wiring stays out of the way.</p><h3>Where to Take This Next</h3><p>The “agent + reference library” pattern generalizes far beyond test generation. The same architecture works for code review bots (references are your team’s review standards), documentation writers (your style guide and existing docs), bug triagers (past bug reports with resolutions), and support drafters (help center articles, example replies).</p><p>In every case the recipe is the same: a curated reference folder, tools to read it, and a model good enough to follow examples. The hard work is curating the references, not building the agent.</p><p>That’s actually liberating, if you sit with it. The future of useful internal AI tools is probably less about prompt-engineering wizardry and more about <em>what’s worth writing down clearly enough that an agent can use it?</em></p><p>A lot of teams have been meaning to write that stuff down anyway. Now there are two reasons to.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4abd20b86d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Scored My RAG Pipeline and Everything Was Broken — Here’s How I Fixed It With RAGAS]]></title>
            <link>https://medium.com/@gurudatt.sa26/i-scored-my-rag-pipeline-and-everything-was-broken-heres-how-i-fixed-it-with-ragas-8dffdbfc2998?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dffdbfc2998</guid>
            <category><![CDATA[ai-model]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[model-training]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Sun, 03 May 2026 08:22:42 GMT</pubDate>
            <atom:updated>2026-05-03T08:22:42.972Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*R3UFf5kasFYTlOWj2u2qVA.png" /></figure><h3>The Problem Nobody Talks About</h3><p>You’ve built a RAG pipeline. ChromaDB is storing your embeddings. Ollama is generating answers. You ask a question, you get an answer. It <em>looks</em> right.</p><p>But is it?</p><p>How do you know the right chunks are being retrieved? How do you know the LLM isn’t making things up? How do you know a change you made yesterday didn’t break three other queries?</p><p>You don’t. Not until you measure.</p><p>This article walks through exactly how I added RAGAS evaluation to my ChromaDB + Ollama RAG pipeline — a real MCP knowledge base server — and used the scores to systematically fix it. No theory dumps. Just the actual journey: what broke, what the scores told me, and what I changed.</p><h3>What Is RAGAS?</h3><p>RAGAS (Retrieval Augmented Generation Assessment) is a Python library that scores your RAG pipeline. It uses an LLM to judge the output of your LLM — the “LLM-as-a-judge” approach.</p><p>It scores three things separately:</p><p><strong>Faithfulness</strong> — did the answer stick to the retrieved chunks, or did the LLM hallucinate? The judge LLM breaks the answer into individual claims and checks each one against the chunks. Supported claims divided by total claims gives you the score.</p><p><strong>Answer Relevancy</strong> — does the answer actually address the question? RAGAS generates artificial questions from the answer and measures how similar they are to the original question using cosine similarity.</p><p><strong>Context Precision</strong> — did ChromaDB return the right chunks? Were the relevant ones ranked at the top, or buried under irrelevant noise?</p><p>Each score is between 0 and 1. Higher is better. And crucially — each score points to a different part of the pipeline, so you know exactly what to fix.</p><h3>My Setup</h3><p>I had a working MCP knowledge base server:</p><ul><li><strong>ChromaDB</strong> as the vector store</li><li><strong>Ollama</strong> running locally for both embeddings and generation</li><li><strong>nomic-embed-text</strong> as the embedding model</li><li><strong>llama3.1:8b</strong> as the generation model</li><li>Internal Word documents (project documentation) as the knowledge base</li></ul><p>The server exposed a search_knowledge_base tool that embedded the query, searched ChromaDB, and returned the top 5 chunks with relevance scores. Generation happened on the client side.</p><h3>Step 1 — Building the Eval Script</h3><p>For RAGAS evaluation, I needed to add generation to the pipeline (since my MCP server only did retrieval). I extracted the query logic from my server and added an Ollama generation step:</p><pre>def query_rag(question: str, top_k: int = 5):<br>    # Retrieve — same as MCP server<br>    query_embedding = ollama.embeddings(<br>        model=EMBEDDING_MODEL,<br>        prompt=question,<br>    )[&quot;embedding&quot;]</pre><pre>    results = collection.query(<br>        query_embeddings=[query_embedding],<br>        n_results=top_k,<br>    )</pre><pre>    chunks = results[&quot;documents&quot;][0]</pre><pre>    # Generate — ask Ollama using the chunks<br>    context = &quot;\n\n&quot;.join(chunks)<br>    prompt = (<br>        f&quot;Answer the question using ONLY the provided context.\n&quot;<br>        f&quot;If the context doesn&#39;t contain the answer, say &#39;I don&#39;t know.&#39;\n\n&quot;<br>        f&quot;Context:\n{context}\n\n&quot;<br>        f&quot;Question:\n{question}\n\n&quot;<br>        f&quot;Answer:&quot;<br>    )</pre><pre>    response = ollama.chat(<br>        model=LLM_MODEL,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>    )</pre><pre>    return {<br>        &quot;answer&quot;: response[&quot;message&quot;][&quot;content&quot;],<br>        &quot;chunks&quot;: chunks,<br>    }</pre><p>Then I wired it into RAGAS. The key challenge: RAGAS defaults to OpenAI, but I’m running everything locally with Ollama. Ollama exposes an OpenAI-compatible API at localhost:11434/v1, so I used the OpenAI client pointed at localhost:</p><pre>from openai import OpenAI, AsyncOpenAI<br>from ragas.llms import llm_factory<br>from ragas.embeddings import OpenAIEmbeddings<br><br># Judge LLM<br>ollama_client = OpenAI(api_key=&quot;dummy&quot;, base_url=&quot;http://localhost:11434/v1&quot;)<br>judge_llm = llm_factory(LLM_MODEL, provider=&quot;openai&quot;, client=ollama_client)<br># Judge embeddings<br>ollama_async_client = AsyncOpenAI(api_key=&quot;ollama&quot;, base_url=&quot;http://localhost:11434/v1&quot;)<br>judge_embeddings = OpenAIEmbeddings(ollama_async_client, &quot;mxbai-embed-large&quot;)</pre><p>No OpenAI API key. No data leaving my machine. Just a local client talking to local Ollama.</p><h3>Step 2 — Writing the Ground Truth Dataset</h3><p>This is the manual part. I read through my internal documents and wrote question-answer pairs:</p><pre>eval_data = [<br>    {<br>        &quot;question&quot;: &quot;What is the purpose of the onboarding flow?&quot;,<br>        &quot;ground_truth&quot;: &quot;The onboarding flow guides new users through account setup, profile creation, and initial configuration in three steps.&quot;<br>    },<br>    {<br>        &quot;question&quot;: &quot;How does the notification system work?&quot;,<br>        &quot;ground_truth&quot;: &quot;Notifications are triggered by status changes and delivered via email and in-app alerts within 5 minutes.&quot;<br>    },<br>    {<br>        &quot;question&quot;: &quot;What happens when a user resets their password?&quot;,<br>        &quot;ground_truth&quot;: &quot;A reset link is sent to the registered email. The link expires after 24 hours. The user must create a new password meeting the complexity requirements.&quot;<br>    },<br>    # ... 8 questions total<br>]</pre><p>I wrote eight questions covering different sections of the document. The key is mixing broad questions (“What is X?”) with specific ones (“How does Y work?”) — this reveals whether your pipeline handles both well. Enough to spot patterns, not so many that it takes forever to run locally.</p><h3>Step 3 — The First Score (The Shock)</h3><p>I ran the evaluation. Three of my eight questions scored <strong>all zeros</strong> across every metric.</p><pre>Q4: How does the notification system work?        → 0.0, 0.0, 0.0<br>Q6: What happens during password reset?           → 0.0, 0.0, 0.0  <br>Q7: How does the search feature work?             → 0.0, 0.0, 0.0</pre><p>Ollama returned “I don’t know” for all three. But the answers were definitely in my documents.</p><h3>Step 4 — Diagnosing the Problem</h3><p>I wrote a debug script to see what chunks ChromaDB was actually returning:</p><pre>q = &quot;How does the notification system work?&quot;<br>embedding = ollama.embeddings(model=&quot;nomic-embed-text&quot;, prompt=q)[&quot;embedding&quot;]<br>results = collection.query(query_embeddings=[embedding], n_results=5)<br><br>for i, (doc, dist) in enumerate(zip(results[&quot;documents&quot;][0], results[&quot;distances&quot;][0])):<br>    print(f&quot;Chunk {i+1} (distance: {dist:.4f}): {doc[:100]}...&quot;)</pre><p>The output revealed the problem:</p><pre>Chunk 1 (distance: 0.4494): User preferences are stored in the settings module...<br>Chunk 2 (distance: 0.4537): The admin panel provides access to all configurations...<br>Chunk 3 (distance: 0.4758): In Summary, the system handles multiple workflows...</pre><p>The chunk that actually described the notification system was nowhere in the top 5. I expanded the search to all chunks:</p><pre>Rank 10 (distance: 0.5267): Notifications are triggered by status changes and delivered via...</pre><p>Rank 10. Distance 0.53. The embedding model thought “How does the notification system work?” was more similar to a chunk about admin configurations than to a chunk that actually describes notifications.</p><p><strong>This was an embedding model problem.</strong></p><h3>Step 5 — Fixing Retrieval (Embedding Model Switch)</h3><p>I switched from nomic-embed-text (768 dimensions) to mxbai-embed-large (1024 dimensions):</p><pre>ollama pull mxbai-embed-large</pre><p>Re-ingested all documents and ran the same debug query:</p><pre>Rank 1 (distance: 0.4131): Notifications are triggered by status changes and delivered via...</pre><p>The notification chunk jumped from Rank 10 to <strong>Rank 1</strong>. The better embedding model correctly matched the query to the relevant chunk.</p><p>Scores after the embedding model switch:</p><pre>Before:  Faithfulness 0.60  |  Relevancy 0.46  |  Precision 0.61<br>After:   Faithfulness 0.67  |  Relevancy 0.64  |  Precision 0.70</pre><p>No more zeros. But still not great.</p><h3>Step 6 — Fixing Generation (Prompt + Temperature)</h3><p>The scores told me retrieval was mostly fixed (precision 0.70), but the LLM was still hallucinating (faithfulness 0.67) and drifting off-topic (relevancy 0.64).</p><p>Two changes:</p><p><strong>Tighter prompt:</strong></p><pre>prompt = (<br>    &quot;You are a helpful assistant that answers questions using only the provided context.\n&quot;<br>    &quot;Rules:\n&quot;<br>    &quot;- Use ONLY the information in the context below\n&quot;<br>    &quot;- Include ALL relevant details from the context that answer the question\n&quot;<br>    &quot;- Do NOT add anything from your own knowledge\n&quot;<br>    &quot;- If the context does not contain the answer, say &#39;I don&#39;t know&#39;\n&quot;<br>    &quot;- Answer in 2-4 complete sentences\n\n&quot;<br>    f&quot;Context:\n{context}\n\n&quot;<br>    f&quot;Question: {question}\n\n&quot;<br>    &quot;Answer:&quot;<br>)</pre><p><strong>Low temperature</strong> to make Ollama less creative (creativity = hallucination in RAG):</p><pre>response = ollama.chat(<br>    model=LLM_MODEL,<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>    options={&quot;temperature&quot;: 0.1},<br>)</pre><p>Final scores:</p><pre>Before prompt fix:  Faithfulness 0.67  |  Relevancy 0.64  |  Precision 0.70<br>After prompt fix:   Faithfulness 1.00  |  Relevancy 0.81  |  Precision 1.00</pre><h3>The Complete Journey</h3><pre>Starting point:            0.60  →  0.46  →  0.61  (with 3 questions scoring zero)<br>After embedding fix:       0.67  →  0.64  →  0.70  (zeros eliminated)<br>After prompt + temperature: 1.00  →  0.81  →  1.00  (production-ready)</pre><p>Three changes. Each one targeted by the scores:</p><ol><li><strong>Scores showed retrieval was broken</strong> → switched embedding model</li><li><strong>Scores showed generation was hallucinating</strong> → tightened prompt</li><li><strong>Scores showed generation was too creative</strong> → lowered temperature</li></ol><p>No guessing. No random tweaking. The scores told me exactly where the problem was and which part of the pipeline to fix.</p><h3>What I Learned</h3><p><strong>Scores point to different workers.</strong> A RAG pipeline has two workers: the retriever (ChromaDB) and the generator (LLM). Context precision scores the retriever. Faithfulness and relevancy score the generator. When a score is low, you know which worker to blame.</p><p><strong>The embedding model matters more than chunk size.</strong> I spent time experimenting with chunk sizes — 1200, 600, 500, 300 characters. None of it mattered because the embedding model couldn’t connect a query to the chunk containing the answer. Switching the model fixed it instantly.</p><p><strong>Temperature is crucial for RAG.</strong> High temperature means the LLM gets creative. In RAG, creative = hallucination. Setting temperature to 0.1 forced Ollama to stick to the facts in the chunks.</p><p><strong>One question isn’t enough.</strong> My first eval had one question that scored 1.0 across the board. I thought everything was perfect. Adding seven more questions revealed that three of them scored zero. Coverage matters in eval datasets just like it matters in test suites.</p><p><strong>The debug script is your best friend.</strong> When scores are bad, don’t guess — look at what chunks are actually being retrieved. Print them. Read them. Compare them to what should have been retrieved. The answer is always in the data.</p><h3>The Practical Order of Fixes</h3><p>If you’re starting from scratch, this is the order that gives you the biggest improvements with the least effort:</p><pre>1. Write 8-15 eval questions with ground truth     ← free, essential<br>2. Run RAGAS, get baseline scores                   ← free, diagnostic<br>3. Fix the prompt (add strict rules)                ← free, high impact<br>4. Lower temperature to 0.1                         ← free, high impact<br>5. Try a different embedding model                  ← free, big impact<br>6. Adjust chunk size                                ← free, moderate impact<br>7. Fine-tune the LLM                               ← last resort, expensive</pre><p>Most RAG problems get fixed in steps 3–5 without spending anything.</p><h3>Tools Used</h3><ul><li><strong>ChromaDB</strong> — vector store (local, free)</li><li><strong>Ollama</strong> — LLM and embeddings (local, free)</li><li><strong>mxbai-embed-large</strong> — embedding model (768 → 1024 dimensions, better semantic matching)</li><li><strong>llama3.1:8b</strong> — generation model</li><li><strong>RAGAS</strong> — evaluation framework</li><li><strong>Python</strong> — everything runs in a single script</li></ul><p>The entire eval pipeline runs locally. No API keys, no cloud costs, no data leaving your machine.</p><p><em>The eval dataset is your test suite. The RAGAS scores are your test results. The feedback loop — score, diagnose, fix, repeat — is the same engineering discipline you already practice in software testing. The only difference is that instead of asserting return values, you’re asserting the quality of language.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dffdbfc2998" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Does a Vector Database Know That “Cat” and “Kitten” Are Similar?]]></title>
            <link>https://medium.com/@gurudatt.sa26/how-does-a-vector-database-know-that-cat-and-kitten-are-similar-f56c2ba7770c?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f56c2ba7770c</guid>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Sat, 02 May 2026 11:14:36 GMT</pubDate>
            <atom:updated>2026-05-02T11:14:36.363Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Does a Vector Database Know That “Cat” and “Kitten” Are Similar? — Cosine, L2, and IP Explained Simply</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wr0FTTJN9C5RoN7iYeXdQg.png" /></figure><p>If you’re building anything with AI — a chatbot, a RAG pipeline, a semantic search tool — you’ll eventually use a vector database like ChromaDB. And at some point, you’ll run into a line like this:</p><pre>collection = client.create_collection(COLLECTION_NAME, metadata={<br>    &quot;hnsw:space&quot;: &quot;cosine&quot;,<br>})</pre><p>When I first saw this, I had the same question you probably have right now: <strong>What does “cosine” mean here? Why is it needed? And what are the other options?</strong></p><p>This article is for you if you’re a developer who is just getting into the world of embeddings and vector databases. No math background needed. No prior AI experience assumed. I’ll start from scratch — how text gets converted into numbers, how those numbers get compared, and what cosine, L2, and IP actually do under the hood — using a simple “Cat vs Kitten vs Car” example that makes the whole thing click.</p><p>Let’s break it down.</p><h3>First Things First — Words Become Numbers</h3><p>Computers don’t understand words. So before ChromaDB can do anything useful, every piece of text you store gets converted into an <strong>embedding</strong> — a long list of numbers.</p><pre>Cat:    [0.82, -0.15, 0.63, 0.41, ...]<br>Dog:    [0.79, -0.12, 0.58, 0.44, ...]<br>Car:    [0.11,  0.85, -0.30, 0.22, ...]</pre><p>These numbers come from an embedding model that has been trained on massive amounts of text. Each number roughly captures some aspect of meaning. You don’t control what each number means — the model learned it during training. But loosely, some numbers might represent concepts like “is it an animal?”, “is it small?”, “is it a pet?” — spread across all the numbers together.</p><p>Notice something: “Cat” and “Dog” have similar numbers because they share similar meaning. “Car” has very different numbers because it’s a completely different concept.</p><p>ChromaDB stores these lists of numbers alongside your original text.</p><h3>Now You Search — And Similarity Math Kicks In</h3><p>When you search for “Kitten,” it also becomes an embedding:</p><pre>Kitten: [0.80, -0.14, 0.61, 0.39, ...]</pre><p>Very close to “Cat” because the meaning is close.</p><p>ChromaDB now needs to answer: <strong>which stored embeddings are most similar to the query embedding?</strong></p><p>That’s where &quot;hnsw:space&quot;: &quot;cosine&quot; comes in. It tells ChromaDB <strong>how</strong> to compare these lists of numbers. And there are three options: <strong>Cosine</strong>, <strong>L2</strong>, and <strong>IP</strong>.</p><h3>The Arrow Analogy</h3><p>To understand all three, imagine each embedding as an <strong>arrow drawn from the center of a page</strong>. If you have just 2 numbers, you can actually plot it:</p><pre>Cat:    [0.8, 0.6]<br>Kitten: [0.7, 0.5]<br>Car:    [0.1, 0.9]</pre><p>On a graph:</p><pre>↑ 0.9<br>        |         • Car [0.1, 0.9]<br>        |<br>  0.6   |  • Cat [0.8, 0.6]<br>  0.5   | • Kitten [0.7, 0.5]<br>        |<br>        +-------------------→<br>           0.1  0.7  0.8</pre><p>Cat and Kitten arrows point in almost the same direction. Car’s arrow points somewhere completely different.</p><p>Now here’s how the three methods look at the <strong>same</strong> arrows:</p><ul><li><strong>Cosine</strong> — do the arrows <strong>point the same way</strong>?</li><li><strong>L2</strong> — how far apart are the <strong>tips</strong> of the arrows?</li><li><strong>IP</strong> — do the arrows point the same way <strong>and are both long</strong>?</li></ul><p>Let’s do the actual math for each.</p><h3>Cosine — Comparing Direction</h3><p>Cosine measures the angle between two arrows. Same direction = similar. Different direction = not similar. It doesn’t care how long the arrows are.</p><p><strong>The math:</strong> Multiply matching numbers, add them up, then divide by both lengths.</p><p><strong>Step 1 — Dot product (multiply and add):</strong></p><pre>Kitten vs Cat:  (0.7×0.8) + (0.5×0.6) = 0.56 + 0.30 = 0.86<br>Kitten vs Car:  (0.7×0.1) + (0.5×0.9) = 0.07 + 0.45 = 0.52</pre><p><strong>Step 2 — Calculate lengths:</strong></p><pre>Kitten: √(0.7² + 0.5²) = √(0.49 + 0.25) = √0.74 = 0.86<br>Cat:    √(0.8² + 0.6²) = √(0.64 + 0.36) = √1.00 = 1.00<br>Car:    √(0.1² + 0.9²) = √(0.01 + 0.81) = √0.82 = 0.91</pre><p><strong>Step 3 — Divide:</strong></p><pre>Kitten vs Cat:  0.86 / (0.86 × 1.00) = 1.00  ← very similar<br>Kitten vs Car:  0.52 / (0.86 × 0.91) = 0.66  ← less similar</pre><p>Cat wins. Kitten is most similar to Cat. Makes sense.</p><h3>L2 — Measuring the Gap Between Points</h3><p>L2 (Euclidean distance) is the simplest. It measures the straight-line distance between two points. You’re literally using a ruler.</p><p><strong>The math:</strong> Subtract the numbers, square them, add them, take the square root.</p><pre>Kitten vs Cat:  √((0.7-0.8)² + (0.5-0.6)²) = √(0.01 + 0.01) = 0.14  ← close<br>Kitten vs Car:  √((0.7-0.1)² + (0.5-0.9)²) = √(0.36 + 0.16) = 0.72  ← far</pre><p>Smaller number = more similar. Cat wins again.</p><h3>IP — Direction Plus Magnitude</h3><p>Inner Product is like cosine but without the dividing step. It cares about direction <strong>and</strong> how big the numbers are.</p><p><strong>The math:</strong> Just multiply matching numbers and add them up.</p><pre>Kitten vs Cat:  (0.7×0.8) + (0.5×0.6) = 0.56 + 0.30 = 0.86<br>Kitten vs Car:  (0.7×0.1) + (0.5×0.9) = 0.07 + 0.45 = 0.52</pre><p>Bigger number = more similar. Cat wins once more.</p><h3>So What’s the Difference? They All Picked Cat!</h3><p>They did. But they behave differently in edge cases. Let’s add a new entry:</p><pre>Big Cat: [1.6, 1.2]   (same direction as Cat, but doubled)</pre><p>Method Kitten vs Cat Kitten vs Big Cat <strong>Cosine</strong> 0.99 0.99 ← same <strong>L2</strong> 0.14 1.08 ← different <strong>IP</strong> 0.86 1.72 ← different</p><p><strong>Cosine</strong> treats Cat and Big Cat as equally similar to Kitten — because the direction is the same. The arrow just got longer, but it’s still pointing the same way.</p><p><strong>L2</strong> sees Big Cat as farther away — the dot moved further on the graph.</p><p><strong>IP</strong> actually scores Big Cat <em>higher</em> — bigger numbers = bigger score.</p><h3>When to Use Which?</h3><p><strong>Cosine</strong> — Use this for text search, RAG, semantic search. Almost always the right choice. You care about meaning (direction), not how the model scales the numbers.</p><p><strong>L2</strong> — Use when the actual position of the point matters. Good for image similarity, anomaly detection (anything far from the cluster is unusual), or when your model explicitly places similar items close together in the number space.</p><p><strong>IP</strong> — Use when bigger numbers carry meaning, like confidence or relevance strength. Some ranking models are designed this way — a document strongly about cats gets bigger numbers than one that barely mentions cats. IP captures that.</p><h3>The Simple Rule</h3><p>Don’t know what to pick? Use <strong>cosine</strong>.</p><p>Your embedding model’s documentation says to use something else? Use what it says.</p><p>Most popular embedding models — OpenAI, Sentence Transformers, Cohere — are designed to work best with cosine.</p><h3>The One-Line Summary</h3><p>Words become numbers. Numbers become arrows. ChromaDB compares arrows. The hnsw:space setting just decides <strong>how</strong> it compares them — by direction (cosine), by gap (L2), or by direction plus size (IP).</p><p>That’s the whole thing.</p><p><em>If this helped you understand vector similarity, consider sharing it with someone who’s just getting started with vector databases. Sometimes the simplest explanation is the one that sticks.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f56c2ba7770c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Full-Stack Verification: Let a Playwright Agent Test Your UI, Then Verify the Database — Without…]]></title>
            <link>https://medium.com/@gurudatt.sa26/full-stack-verification-let-a-playwright-agent-test-your-ui-then-verify-the-database-without-21b37e1a485a?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/21b37e1a485a</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[playwright-test]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-testing]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 07:02:31 GMT</pubDate>
            <atom:updated>2026-04-10T07:02:31.117Z</atom:updated>
            <content:encoded><![CDATA[<h3>Full-Stack Verification: Let a Playwright Agent Test Your UI, Then Verify the Database — Without Writing SQL</h3><h3>Your checkout test clicks “Place Order” and sees a green toast. But did the row actually land?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AeOlExgLZFV57yGiYaLB0Q.png" /></figure><p>Most E2E tests are half-tests. They drive a browser, fill some forms, click a button, and assert that a success message appeared. Ship it.</p><p>But here’s what they <em>don’t</em> tell you: Did the order actually persist? Did the line items get written correctly? Did the inventory decrement? The UI said “confirmed,” but the UI lies sometimes — a swallowed error, a rolled-back transaction, a queue that silently dropped a message.</p><p>The traditional fix is ugly: you wire up an ORM or a raw database driver inside your test harness, manage separate credentials, write SQL assertions by hand, and pray your schema doesn’t drift. It works. It’s also a maintenance tax you pay on every test.</p><p>There’s a better pattern now. And it involves zero SQL on your part.</p><h3>The Idea</h3><p>What if a single AI agent could do both halves — drive the browser checkout <em>and</em> then pivot to the database to verify the write — using two MCP (Model Context Protocol) tool servers?</p><p>That’s what this tutorial walks through:</p><ol><li><strong>Playwright MCP</strong> handles browser automation (navigate, click, fill, snapshot).</li><li><strong>PostgreSQL MCP</strong> handles database introspection and read-only queries.</li></ol><p>The agent sequences them naturally. It completes the checkout, captures the order ID from the confirmation page, then switches context and uses the database tools to verify the data landed correctly. You describe the whole thing in plain English.</p><h3>What You’ll Need</h3><ul><li>A running web app with a checkout flow (any stack — this is browser-level)</li><li>PostgreSQL or MySQL with the app’s data</li><li>An MCP client (Claude Desktop, Claude Code, or any MCP-compatible client)</li><li>The Playwright and database MCP server packages</li></ul><h3>Step 1: Configure Both MCP Servers</h3><p>Add both servers to your MCP client configuration. Here’s what claude_desktop_config.json looks like:</p><pre>{<br>  &quot;mcpServers&quot;: {<br>    &quot;playwright&quot;: {<br>      &quot;command&quot;: &quot;npx&quot;,<br>      &quot;args&quot;: [&quot;@anthropic/mcp-playwright&quot;]<br>    },<br>    &quot;postgres&quot;: {<br>      &quot;command&quot;: &quot;npx&quot;,<br>      &quot;args&quot;: [<br>        &quot;@anthropic/mcp-postgres&quot;,<br>        &quot;postgresql://readonly:****@localhost:5432/shop&quot;<br>      ]<br>    }<br>  }<br>}</pre><p>Two things to note here. First, the database connection uses a <strong>read-only user</strong>. This is important — you’re giving an AI agent access to your database, so you want to make sure it can only read, never write. Second, if you’re on MySQL instead of PostgreSQL, just swap @anthropic/mcp-postgres for @anthropic/mcp-mysql. The tool interface is identical.</p><h3>Step 2: Drive the Checkout</h3><p>Now you give the agent a prompt describing the UI flow:</p><blockquote><em>Navigate to </em><em>localhost:3000/products. Add the &quot;Wireless Charger&quot; to the cart. Proceed to checkout. Fill in the customer name &quot;Ada Lovelace&quot; and email &quot;ada@test.dev&quot;. Select &quot;Credit Card&quot; as the payment method and click &quot;Place Order.&quot; Wait for the confirmation page and capture the order ID.</em></blockquote><p>Under the hood, the agent translates this into a sequence of Playwright MCP tool calls:</p><pre>browser_navigate({ url: &quot;http://localhost:3000/products&quot; })<br>browser_click({ selector: &quot;text=Wireless Charger&quot; })<br>browser_click({ selector: &quot;button:has-text(&#39;Add to Cart&#39;)&quot; })<br>browser_click({ selector: &quot;a:has-text(&#39;Checkout&#39;)&quot; })<br>browser_fill({ selector: &quot;#name&quot;, value: &quot;Ada Lovelace&quot; })<br>browser_fill({ selector: &quot;#email&quot;, value: &quot;ada@test.dev&quot; })<br>browser_click({ selector: &quot;text=Credit Card&quot; })<br>browser_click({ selector: &quot;button:has-text(&#39;Place Order&#39;)&quot; })<br>browser_snapshot()</pre><p>The agent reads the confirmation page snapshot and extracts: <strong>Order #10472</strong>.</p><p>No variable assignment, no test fixtures. The agent just remembers the order ID in its context and carries it forward.</p><h3>Step 3: Verify the Database — Zero SQL</h3><p>Here’s where it gets interesting. You continue the same prompt (or add a second instruction):</p><blockquote><em>Now check the database. Confirm that order 10472 exists in the orders table and belongs to ada@test.dev. Verify that the line_items table has an entry for “Wireless Charger” with quantity 1. Also check that the product’s inventory stock was decremented.</em></blockquote><p>The agent switches to the PostgreSQL MCP server. Its first move is to <strong>discover the schema</strong> — it doesn’t assume your column names:</p><pre>list_tables()<br>→ orders, line_items, products, customers ...<br>describe_table({ table: &quot;orders&quot; })<br>→ id, customer_email, status, total, created_at ...</pre><p>Then it constructs and runs the verification queries:</p><pre>read_query({<br>  sql: &quot;SELECT * FROM orders WHERE id = 10472&quot;<br>})<br>read_query({<br>  sql: &quot;SELECT * FROM line_items WHERE order_id = 10472&quot;<br>})<br>read_query({<br>  sql: &quot;SELECT name, stock FROM products<br>        WHERE name = &#39;Wireless Charger&#39;&quot;<br>})</pre><p><strong>You never wrote those queries.</strong> The agent composed them by inspecting table schemas through list_tables and describe_table, then built the SQL on the fly. If your column is qty instead of quantity, or product_title instead of name, the agent adapts — it read the schema first.</p><p>The agent then returns something like:</p><pre>Full-Stack Verification — Order #10472<br>───────────────────────────────────────<br>✓ orders row exists<br>    customer_email: ada@test.dev<br>    status: confirmed<br>    total: $34.99<br>✓ line_items row found<br>    product_name: Wireless Charger<br>    quantity: 1<br>    unit_price: $34.99<br>✓ products.stock decremented<br>    previous: 48 → current: 47<br>3/3 assertions passed</pre><p>Three verifications, zero lines of SQL written by you.</p><h3>The Combined Prompt</h3><p>In practice, you can issue the whole thing as a single instruction. The agent will sequence the browser phase first, extract the order ID, then pivot to the database automatically:</p><pre>## Full-Stack Checkout Verification<br>Use the Playwright MCP server to:<br>1. Navigate to localhost:3000/products<br>2. Add &quot;Wireless Charger&quot; to the cart<br>3. Complete checkout as &quot;Ada Lovelace&quot; (ada@test.dev),<br>   paying by Credit Card<br>4. Capture the order ID from the confirmation page<br>Then switch to the Postgres MCP server and verify:<br>- The order row exists with the correct email<br>- A line_item for &quot;Wireless Charger&quot; (qty 1) is present<br>- The product stock was decremented by 1<br>Report pass/fail for each assertion.</pre><p>That’s the entire test. One prompt. Two MCP servers. Full-stack coverage.</p><h3>Making It Production-Ready</h3><p>A few patterns that help when you move beyond demos:</p><p><strong>Use read-only credentials, always.</strong> The read_query tool should be the only database tool enabled. This prevents the agent from accidentally mutating data during verification. Defense in depth.</p><p><strong>Add timing guards for async writes.</strong> If your app uses queues, event sourcing, or eventual consistency, the row might not exist yet when the agent queries. Tell it: <em>“Wait up to 5 seconds for the order row to appear, polling every 500ms.”</em> The agent will loop the query until it resolves or times out.</p><p><strong>Snapshot before and after.</strong> For regression testing, ask the agent to query the relevant tables <em>before</em> the UI action and <em>after</em>, then diff the results. This catches unintended side effects — duplicate writes, missing audit columns, foreign key violations that got swallowed.</p><p><strong>Extend beyond SQL.</strong> The same two-phase pattern works with any backend MCP server. Redis MCP to verify cache invalidation. Elasticsearch MCP to confirm a document got indexed. S3 MCP to check that a file upload actually landed. Stack as many verification layers as your confidence demands.</p><h3>Why This Matters</h3><p>The traditional approach to full-stack test verification requires you to maintain a parallel testing infrastructure: database drivers, connection pools, ORM configurations, raw SQL strings that break when someone renames a column. It’s not hard work, but it’s <em>constant</em> work, and it sits in a weird no-man’s-land between your test suite and your application code.</p><p>The MCP approach collapses all of that into tool calls that the agent discovers and composes at runtime. You describe <em>what</em> to verify; the agent figures out <em>how</em>. When your schema changes, you don’t update test assertions — the agent re-discovers the schema on the next run.</p><p>It’s not magic. It’s just moving the adaptation layer from your code to the agent’s context. And for verification tasks — where you’re reading, not writing — that tradeoff is almost always worth it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21b37e1a485a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your Playwright getByRole Locator Isn't Working — Understanding ARIA and Accessible Names]]></title>
            <link>https://medium.com/@gurudatt.sa26/why-your-playwright-getbyrole-locator-isnt-working-understanding-aria-and-accessible-names-9179a417d54f?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9179a417d54f</guid>
            <category><![CDATA[accessibility]]></category>
            <category><![CDATA[playwrights]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 10:59:16 GMT</pubDate>
            <atom:updated>2026-04-04T11:09:05.513Z</atom:updated>
            <content:encoded><![CDATA[<p><em>You wrote the perfect locator. Playwright’s Pick Locator even agreed. But the test still fails. Here’s why — and how to never get stuck again.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*7T6CGidwjs012QmWZb0KkA.png" /></figure><h3>The Problem That Tripped Me Up</h3><p>I was automating the Playwright documentation site and needed to interact with the search box. The HTML looked straightforward:</p><pre>&lt;input<br>  class=&quot;DocSearch-Input&quot;<br>  aria-labelledby=&quot;docsearch-label&quot;<br>  placeholder=&quot;Search docs&quot;<br>  type=&quot;search&quot;<br>/&gt;</pre><p>I wrote what seemed like the obvious locator:</p><pre>page.getByRole(&#39;searchbox&#39;, { name: &#39;Search docs&#39; })</pre><p>And it failed. The test just hung there, waiting for an element that clearly existed on the page.</p><p>Then I tried:</p><pre>page.getByRole(&#39;searchbox&#39;, { name: &#39;Search&#39; })</pre><p>And it worked perfectly. But why? The placeholder clearly says “Search docs” — where did the “docs” go?</p><h3>The Root Cause: Accessible Name Resolution</h3><p>When you use getByRole with a name option, Playwright doesn&#39;t match against the placeholder text, the id, or any arbitrary attribute. It matches against the element&#39;s <strong>computed accessible name</strong> — the same name that screen readers announce to users.</p><p>The accessible name is resolved using a strict priority chain:</p><ol><li><strong>aria-labelledby</strong> — references another element&#39;s text content (highest priority)</li><li><strong>aria-label</strong> — directly provides a label string</li><li><strong>&lt;label&gt; element</strong> — via the for attribute</li><li><strong>placeholder</strong> / <strong>title</strong> — fallback (lowest priority)</li></ol><p>In my case, the input had aria-labelledby=&quot;docsearch-label&quot;. That attribute pointed to a &lt;label&gt; element elsewhere in the DOM with the text <strong>&quot;Search&quot;</strong> — not &quot;Search docs&quot;. Since aria-labelledby has the highest priority, it overrode the placeholder=&quot;Search docs&quot; entirely.</p><p>The placeholder was right there in front of me, but it was irrelevant to how Playwright resolved the name.</p><h3>Wait — input isn&#39;t a Role?</h3><p>Before we even get to the name issue, there’s another common confusion. When I first saw the &lt;input&gt; element, my instinct was to write:</p><pre>page.getByRole(&#39;input&#39;, { name: &#39;Search docs&#39; })</pre><p>This doesn’t work. input is an <strong>HTML tag</strong>, not an <strong>ARIA role</strong>. Playwright&#39;s getByRole uses ARIA roles, which are semantic descriptions of what an element does — not what tag it uses.</p><p>Here’s how common HTML elements map to ARIA roles:</p><p>HTML Element ARIA Role &lt;input type=&quot;text&quot;&gt; textbox &lt;input type=&quot;search&quot;&gt; searchbox &lt;input type=&quot;checkbox&quot;&gt; checkbox &lt;input type=&quot;radio&quot;&gt; radio &lt;button&gt; button &lt;a href=&quot;...&quot;&gt; link &lt;h1&gt; to &lt;h6&gt; heading &lt;select&gt; combobox &lt;ul&gt; list</p><p>These are called <strong>implicit ARIA roles</strong> — the browser assigns them automatically based on the element type.</p><h3>How to Debug: The Accessibility Tab</h3><p>Here’s the trick that would have saved me 30 minutes. Chrome DevTools has a built-in way to see exactly what Playwright sees:</p><ol><li>Right-click the element → <strong>Inspect</strong></li><li>In DevTools, find the <strong>Accessibility</strong> tab (next to Styles, Computed — you may need to click &gt;&gt; to reveal it)</li><li>Look at <strong>Computed Properties</strong> → <strong>Name</strong></li></ol><p>When I checked this for my search input, it showed:</p><pre>Name: &quot;Search&quot;<br>  ▸ aria-labelledby:<br>      label#docsearch-label.DocSearch-MagnifierLabel...<br>    aria-label: Not specified<br>    From label (for= attribute): ...<br>    title: Not specified<br>    placeholder: &quot;Search docs&quot;  ← strikethrough, overridden</pre><p>The Accessibility tab showed me the entire resolution chain. aria-labelledby resolved to &quot;Search&quot;, and placeholder was struck through because it was overridden. This is your <strong>source of truth</strong> whenever getByRole isn&#39;t matching what you expect.</p><h3>What About Partial Matching?</h3><p>Here’s another subtlety. getByRole does <strong>substring matching</strong> by default for the name option:</p><pre>// All of these match an element with accessible name &quot;Search docs&quot;<br>page.getByRole(&#39;searchbox&#39;, { name: &#39;Search&#39; })      // ✅ substring<br>page.getByRole(&#39;searchbox&#39;, { name: &#39;Search docs&#39; }) // ✅ full match<br>page.getByRole(&#39;searchbox&#39;, { name: &#39;docs&#39; })         // ✅ substring</pre><p>If you want an exact match, use the exact option:</p><pre>page.getByRole(&#39;searchbox&#39;, { name: &#39;Search&#39;, exact: true })  // ✅ exact<br>page.getByRole(&#39;searchbox&#39;, { name: &#39;docs&#39;, exact: true })    // ❌ won&#39;t match</pre><p>Using exact: true is generally recommended for robustness — it prevents accidentally matching the wrong element if the page has multiple elements with similar names.</p><p>In my case, Playwright’s Pick Locator suggested { name: &#39;Search&#39; } because it finds the <strong>shortest unique match</strong>. Since there was only one searchbox on the page, &quot;Search&quot; was sufficient.</p><h3>What is ARIA, Anyway?</h3><p>If you’ve been writing Playwright tests without thinking about ARIA, here’s a quick primer.</p><p><strong>ARIA</strong> stands for <strong>Accessible Rich Internet Applications</strong>. It’s a set of HTML attributes that describe UI elements for assistive technologies like screen readers. A sighted user can see that a styled &lt;div&gt; is acting as a button. A screen reader cannot — unless you tell it:</p><pre>&lt;!-- Screen reader has no idea this is a button --&gt;<br>&lt;div onclick=&quot;submit()&quot;&gt;Submit&lt;/div&gt;</pre><pre>&lt;!-- Now it knows --&gt;<br>&lt;div role=&quot;button&quot; aria-label=&quot;Submit&quot; onclick=&quot;submit()&quot;&gt;Submit&lt;/div&gt;</pre><p>ARIA provides three categories of attributes:</p><p><strong>Roles</strong> — what the element <em>is</em>: role=&quot;button&quot;, role=&quot;searchbox&quot;, role=&quot;dialog&quot;, role=&quot;navigation&quot;</p><p><strong>Properties</strong> — describe the element: aria-label=&quot;Search&quot;, aria-labelledby=&quot;label-id&quot;, aria-placeholder=&quot;Type here&quot;</p><p><strong>States</strong> — current condition: aria-expanded=&quot;true&quot;, aria-checked=&quot;false&quot;, aria-disabled=&quot;true&quot;</p><p>Native HTML elements come with implicit roles built in, so you usually don’t need to add ARIA manually. But when developers build custom components with &lt;div&gt; and &lt;span&gt;, ARIA becomes essential for accessibility — and for your Playwright tests.</p><h3>Practical Takeaways for Your POM</h3><p>Here’s how this knowledge applies to a real Page Object Model:</p><pre>import { Locator, Page } from &quot;@playwright/test&quot;;<br><br>export class HomePage {<br>  private readonly searchButton: Locator;<br>  private readonly searchBox: Locator;<br>  private readonly heading: Locator;<br>  constructor(private readonly page: Page) {<br>    this.searchButton = page.getByRole(&#39;button&#39;, { name: &#39;Search&#39; });<br>    this.searchBox = page.getByRole(&#39;searchbox&#39;, { name: &#39;Search&#39; });<br>    this.heading = page.getByRole(&#39;heading&#39;, { level: 1 });<br>  }<br>  // Returning a Locator - synchronous, no async needed<br>  getHeading(): Locator {<br>    return this.heading;<br>  }<br>  // Performing an action - async needed<br>  async clickSearch() {<br>    await this.searchButton.click();<br>  }<br>  // Performing an action - async needed<br>  async searchFor(term: string) {<br>    await this.searchButton.click();<br>    await this.searchBox.fill(term);<br>  }<br>}</pre><p>And in the test:</p><pre>// Locator assertions — pass the locator directly to expect<br>await expect(homePage.getHeading()).toHaveText(/Playwright enables reliable/);<br><br>// Action then assert<br>await homePage.clickSearch();<br>await expect(page.getByRole(&#39;searchbox&#39;, { name: &#39;Search&#39; })).toBeVisible();</pre><p>Notice that getHeading() returns a Locator synchronously — no async/await needed. The await happens on expect().toHaveText() because that&#39;s where Playwright polls the DOM and retries.</p><h3>The Debugging Checklist</h3><p>Next time getByRole doesn&#39;t work, run through this:</p><ol><li><strong>Am I using the right role?</strong> Check the ARIA role, not the HTML tag. &lt;input type=&quot;search&quot;&gt; is searchbox, not input.</li><li><strong>Am I using the right name?</strong> Open DevTools → Accessibility tab → check the <strong>computed Name</strong>. Don’t assume it matches the placeholder or visible text.</li><li><strong>Is </strong><strong>aria-labelledby overriding my expected name?</strong> This is the most common gotcha. aria-labelledby always wins over aria-label, &lt;label&gt;, and placeholder.</li><li><strong>Is the element even visible?</strong> Some elements only appear after an interaction (like clicking a search button to open a modal).</li><li><strong>Do I need </strong><strong>exact: true?</strong> If substring matching is catching the wrong element, lock it down with exact: true</li></ol><h3>Conclusion</h3><p>getByRole is Playwright&#39;s recommended locator strategy because it&#39;s resilient, semantic, and accessibility-aware. But it only works well when you understand what it&#39;s actually matching against — the ARIA role and the computed accessible name, not the HTML tag and the placeholder text.</p><p>The DevTools Accessibility tab is your best friend here. It shows you exactly what Playwright sees, with no guesswork. Once you build the habit of checking it, you’ll never get stuck on a getByRole mismatch again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9179a417d54f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playwright 1.59 Just Dropped — Here’s How Effective Testers Can Use Every New Feature]]></title>
            <link>https://medium.com/@gurudatt.sa26/playwright-1-59-just-dropped-heres-how-effective-testers-can-use-every-new-feature-908bc2cf76f3?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/908bc2cf76f3</guid>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[playwright-test]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 09:01:54 GMT</pubDate>
            <atom:updated>2026-04-01T09:01:54.051Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A practical guide to the most agentic Playwright release yet</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*588xGmDTe4uioLHeAPp4Ew.png" /></figure><p><a href="https://playwright.dev/docs/release-notes#version-159">Playwright 1.59</a> isn’t just another incremental update. It’s a statement about where test automation is heading — and that direction is deeply agentic. If you’re a tester who’s been waiting for your tooling to catch up with the AI-driven workflows everyone keeps talking about, this is the release that bridges the gap.</p><p>Let’s walk through every major feature and, more importantly, how you can put each one to work immediately.</p><h3>🎬 Screencast API — Your Tests Can Now Show Their Work</h3><p>The headline feature is the new page.screencast API, and it&#39;s not just video recording with a new name. It&#39;s a unified interface for capturing what&#39;s happening on the page with context baked in.</p><h3>What’s new</h3><p>The API gives you five capabilities under one roof: screencast recordings with start/stop control, action annotations that highlight interacted elements, visual overlays for chapter titles and custom HTML, real-time frame capture for streaming JPEG frames to external tools, and agentic video receipts for producing visual proof of automated work.</p><h3>How effective testers should use it</h3><p><strong>Replace flaky screenshot-on-failure with targeted screencasts.</strong> Instead of a single frozen frame when something breaks, wrap your critical flows in screencast recordings:</p><pre>await page.screencast.start({ path: &#39;checkout-flow.webm&#39; });<br>await page.screencast.showActions({ position: &#39;top-right&#39; });</pre><pre>// Your checkout test steps here<br>await page.locator(&#39;#add-to-cart&#39;).click();<br>await page.locator(&#39;#checkout&#39;).click();<br>await expect(page.locator(&#39;.order-confirmation&#39;)).toBeVisible();</pre><pre>await page.screencast.stop();</pre><p>Now when a test fails, you don’t just know <em>what</em> failed — you can see the entire sequence of events leading up to it, complete with annotations showing which elements were interacted with.</p><p><strong>Use chapters for long E2E flows.</strong> If you have a test that covers an entire user journey — login, browse, add to cart, checkout, confirm — chapters let you segment the recording:</p><pre>await page.screencast.showChapter(&#39;Authentication&#39;, {<br>  description: &#39;Logging in with test credentials&#39;,<br>  duration: 1000,<br>});<br>// login steps...</pre><pre>await page.screencast.showChapter(&#39;Product Selection&#39;, {<br>  description: &#39;Adding items to cart&#39;,<br>  duration: 1000,<br>});<br>// shopping steps...</pre><p>When you or a stakeholder reviews the video, chapters act as a table of contents. No more scrubbing through a 45-second video trying to find the moment things went wrong.</p><p><strong>Feed frames to AI vision models.</strong> The onFrame callback opens up a category of testing that was previously cumbersome — visual validation powered by AI:</p><pre>await page.screencast.start({<br>  onFrame: ({ data }) =&gt; sendToVisionModel(data),<br>  size: { width: 800, height: 600 },<br>});</pre><p>This is how you build a test that can answer questions like “does this page look broken?” without writing pixel-perfect screenshot assertions.</p><p><strong>Build agentic video receipts for CI.</strong> This is the feature I’m most excited about. If you’re running AI-powered test generation or self-healing tests, the agent can now produce a narrated video walkthrough of exactly what it did:</p><pre>await page.screencast.start({ path: &#39;receipt.webm&#39; });<br>await page.screencast.showActions({ position: &#39;top-right&#39; });</pre><pre>await page.screencast.showChapter(&#39;Verifying checkout flow&#39;, {<br>  description: &#39;Added coupon code support per ticket #1234&#39;,<br>});</pre><pre>await page.locator(&#39;#coupon&#39;).fill(&#39;SAVE20&#39;);<br>await page.locator(&#39;#apply-coupon&#39;).click();<br>await expect(page.locator(&#39;.discount&#39;)).toContainText(&#39;20%&#39;);</pre><pre>await page.screencast.showChapter(&#39;Done&#39;, {<br>  description: &#39;Coupon applied, discount reflected in total&#39;,<br>});</pre><pre>await page.screencast.stop();</pre><p>The resulting video is faster to review than reading through text logs. Attach it to your CI artifacts or post it to Slack — stakeholders who would never read a test log will absolutely watch a 30-second annotated video.</p><h3>🔗 Browser Bind API — Multi-Client Browser Sharing</h3><p>The new browser.bind() API lets you make a running browser available to multiple clients simultaneously — playwright-cli, @playwright/mcp, or other Playwright API consumers.</p><h3>What’s new</h3><p>You can bind a browser session with a name, then connect to it from anywhere:</p><pre>const { endpoint } = await browser.bind(&#39;my-session&#39;, {<br>  workspaceDir: &#39;/my/project&#39;,<br>});</pre><p>Then from the CLI:</p><pre>playwright-cli attach my-session<br>playwright-cli -s my-session snapshot</pre><p>Or from the MCP server:</p><pre>@playwright/mcp --endpoint=my-session</pre><p>Multiple clients can connect at the same time, and you can bind over WebSocket by passing host and port options.</p><h3>How effective testers should use it</h3><p><strong>Debug agent-driven tests in real time.</strong> If you have an AI agent running tests, bind the browser it’s using and attach to it from your terminal. You can watch what it’s doing, take snapshots, and intervene manually — without stopping the agent.</p><p><strong>Share a browser state across tools.</strong> Set up authentication in one client, then connect from another to run tests against the authenticated session. This eliminates the “log in again” overhead when switching between manual exploration and automated testing.</p><p><strong>Pair with MCP servers for hybrid workflows.</strong> Bind a browser from your test setup, then point your MCP-enabled IDE at it. Your coding agent can see and interact with the same browser your tests are running against.</p><h3>📊 Dashboard — See What Your Agents Are Doing</h3><p>Run playwright-cli show to open a dashboard that lists all bound browsers, their statuses, and lets you interact with them.</p><h3>How effective testers should use it</h3><p><strong>Monitor parallel test runs visually.</strong> If you’re running tests across multiple browsers or multiple agents, the dashboard gives you a live bird’s-eye view of what each one is doing. Click into any session for manual intervention.</p><p><strong>Use it in your daily workflow.</strong> Pass PLAYWRIGHT_DASHBOARD=1 as an environment variable to see all @playwright/test browsers in the dashboard. This turns your local test runs into something you can observe and debug visually, rather than staring at terminal output.</p><p><strong>Open DevTools on running sessions.</strong> The dashboard lets you open DevTools to inspect pages from background browsers. This is invaluable when a test is hanging and you need to understand what state the page is in.</p><h3>🐛 CLI Debugger for Agents — --debug=cli</h3><p>Coding agents can now run npx playwright test --debug=cli to attach and debug tests over playwright-cli. This is purpose-built for agentic workflows where an AI is fixing failing tests.</p><h3>How effective testers should use it</h3><p><strong>Let your AI agent step through failing tests.</strong> When a test breaks, an agent can attach to the debugger, step through the test line by line, inspect the page state at each step, and figure out what went wrong — all from the command line, without needing a GUI.</p><pre>$ npx playwright test --debug=cli<br>### Debugging Instructions<br>- Run &quot;playwright-cli attach tw-87b59e&quot; to attach to this test</pre><pre>$ playwright-cli attach tw-87b59e<br>$ playwright-cli --session tw-87b59e step-over</pre><p><strong>Build self-healing test pipelines.</strong> Combine --debug=cli with the healer agent from npx playwright init-agents. When a test fails in CI, the healer can attach via CLI debug, understand the failure, and propose a fix — all without human intervention.</p><h3>📋 CLI Trace Analysis — Traces Without the GUI</h3><p>Agents can now run npx playwright trace to explore Playwright Traces directly from the command line.</p><h3>How effective testers should use it</h3><p><strong>Analyze failures in headless CI environments.</strong> You no longer need to download trace files and open them in a browser. An agent (or a script) can parse the trace, extract the relevant information, and either fix the test or file a detailed bug report.</p><p><strong>Feed trace data into your reporting pipeline.</strong> Extract timing information, network requests, and action sequences from traces programmatically. Build custom dashboards or feed the data into your test analytics tool.</p><h3>Putting It All Together: The Agentic Testing Loop</h3><p>Here’s where 1.59 really shines — when you combine these features into a cohesive workflow:</p><ol><li><strong>Run tests</strong> with PLAYWRIGHT_DASHBOARD=1 to monitor them visually.</li><li><strong>When a test fails</strong>, the healer agent attaches via --debug=cli.</li><li><strong>The agent analyzes the trace</strong> using npx playwright trace.</li><li><strong>It fixes the test</strong> and runs it again.</li><li><strong>It records a screencast receipt</strong> of the passing test with chapter annotations.</li><li><strong>The receipt video</strong> gets attached to the CI artifacts for human review.</li></ol><p>This is the test automation flywheel that Playwright 1.59 enables. The human tester shifts from “person who writes and debugs tests” to “person who reviews agent-produced evidence and makes strategic decisions about test coverage.”</p><h3>Who Benefits Most</h3><ul><li><strong>QA Leads and SDETs</strong> who are building automation frameworks: the Screencast API and Browser Bind give you infrastructure-level capabilities that were previously custom-built.</li><li><strong>Teams using AI-assisted testing</strong>: the CLI debugger and trace analysis features are designed specifically for agentic workflows.</li><li><strong>Anyone who struggles with test failure diagnosis</strong>: screencast recordings with action annotations replace the “what happened?” guessing game with visual evidence.</li><li><strong>Stakeholders who don’t read test reports</strong>: a 30-second annotated video is worth a thousand log lines.</li></ul><h3>Getting Started</h3><p>Upgrade to Playwright 1.59:</p><pre>npm install -D @playwright/test@latest<br>npx playwright install</pre><p>Start with the Screencast API — it has the lowest barrier to entry and the highest immediate impact. Record your most critical E2E flow, add a few chapter markers, enable action annotations, and share the resulting video with your team. Once you see the reaction, you’ll want to instrument everything.</p><p>Then explore browser.bind() and the dashboard. These are the features that will fundamentally change how you interact with your test infrastructure — from passively reading logs to actively observing and intervening in running tests.</p><p>The future of testing is observable, explainable, and visual. Playwright 1.59 just made that future available today.</p><p><em>If you found this useful, follow me for more articles on AI-driven test automation, Playwright advanced patterns, and building autonomous QA systems.</em></p><p><em>What feature are you most excited about? Drop a comment below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=908bc2cf76f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playwright Fixture Scopes: Know When to Use What]]></title>
            <link>https://medium.com/@gurudatt.sa26/playwright-fixture-scopes-know-when-to-use-what-419a79fd9e4e?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/419a79fd9e4e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[playwrights]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 09:23:44 GMT</pubDate>
            <atom:updated>2026-03-26T09:23:44.724Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering test, worker and file — and the default that protects you from yourself</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sePoGOFBT-tx2sEHwhQiAA.png" /></figure><p>If you have written more than a handful of Playwright tests, you have used fixtures — browser, page, request, and all the custom ones you bolt on top. But have you ever stopped to ask: <em>what is the lifetime of this fixture?</em> Is it recreated for every test? Shared across a file? Does it die when the worker shuts down?</p><p>These questions matter enormously for both performance and correctness. Get the scope wrong and you end up with a suite that recreates expensive resources thousands of times needlessly — or worse, tests that bleed state into each other and produce phantom failures.</p><blockquote><em>💡 </em><strong><em>Already familiar with the basics?</em></strong><em> Jump straight to the Decision Framework section for a quick cheat-sheet you can paste into your team wiki.</em></blockquote><h3>1. What Is a Fixture Scope?</h3><p>A fixture’s scope controls two things:</p><ul><li><strong>When is the fixture set up</strong> — the code before await use(...)</li><li><strong>When is it torn down</strong> — the code after await use(...)</li></ul><p>Playwright exposes three scope values: test, worker, and file. You declare scope inside the options object — the second element of the fixture tuple:</p><pre>import { test as base } from &#39;@playwright/test&#39;;<br>const test = base.extend&lt;{ myFixture: string }&gt;({<br>  myFixture: [<br>    async ({}, use) =&gt; {<br>      await use(&#39;hello&#39;);<br>    },<br>    { scope: &#39;test&#39; }   // &lt;-- scope defined here<br>  ],<br>});</pre><p>The plain function shorthand (no tuple) always implies test scope. The tuple is only needed when you want a different scope, or when you need auto: true.</p><h3>2. test Scope — The Default</h3><p>When you define a fixture without specifying a scope, Playwright uses test scope. This is a deliberate, safe default: every test gets a completely fresh fixture instance.</p><pre>// These two declarations are exactly equivalent<br><br>// ① Implicit test scope<br>const test = base.extend&lt;{ db: Database }&gt;({<br>  db: async ({}, use) =&gt; {<br>    const db = await Database.connect();<br>    await use(db);<br>    await db.close();<br>  },<br>});<br><br>// ② Explicit test scope<br>const test = base.extend&lt;{ db: Database }&gt;({<br>  db: [<br>    async ({}, use) =&gt; {<br>      const db = await Database.connect();<br>      await use(db);<br>      await db.close();<br>    },<br>    { scope: &#39;test&#39; },<br>  ],<br>});</pre><h3>What happens at runtime</h3><p>Playwright calls your setup code before every test that requests the fixture. Once the test finishes — pass or fail — everything after await use(...) runs as teardown. The next test gets a brand-new instance.</p><pre>test(&#39;first test&#39;, async ({ db }) =&gt; {<br>  // db instance #1 — fresh connection<br>  await db.insert({ name: &#39;Alice&#39; });<br>});<br><br>test(&#39;second test&#39;, async ({ db }) =&gt; {<br>  // db instance #2 - completely new connection<br>  // Alice does NOT exist here unless seeded again<br>});</pre><blockquote><em>⚠️ </em><strong><em>Cost consideration:</em></strong><em> If your fixture takes 200ms to set up (a typical cold DB connection) and you have 500 tests using it, you are adding 100 seconds of pure overhead. That is exactly when you should evaluate </em><em>worker or </em><em>file scope.</em></blockquote><h3>3. worker Scope — One Instance per Worker Process</h3><p>A worker-scoped fixture is created once when a Playwright worker process starts, shared by every test that worker runs, and torn down only when the worker shuts down.</p><p>Playwright runs tests in parallel using multiple worker processes (controlled by --workers). A worker-scoped fixture is <strong>not</strong> shared across workers — it is shared <em>within</em> a single worker.</p><pre>import { test as base, Browser } from &#39;@playwright/test&#39;;<br>// Worker fixtures go in the SECOND generic parameter<br>const test = base.extend&lt;{}, { sharedBrowser: Browser }&gt;({<br>  sharedBrowser: [<br>    async ({ playwright }, use) =&gt; {<br>      console.log(&#39;Browser launched once for this worker&#39;);<br>      const browser = await playwright.chromium.launch();<br>      await use(browser);<br>      await browser.close();<br>    },<br>    { scope: &#39;worker&#39; },<br>  ],<br>});<br>// Both tests share the same browser instance<br>test(&#39;login page&#39;, async ({ sharedBrowser }) =&gt; {<br>  const page = await sharedBrowser.newPage();<br>  // ...<br>});<br>test(&#39;dashboard&#39;, async ({ sharedBrowser }) =&gt; {<br>  const page = await sharedBrowser.newPage();<br>  // ...<br>});</pre><h3>Why the second generic parameter?</h3><p>TypeScript requires worker-scoped fixtures in the WorkerFixtures type (second generic) rather than TestFixtures (first generic). This is a compile-time signal that the fixture outlives any individual test — useful for code reviews and onboarding.</p><h3>When to reach for worker scope</h3><ul><li><strong>Database connection pooling.</strong> Opening and closing a DB connection per-test is often the #1 source of suite slowness.</li><li><strong>Auth tokens / sessions.</strong> Log in via API once to get a JWT. If the token is valid for an hour, create it once per worker — not once per test.</li><li><strong>Static seed data.</strong> Configuration tables and lookup lists that never mutate during the test run.</li><li><strong>External SDK clients.</strong> AWS S3, Stripe, and third-party API wrappers rarely need per-test instantiation.</li></ul><blockquote><em>🚨 </em><strong><em>Isolation warning:</em></strong><em> Worker-scoped fixtures are shared — any mutation one test makes is visible to every subsequent test in that worker. Use </em><em>worker scope only for resources that are either read-only during tests, or that you explicitly reset via a test-scoped isolation wrapper.</em></blockquote><h3>4. file Scope — One Instance per Spec File</h3><p>file scope sits between test and worker. The fixture is created when the first test in a file runs, shared by all tests in that file, and torn down after the last test in the file finishes.</p><pre>const test = base.extend&lt;{}, { fileServer: HttpServer }&gt;({<br>  fileServer: [<br>    async ({}, use) =&gt; {<br>      const server = await startMockServer({ port: 0 });<br>      await use(server);<br>      await server.stop();<br>    },<br>    { scope: &#39;file&#39; },<br>  ],<br>});<br><br>// All tests in THIS FILE share the same mock server.<br>// Tests in other files get their own instance.<br>test(&#39;GET /users returns 200&#39;, async ({ fileServer, request }) =&gt; {<br>  const res = await request.get(`${fileServer.url}/users`);<br>  expect(res.status()).toBe(200);<br>});<br>test(&#39;POST /users creates a record&#39;, async ({ fileServer, request }) =&gt; {<br>  const res = await request.post(`${fileServer.url}/users`, {<br>    data: { name: &#39;Bob&#39; }<br>  });<br>  expect(res.status()).toBe(201);<br>});</pre><h3>When to reach for file scope</h3><ul><li><strong>Feature-specific suites.</strong> A checkout.spec.ts can spin up one mock payment gateway shared by all checkout tests.</li><li><strong>The </strong><strong>beforeAll equivalent.</strong> File scope is the Playwright analog of Jest&#39;s beforeAll — but expressed as a composable fixture rather than a lifecycle hook.</li><li><strong>Expensive UI state.</strong> Navigate to a complex SPA state once, then run multiple read-only assertions without re-navigating each time.</li></ul><h3>5. Scope Comparison at a Glance</h3><p>Scope Lifetime Default? Best For test Single test only ✅ Yes Isolated unit-like tests, UI state-sensitive tests worker Entire worker process No DB connections, auth tokens, SDK clients file All tests in one file No Feature-level context, beforeAll equivalent patterns</p><h3>6. Composing Scopes: The Classic Pattern</h3><p>The most powerful fixture pattern layers a worker-scoped base resource with a test-scoped wrapper that handles per-test isolation — the best of both worlds.</p><pre>const test = base.extend&lt;<br>  { db: DatabaseClient },         // test-scoped  → TestFixtures (1st generic)<br>  { dbPool: ConnectionPool }      // worker-scoped → WorkerFixtures (2nd generic)<br>&gt;({<br><br>// Worker scope: open the pool once per worker<br>  dbPool: [<br>    async ({}, use) =&gt; {<br>      const pool = await createPool({ max: 5 });<br>      await use(pool);<br>      await pool.end();<br>    },<br>    { scope: &#39;worker&#39; },<br>  ],<br>  // Test scope: borrow a connection, wrap in a transaction, rollback after each test<br>  db: async ({ dbPool }, use) =&gt; {<br>    const client = await dbPool.connect();<br>    await client.query(&#39;BEGIN&#39;);<br>    await use(client);<br>    await client.query(&#39;ROLLBACK&#39;);  // no mutations survive to the next test<br>    client.release();<br>  },<br>});<br>test(&#39;inserts a user&#39;, async ({ db }) =&gt; {<br>  await db.query(&#39;INSERT INTO users(name) VALUES($1)&#39;, [&#39;Eve&#39;]);<br>  const { rows } = await db.query(&#39;SELECT * FROM users&#39;);<br>  expect(rows).toHaveLength(1);<br>  // ROLLBACK fires after this test - next test starts clean<br>});</pre><p>The pool is expensive — it lives at worker scope. The transaction wrapper is cheap and gives per-test isolation — it lives at test scope. You get connection reuse <em>and</em> test independence.</p><h3>7. Decision Framework</h3><p>When you write test.extend(), run through this tree:</p><pre>Is the fixture EXPENSIVE to create (&gt;50ms)?<br>  ├── YES → Can tests safely share it without isolation issues?<br>  │         ├── YES → worker scope<br>  │         └── NO  → worker scope + test scope isolation wrapper<br>  │                   (transaction rollback, temp namespace, cleanup hook)<br>  └── NO  → test scope  ← the default, start here<br><br>Is it expensive AND logically tied to one spec file?<br>  └── YES → file scope<br>Does every test need a completely fresh instance regardless of cost?<br>  └── YES → test scope - always</pre><p>As a rule of thumb:</p><ul><li><strong>test</strong> is your baseline. Start here. Only change scope if you have a measured performance problem or an explicit sharing requirement.</li><li><strong>worker</strong> when setup is genuinely expensive and isolation is handled at another layer (transactions, namespacing, cleanup hooks).</li><li><strong>file</strong> when a resource is meaningful only within one spec file and you want clean setup/teardown semantics without littering beforeAll/afterAll blocks across the file.</li></ul><h3>8. Real-World Anti-Patterns</h3><h3>Anti-pattern 1: Worker-scoping everything for performance</h3><p>Tempting, but dangerous. If a test mutates shared state and the next test assumes a clean slate, you get flaky tests that pass locally but fail in CI. Only use worker scope when the shared resource is either immutable during tests, or you have an explicit reset mechanism in place.</p><h3>Anti-pattern 2: Worker fixture depending on a test fixture</h3><p>A worker fixture is set up before any test runs, so it cannot depend on something that only exists within a test. Playwright throws at startup if you attempt this.</p><pre>// ❌ INVALID — worker fixture depending on a test fixture<br>const test = base.extend({<br>  testFixture: async ({}, use) =&gt; { await use(&#39;value&#39;); },<br><br>workerFixture: [<br>    async ({ testFixture }, use) =&gt; {   // ← Playwright will throw at startup<br>      await use(testFixture);<br>    },<br>    { scope: &#39;worker&#39; },<br>  ],<br>});<br>// ✅ VALID - test fixture depending on a worker fixture<br>const test = base.extend&lt;<br>  { testFixture: string },<br>  { workerFixture: string }<br>&gt;({<br>  workerFixture: [<br>    async ({}, use) =&gt; { await use(&#39;shared&#39;); },<br>    { scope: &#39;worker&#39; },<br>  ],<br>  testFixture: async ({ workerFixture }, use) =&gt; {<br>    await use(`${workerFixture}-per-test`);<br>  },<br>});</pre><h3>Anti-pattern 3: Treating file scope as a direct beforeAll replacement</h3><p>File scope is <em>better</em> than beforeAll, but not identical. Playwright can distribute tests from the same file across multiple workers when using --shard or fullyParallel: true. In that case each worker gets its own file-scoped instance. Don&#39;t assume a file-scoped fixture is a process-global singleton.</p><h3>9. Scope &amp; Parallelism</h3><p>This is where most engineers get surprised. Here is the mental model for a 4-worker run:</p><pre>workers: 4<br><br>Worker 1:  worker-scope created once  |  test-scope × N tests<br>Worker 2:  worker-scope created once  |  test-scope × M tests<br>Worker 3:  worker-scope created once  |  test-scope × P tests<br>Worker 4:  worker-scope created once  |  test-scope × Q tests<br>// worker-scoped fixture is NOT shared between Worker 1 and Worker 2<br>// Each worker process has its own isolated instance</pre><p>Practical implication: with 4 workers and a worker-scoped DB fixture, you will have <strong>4 open connections</strong>. Factor this into your connection pool configuration. If your DB allows only 10 connections and you spin up 10 workers, you will hit connection exhaustion.</p><h3>10. Quick Reference</h3><p>Scope Setup / Teardown Shared Between TypeScript Generic test Before / after each test Nothing — fully isolated TestFixtures (1st param) worker Worker start / exit All tests on that worker WorkerFixtures (2nd param) file First / last test in file All tests in same file (on same worker) WorkerFixtures (2nd param)</p><h3>Wrapping Up</h3><p>Playwright’s fixture system is one of its most powerful features, and scope is the lever that lets you tune the balance between isolation and performance. The default of test scope keeps you safe — but understanding when to reach for worker or file scope, and how to compose them, is what separates a functional test suite from a fast, robust, and maintainable one.</p><p>Key takeaways:</p><ul><li><strong>test</strong> is the default and safest choice. Start here, always.</li><li><strong>worker</strong> is for expensive, shareable resources — always pair it with an isolation strategy.</li><li><strong>file</strong> is for resources meaningful at the feature/spec level.</li><li>The <strong>worker pool + test-scoped transaction rollback</strong> pattern is your best friend for database-heavy suites.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=419a79fd9e4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SKILL.md Is Just a Text File — And That’s the Point]]></title>
            <link>https://medium.com/@gurudatt.sa26/skill-md-is-just-a-text-file-and-thats-the-point-a28c79ae32da?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a28c79ae32da</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[claude]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Fri, 20 Mar 2026 13:53:22 GMT</pubDate>
            <atom:updated>2026-03-20T13:53:22.658Z</atom:updated>
            <content:encoded><![CDATA[<h3>How LLMs “learn” your standards, why it works everywhere, and what it means for AI-driven test automation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zlmfKqz26urxSi7I-T24Kg.png" /></figure><p>There is a moment every developer has when working with AI tools where they wonder: <em>how does the AI know my conventions?</em> How does Claude know to use Page Object Model? How does Cursor know to write tests with data-testid locators? How does my agent know to follow my framework structure?</p><p>The answer is almost disappointingly simple.</p><p>You told it. In a text file.</p><h3>The Mechanism Is Just Prompt Injection</h3><p>Every AI tool that “knows your standards” is doing the same thing under the hood:</p><ol><li>Read a text file</li><li>Paste it at the top of the prompt</li><li>Send the prompt to the LLM</li></ol><p>That’s it. No fine-tuning. No model training. No magic. Just text prepended to your request.</p><p>Here’s what it looks like in code:</p><pre>const skill    = fs.readFileSync(&#39;./SKILL.md&#39;, &#39;utf-8&#39;);<br>const prompt   = `Follow these standards:\n${skill}\n\nNow do: ${userRequest}`;<br>const response = await llm.generate(prompt);</pre><p>When Anthropic’s Claude reads a SKILL.md file, when Cursor reads .cursorrules, when GitHub Copilot reads copilot-instructions.md — they are all doing a variation of this exact thing. The file format differs. The underlying mechanism is identical.</p><h3>Why This Matters for Test Automation</h3><p>In traditional test automation, “standards” live in your head, in a wiki nobody reads, or in code reviews that catch violations after the fact. The feedback loop is slow.</p><p>With LLM-driven test generation, you can encode your standards in a text file and have every generated test follow them — automatically, every time. The feedback loop collapses to zero.</p><p>Here is a real example. Imagine you have built an autonomous UI exploration agent — a system where an LLM observes a page’s DOM, decides what to click, executes the action via Playwright, observes the result, and loops until a goal is reached. After the agent finishes exploring, it has a history like this:</p><pre>Step 1: Navigated to https://www.saucedemo.com<br>Step 2: Typed &quot;standard_user&quot; into #user-name<br>Step 3: Typed &quot;secret_sauce&quot; into #password<br>Step 4: Clicked #login-button<br>Step 5: URL is now /inventory</pre><p>Without a standards file, asking the LLM to convert this into a test might produce anything — raw Playwright with hardcoded selectors, no page objects, assertions on raw text, inconsistent naming. Technically correct, architecturally wrong.</p><p>With a SKILL.md:</p><pre># Playwright Test Standards</pre><pre>## Structure<br>- Page Object Model — every page gets its own class in /pages<br>- Locator priority: data-testid &gt; aria-label &gt; id<br>- Never use CSS selectors or XPath</pre><pre>## Assertions<br>- After login: assert URL contains /inventory<br>- Use expect() after every significant navigation</pre><pre>## Naming<br>- test() names describe user actions, not technical steps<br>- Good: &quot;user can login with valid credentials&quot;<br>- Bad: &quot;test login function&quot;</pre><p>The same history now produces a test that fits your framework exactly. The LLM did not change. Your prompt did.</p><h3>The SKILL.md Is Not Claude-Specific</h3><p>This is the part most developers miss. SKILL.md sounds like an Anthropic concept. It is not.</p><p>Anthropic’s tooling happens to look for files in /mnt/skills/ and inject them automatically. But the file itself — the text, the standards, the conventions — works with any LLM. You just have to inject it yourself.</p><p>Tool How it reads your standards Claude (claude.ai) Auto-injects from /mnt/skills/ Your agent (Ollama/Gemini/Claude API) fs.readFileSync(&#39;./SKILL.md&#39;) Cursor .cursorrules file in project root GitHub Copilot copilot-instructions.md Any LLM API Read file, paste into system prompt</p><p>Same document. Same standards. Every tool. Write it once, use it everywhere.</p><h3>Building It Into Your Agent</h3><p>Here is how to wire a SKILL.md into a test generation agent in TypeScript:</p><pre>import fs from &#39;fs&#39;;</pre><pre>async function generateTest(goal: string, history: string[]): Promise&lt;string&gt; {</pre><pre>  const skill = fs.readFileSync(&#39;./SKILL.md&#39;, &#39;utf-8&#39;);</pre><pre>  const prompt = `You are a Playwright test engineer.<br>You must follow these standards exactly:</pre><pre>&lt;standards&gt;<br>${skill}<br>&lt;/standards&gt;</pre><pre>Convert this browser automation history into a Playwright test<br>that strictly follows the standards above.</pre><pre>Goal: ${goal}</pre><pre>History:<br>${history.join(&#39;\n&#39;)}</pre><pre>Respond with only the TypeScript code, no explanation.`;</pre><pre>  const response = await fetch(&#39;http://localhost:11434/api/generate&#39;, {<br>    method: &#39;POST&#39;,<br>    headers: { &#39;Content-Type&#39;: &#39;application/json&#39; },<br>    body: JSON.stringify({<br>      model: &#39;llama3.2&#39;,<br>      prompt,<br>      stream: false,<br>      options: { temperature: 0.1 }<br>    })<br>  });</pre><pre>  const data = await response.json() as { response: string };<br>  return data.response.trim()<br>    .replace(/^```typescript\n?/, &#39;&#39;)<br>    .replace(/\n?```$/, &#39;&#39;);<br>}</pre><p>Notice temperature: 0.1. When you want the LLM to follow instructions precisely — not be creative, not improvise — low temperature is essential. Standards compliance is a precision task, not a creative one.</p><h3>The Full Picture: Agent → Standards → Test → CI</h3><p>The autonomous test generation pipeline looks like this:</p><pre>1. Agent runs once<br>   └── Explores the UI, records every action</pre><pre>2. generateTest() is called<br>   └── SKILL.md + history → LLM → .spec.ts file</pre><pre>3. Generated test runs in CI forever<br>   └── No LLM involved at runtime. Zero cost. Full speed.</pre><p>The LLM is a test <em>authoring</em> tool, not a test <em>execution</em> tool. This distinction matters enormously for cost and reliability. You pay for LLM tokens once during authoring. The generated test runs in CI for free, at Playwright speed, with full determinism.</p><h3>What Makes a Good SKILL.md</h3><p>After working with this pattern across multiple frameworks, a few principles stand out.</p><p><strong>Be explicit, not implicit.</strong> “Use good locators” is useless. “Use data-testid first, then aria-label, then id. Never use CSS selectors or XPath.&quot; is actionable.</p><p><strong>Include examples.</strong> LLMs follow examples better than rules. Show the pattern you want, not just the description of the pattern.</p><p><strong>Keep it short.</strong> A 500-line standards document will be ignored — by humans and LLMs alike. Aim for the 20% of conventions that catch 80% of problems.</p><p><strong>Version control it.</strong> Your SKILL.md is as important as your framework code. It should live in the repo, be reviewed like code, and evolve as your standards evolve.</p><p><strong>Make it LLM-agnostic.</strong> Write standards in plain English. Avoid tool-specific jargon. A SKILL.md that works with Claude should work equally well with Ollama or Gemini.</p><h3>The Broader Implication</h3><p>The reason this pattern matters beyond test automation is that it reframes what “AI that knows your codebase” actually means.</p><p>It does not mean the AI has been trained on your code. It means you have written down what matters and given that text to the AI at the moment it needs to act. The AI’s capability is general. The standards file makes that capability specific to your context.</p><p>This is a skill worth developing — not prompting, exactly, but <em>context curation</em>. Knowing what to put in the file, how to phrase it so an LLM follows it reliably, how to keep it maintained as your standards evolve. That skill transfers across every AI tool you will ever use, because the mechanism underneath them is always the same.</p><p>Read a text file. Paste it into the prompt. Send it to the LLM.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a28c79ae32da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Re-Explaining Your Test Conventions to Claude — Use SKILL.md]]></title>
            <link>https://medium.com/@gurudatt.sa26/stop-re-explaining-your-test-conventions-to-claude-use-skill-md-41a8a4d5d9ea?source=rss-b0c4aa941b7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/41a8a4d5d9ea</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[test-automation]]></category>
            <dc:creator><![CDATA[Gurudatt S A]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 07:35:50 GMT</pubDate>
            <atom:updated>2026-03-15T07:35:50.351Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Your AI pair programmer finally follows your rules, every single time.</em></blockquote><p>If you’ve been using Claude (or any AI assistant) to help write Playwright tests, you’ve likely hit this wall: you describe your project’s conventions once, get a beautifully structured test back, and then the next prompt starts from zero again. Generic locators. No fixtures. The wrong assertion style. You end up copy-pasting your conventions into every prompt like a broken record.</p><p>There’s a better way. It’s called <strong>SKILL.md</strong>.</p><h3>What Is a SKILL.md?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*-uKQfwFO0-71UbKinrTBew.png" /></figure><p>A SKILL.md is a structured markdown file — with YAML frontmatter — that tells Claude <em>when</em> to engage a specific behaviour and <em>how</em> to behave when it does. Think of it as a <strong>persistent persona</strong> for Claude within your domain.</p><p>Once defined, Claude loads this skill automatically whenever it detects a matching context. You stop re-explaining. Your team stops re-explaining. The AI pair programmer finally learns your rules.</p><pre>your-project/<br>└── .claude/<br>    └── skills/<br>        └── playwright-automation/<br>            ├── SKILL.md               ← core instructions<br>            └── references/<br>                ├── locator-strategy.md<br>                ├── page-object-template.md<br>                └── ci-config.md</pre><h3>The Three-Level Loading System</h3><p>Before diving into the Playwright specifics, it’s worth understanding the architecture. Skills use a <strong>progressive disclosure</strong> model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*k-IKDgxMdqWm7SAK8N5JpA.png" /></figure><p>For test automation, this is powerful. The heavy stuff — your full locator decision tree, your page object template, your CI sharding config — lives in bundled reference files and only gets loaded when the task actually needs it. Claude’s context window stays lean.</p><h3>A Real Playwright SKILL.md</h3><p>Here’s a production-ready example. Adapt it to your team’s conventions.</p><pre>---<br>name: playwright-automation<br>description: &gt;<br>  Use when writing, reviewing, or generating Playwright tests, page objects,<br>  fixtures, or CI configuration. Triggers on: test generation, locator<br>  strategy, POM scaffolding, fixture setup, assertion patterns, flakiness<br>  investigation, or Playwright CI pipeline setup. Also use when the user<br>  asks to refactor existing tests or review test quality.<br>---<br><br># Playwright Automation Conventions<br><br>## Locator Hierarchy<br>Always prefer locators in this order:<br>1. `data-testid` attributes — the primary contract with the dev team<br>2. ARIA roles — `getByRole(&#39;button&#39;, { name: /submit/i })`<br>3. Labels — `getByLabel(&#39;Email address&#39;)`<br>4. Placeholder — `getByPlaceholder`<br>5. CSS selectors — **only as last resort**, never use nth-child or positional selectors<br><br>Never use XPath unless migrating a legacy suite.<br><br>## Page Object Model<br>- One class per page/component<br>- Constructor receives `page: Page`<br>- Methods return `Promise&lt;void&gt;` or the next page object (for navigation chains)<br>- No assertions inside page objects — keep them in the test layer<br>- Locators are declared as readonly class properties, not inline<br><br>See `references/page-object-template.md` for the scaffold.<br><br>## Assertions<br>- Always use `expect(locator).toBeVisible()` over `waitFor()` for element checks<br>- Prefer `toHaveText()` with regex for resilient text matching: `toHaveText(/welcome/i)`<br>- Never use `page.waitForTimeout()` — always use auto-waiting or `waitForLoadState`<br>- For API assertions: use `expect(response.status()).toBe(200)`<br><br>## Fixtures<br>- All shared state (auth, db seed, page objects) lives in `fixtures/index.ts`<br>- Extend the base `test` from `@playwright/test` — never import it directly in spec files<br>- Worker-scoped fixtures for expensive setup (auth sessions)<br>- Test-scoped fixtures for stateful objects (page objects)<br><br>## Test Structure<br>- Follow the Arrange / Act / Assert pattern, separated by blank lines<br>- One logical scenario per test — no multi-scenario tests<br>- Use `test.step()` for complex flows to improve trace readability<br>- Tag tests with `@smoke`, `@regression`, `@api` using `test.describe` metadata<br><br>## CI Configuration<br>- Always configure with sharding for suites &gt; 50 tests<br>- Use `--reporter=blob` in CI, merge with `merge-reports` post-run<br>- Attach traces on failure only: `trace: &#39;on-first-retry&#39;`<br><br>See `references/ci-config.md` for GitHub Actions and Bitbucket Pipeline templates.<br><br>## What to Avoid<br>- Never scaffold tests without fixtures if the project already has them<br>- Never generate `expect(page).toMatchSnapshot()` unless explicitly asked<br>- Never use hardcoded `baseURL` — always use `playwright.config.ts` env vars</pre><h3>The Bundled References That Make It Powerful</h3><p>The real leverage comes from what you put in the references/ folder. Claude pulls these in when the task calls for them — you don&#39;t pay context cost upfront.</p><h3>references/locator-strategy.md</h3><p>A decision flowchart your team has agreed on. Include examples of good vs. bad locators, and explanations of <em>why</em> the hierarchy exists. This becomes the single source of truth when onboarding new engineers too.</p><h3>references/page-object-template.md</h3><pre>// Template: PageObject scaffold<br>import { type Page, type Locator } from &#39;@playwright/test&#39;;<br><br>export class FeaturePage {<br>  readonly page: Page;<br>  readonly heading: Locator;<br>  readonly submitButton: Locator;<br><br>  constructor(page: Page) {<br>    this.page = page;<br>    this.heading = page.getByRole(&#39;heading&#39;, { level: 1 });<br>    this.submitButton = page.getByRole(&#39;button&#39;, { name: /submit/i });<br>  }<br><br>  async goto() {<br>    await this.page.goto(&#39;/feature&#39;);<br>  }<br><br>  async submit() {<br>    await this.submitButton.click();<br>  }<br>}</pre><p>Bundle this as a reference, and when you ask Claude to “create a page object for the checkout flow”, it scaffolds correctly without you specifying the structure.</p><h3>references/ci-config.md</h3><p>Include your sharding setup, reporter config, and any environment-specific nuances. For teams using GitHub Actions with matrix sharding:</p><pre># Sharding pattern — reference for Claude<br>strategy:<br>  matrix:<br>    shard: [1/4, 2/4, 3/4, 4/4]<br>steps:<br>  - run: npx playwright test --shard=${{ matrix.shard }}<br>  - uses: actions/upload-artifact@v4<br>    with:<br>      name: blob-report-${{ matrix.shard }}<br>      path: blob-report/</pre><p>Now when you ask Claude to “add CI config for our Playwright suite”, it doesn’t guess — it follows your established pattern.</p><h3>Before vs. After: The Real Difference</h3><p><strong>Without a skill:</strong></p><blockquote>You:<em> “Write a Playwright test for the login page”</em></blockquote><blockquote>Claude:<em> Generates a test with </em><em>page.locator(&#39;input[type=email]&#39;), inline </em><em>waitForTimeout(2000), no fixture usage, and a hardcoded base URL. Functionally correct. Stylistically wrong for your codebase.</em></blockquote><p><strong>With the Playwright skill:</strong></p><blockquote>You:<em> “Write a Playwright test for the login page”</em></blockquote><blockquote>Claude:<em> Generates a test that extends your custom </em><em>test from </em><em>fixtures/index.ts, uses </em><em>getByLabel(&#39;Email address&#39;) and </em><em>getByRole(&#39;button&#39;, { name: /sign in/i }), follows the Arrange/Act/Assert structure, uses </em><em>waitForLoadState instead of timeout, and adds </em><em>@smoke tag — because that&#39;s what your SKILL.md says.</em></blockquote><p>No re-explaining. No prompt archaeology. Just conventions applied consistently.</p><h3>Why This Matters for Principal Engineering</h3><p>Encoding team conventions in a SKILL.md is a <strong>force-multiplier leadership move</strong>. You write it once — typically as a QA Lead or Principal Engineer establishing standards — and the entire team benefits. New joiners get AI assistance that follows project conventions from day one. Senior engineers stop spending review cycles on style corrections. The AI becomes a consistent member of the team, not a wildcard.</p><p>This is the difference between using AI as a personal productivity tool and using it as a <strong>team infrastructure component</strong>.</p><h3>Token Economics: Does SKILL.md Save Tokens?</h3><p>This is a fair question — and the answer is <em>yes, but not in the way you might expect.</em></p><p><strong>SKILL.md does not reduce what Claude reads.</strong> The content still loads into the context window, so there’s no magic compression happening. But the token story is more nuanced than it first appears.</p><p><strong>Where you actually save tokens:</strong></p><p>The tax you pay today is on your <em>prompt side</em>. Every time you start a new conversation, you’re pasting the same 150–300 words of conventions: “use data-testid, follow POM, extend fixtures, never use waitForTimeout…” That repetition is the real waste — and SKILL.md eliminates it entirely. Your prompts become short and focused.</p><p>The three-level loading system is where the architectural efficiency lives:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5HavB_d0eiQHvMzR2jGBA.png" /></figure><p>That 500-line locator strategy doc in references/? It doesn&#39;t burn context on every message. It only loads when Claude is actively working on a locator problem. For a team running dozens of AI-assisted test sessions daily, that selective loading adds up.</p><p><strong>The honest trade-off:</strong></p><p>If you trigger the skill on every single short message, you’re technically using <em>more</em> tokens than a bare prompt would. The savings materialise when:</p><ul><li>Your manual convention-pasting was longer than the SKILL.md body</li><li>You’re working across many prompts in a session</li><li>Bundled references are being used selectively rather than loaded upfront</li></ul><p><strong>The right mental model:</strong> SKILL.md is not a token-compression tool. It’s a <strong>consistency and scalability tool</strong> that happens to be token-efficient by design. The progressive loading architecture means you’re not paying for what you don’t use — and that’s the real engineering win.</p><h3>Getting Started in 15 Minutes</h3><ol><li>Create .claude/skills/playwright-automation/SKILL.md in your repo root</li><li>Add the YAML frontmatter with name and a detailed description (the description is what triggers the skill — be specific about contexts)</li><li>Write your locator hierarchy, assertion style, and fixture conventions as markdown sections</li><li>Add a references/ folder with your page object template</li><li>Commit it — now every team member with Claude access inherits the same AI behaviour</li></ol><p>Start minimal. A SKILL.md with just locator hierarchy and assertion rules will immediately improve output quality. Add the bundled references as you identify recurring correction patterns in your reviews.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41a8a4d5d9ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>