<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xmartlabs on Medium]]></title>
        <description><![CDATA[Stories by Xmartlabs on Medium]]></description>
        <link>https://medium.com/@xmartlabs?source=rss-35da6de3ebbf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*iAmAp6n3KiDHVM1e.jpg</url>
            <title>Stories by Xmartlabs on Medium</title>
            <link>https://medium.com/@xmartlabs?source=rss-35da6de3ebbf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:24:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xmartlabs/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GraphRAG for Healthcare AI: When Structure Beats Similarity]]></title>
            <link>https://xmartlabs.medium.com/graphrag-for-healthcare-ai-when-structure-beats-similarity-92e909c3b94c?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/92e909c3b94c</guid>
            <category><![CDATA[healthcare-ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[graphrag]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 18:45:35 GMT</pubDate>
            <atom:updated>2026-09-11T18:46:47.204Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7lHSW9EsrkwRztYsf0nOPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*88kpjZcwsjEjiR3U.png" /></figure><p>Most teams building AI products in healthcare eventually face answers that sound right but can’t be traced to any sources. Was the answer hallucinated, or is there substance to it? That could be OK for other systems, such as entertainment or recommendation engines, but it’s not tolerable when the answer touches a diagnosis, a medication, or a treatment plan.</p><p>Retrieval-Augmented Generation helped cut down on hallucinations by grounding answers in retrieved documents, but standard RAG still struggles with a specific kind of question, the kind that requires connecting several facts before it makes sense. Questions such as “What medications treat Alzheimer’s disease but could worsen one of its known complications?” are a chain of reasoning across a diagnosis, a treatment, and a side effect. A plain vector search often misses that chain.</p><p>GraphRAG was built to address this problem. Xmartlabs’ Mathias Claassen (Staff ML Engineer) and Gabriel Lema (Sr ML Engineer) walked through it in a recent webinar on applying GraphRAG to health tech. Since GraphRAG is new ground for the team, this webinar reflects their research and architectural thinking on the approach. What follows organizes what they covered into something useful for anyone deciding whether a graph belongs in their stack.</p><h3>Data quality comes first</h3><p>It’s tempting to treat data cleanup as something to fix later. However, an AI system is only as good as the data underneath it, and if that data is poorly selected or badly structured, no amount of prompt engineering will fix it later.</p><p>Gartner has flagged this in enterprise generative AI adoption: a large share of GenAI projects get abandoned after the proof of concept, and weak data and governance foundations are among the most commonly cited reasons.</p><p>Fixing a shaky data foundation of a running system is expensive. Teams that skip this step tend to redo the same preprocessing work several times, and each run costs more than getting it right the first time. This holds for standard RAG and for GraphRAG. A graph built on inconsistent or duplicated data doesn’t recover just because it’s shaped like a graph.</p><h3>Why traceability matters more in healthcare</h3><p>GraphRAG preserves the relationships between pieces of information instead of treating each document chunk as an isolated unit. Mathias and Gabriel framed the healthcare case around three needs.</p><ul><li>Connected reasoning. Clinical questions often require linking a symptom to a diagnosis, that diagnosis to a treatment, and the treatment to a possible complication, all in one query. A graph makes that chain explicit, so the model doesn’t have to infer it from scattered text.</li><li>Source traceability. Every answer can be grounded in a specific, verified medical concept, which matters when someone needs to check where a claim came from.</li><li>Less reliance on the LLM’s internal knowledge. The less an answer depends on what the model memorized during training, and the more it depends on your own validated data, the lower the risk of a confident but wrong statement.</li></ul><p>In principle, none of this is specific to healthcare. What changes is the cost of being wrong. A hallucinated product recommendation is annoying, while a hallucinated drug interaction can cause real harm, so the traceability bar has to be higher.</p><h3>What is a knowledge graph?</h3><p>A GraphRAG system is built on two primitives: nodes and edges. Nodes are entities or concepts, like Alzheimer’s disease, memory loss, or a specific medication. Edges connect those nodes with a labeled, directed relationship, things like “causes”, “treats”, or “is a symptom of”. A node has a label that identifies its type (Disease, Medication, Patient) and properties that store its attributes. Edges can also have properties, including the supporting text the relationship was extracted from, which makes an answer traceable to its source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ySsrym687Or6QqgX.png" /></figure><p>This structure also affects retrieval speed. A relational database can represent the same information, but pulling out a multi-step relationship usually means a chain of joins that gets slower as the dataset grows. A graph database walks the connection directly. If relationships are central to your queries, that’s a strong argument for this kind of storage.</p><h3>Building the graph without reinventing the wheel</h3><p>The biggest practical risk in building a knowledge graph isn’t the tooling. It’s letting the graph grow without discipline. Mathias was specific about this: start by defining an ontology with domain experts, meaning the agreed set of node labels and the relationships allowed between them. Skip this step, and an LLM extracting relationships from text will eventually invent its own inconsistent structure. The same fact might end up represented twice in opposite directions, or the same disease gets split across two duplicate nodes with its relationships divided between them.</p><p>Two starting points make this easier than building an ontology from nothing.</p><ul><li>FHIR. The standard structure used for medical records already models entities like Patient, Practitioner, Encounter, and MedicationRequest as connected resources. A FHIR bundle is already close to graph-shaped, making it a practical way to pull a graph directly from existing healthcare records.</li><li>UMLS Metathesaurus. For theoretical medical knowledge that isn’t tied to a specific patient, UMLS’s current release defines 127 semantic types (effectively node types) and 54 relationship types. That’s a pre-built vocabulary instead of one invented from scratch, and it groups different names for the same concept under one identifier, which helps prevent the duplication described above.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7M-Bbfn0m2an2sSg.png" /></figure><p>Once the core is in place, teams can extend it with unstructured sources, medical journals, textbooks, even Wikipedia, through an LLM-driven extraction pipeline. Chunk the text, extract candidate triplets and attributes with an LLM, then map those triplets onto the predefined ontology so nothing gets added outside the agreed structure. One approach referenced during the talk, described in the <a href="https://www.sciencedirect.com/science/article/pii/S0933365725002465">MedSumGraph paper</a>, combines UMLS concepts with matched Wikipedia summaries and feeds both to an LLM to extract only the relationships that matter for a given term. The point is deliberately avoiding graphs so large they slow down construction and retrieval alike.</p><h3>Two ways to turn a graph into an answer</h3><p>Gabriel walked through two reasonable architectures rather than picking a single “correct” one.</p><p>The first depends on a well-defined graph and ontology. A natural language question comes in, the system retrieves the relevant schema, and an LLM translates the question plus that schema into a graph query language, commonly Cipher. That query executes against the graph database and returns a subgraph, which goes to an LLM along with the original question for final response generation. Because every piece of that subgraph traces back to a specific node or edge, each statement in the answer can be cited.</p><p>The second skips the strict schema dependency. It extracts candidate triplets from the question, embeds them, and compares those embeddings against pre-embedded triplets already stored in the graph, much like standard RAG’s top-k retrieval. The difference shows up after that. Instead of stopping at the top matches, the system performs an n-hop subgraph expansion from the matched nodes before generating the final answer. It trades some precision for flexibility when the ontology is less rigid or still evolving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9LYki-OU_X9Bneb-.png" /></figure><p>RAG vs. GraphRAG: there isn’t a universal winner</p><p>It would be convenient if GraphRAG were simply “better.” It isn’t. The comparison Xmartlabs walked through is a set of trade-offs across six criteria.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-bFt2_mAWORIWeoU.png" /></figure><p>The honest takeaway is that GraphRAG earns its complexity when relationships and multi-hop reasoning genuinely matter for your queries, not as a default upgrade over RAG.</p><h3>What the benchmarks actually show</h3><p>The webinar also covered a <a href="https://arxiv.org/html/2502.11371v2">study comparing RAG and GraphRAG</a> on four question-answering datasets: NQ, HotpotQA, MultiHop-RAG, and NovelQA. The study adds two more strategies to the comparison. One is a classifier that picks either RAG or GraphRAG for each question. The other is an “integration” approach that combines context from both before generating the response.</p><p>Neither method wins outright. Standard RAG did better on NQ, the simpler benchmark, and GraphRAG did better on MultiHop-RAG, as expected. What did hold across all 4 benchmarks is that the integration strategy beat either method on its own. Going back to the Alzheimer’s question from the intro, that would mean pulling the medication and complication relationships from the graph while a standard document retriever supplies the broader clinical context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iQSMsii4Z3Vj_IKe.png" /></figure><h3>When a graph actually earns its place</h3><p>Not every project needs this. You can use these questions to guide your decision:</p><ol><li><strong>Are relationships between entities central to your queries?</strong> If not, a vector database (unstructured text, semantic search) or a relational database (structured queries, aggregations) is probably simpler and more maintainable.</li><li><strong>If relationships matter, do queries need multi-hop traversal (three or more hops)?</strong> If yes, a graph database is the right call.</li><li><strong>If not, is your schema evolving rapidly?</strong> A graph’s flexible schema handles this better than a rigid relational structure.</li><li><strong>If the schema is stable, do you need explainability or auditability?</strong> If so, a graph’s traceable paths are worth the complexity even without multi-hop needs.</li><li><strong>Otherwise</strong>, simpler joins in a relational database are probably sufficient.</li></ol><p>When your data includes both structured relationships and large volumes of unstructured documents, combining a graph with a vector store, rather than choosing one exclusively, is a legitimate answer. That echoes the benchmark result above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ROmoJKn09bW6rWUx.png" /></figure><h3>Takeaways</h3><ul><li>In healthtech, reliable data sourcing is paramount, and GraphRAG’s added structure can help. Particularly for reasoning on the connection of multiple nodes, GraphRAG shines.</li><li>FHIR and UMLS Metathesaurus are practical starting points for a healthcare ontology rather than designing one from scratch.</li><li>Multiple valid architectures for generating answers from a graph exist, like schema-driven text-to-query and triplet-embedding-and-expansion.</li><li>No retrieval method wins universally. Combining RAG and GraphRAG outperformed either alone across every benchmark referenced in the webinar.</li><li>Your decision to use a graph should follow your query patterns, not whether GraphRAG is newer or sounds more sophisticated.</li></ul><h3>Where this leaves engineering and product teams</h3><p>GraphRAG asks for more upfront modeling work and more careful ontology maintenance. In exchange, it offers something standard RAG structurally cannot: answers that trace back to a specific, verifiable path through your data, and reasoning that spans multiple connected facts instead of stopping at the first similar-looking chunk. For healthcare products, where a wrong or unverifiable answer carries real consequences, that trade is often worth making.</p><p><strong>Want the full technical walkthrough, including the live demo?</strong> You can request the complete GraphRAG for Healthtech webinar recording from Xmartlabs. Reach out to get access, or to talk through whether a knowledge graph makes sense for what you’re building. If you’re earlier in the AI adoption process, our AI Journey series covers how we approached rolling out AI internally across engineering, recruiting, QA, and beyond.</p><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/graphrag-for-healthcare-ai/">https://blog.xmartlabs.com/blog/graphrag-for-healthcare-ai/</a> <em>on Sep 11th, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92e909c3b94c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Compliance Requirements Checkable at Every Deployment]]></title>
            <link>https://xmartlabs.medium.com/making-compliance-requirements-checkable-at-every-deployment-e3a0dcf007d3?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3a0dcf007d3</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 18:09:16 GMT</pubDate>
            <atom:updated>2026-08-07T18:09:16.607Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7HGyOkXNF-qvWE11xEtFdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xd2vengEBdpWgi_s.png" /></figure><h3>Introduction</h3><p>If you’re building software that handles patient data or offers a SaaS product to enterprise clients, you’re likely dealing with HIPAA, SOC 2, or any other regulation.</p><p>The problem is that regulations don’t specify what compliance looks like in code. They describe objectives. Translating those objectives to your infrastructure, evaluating them, and keeping pace with your evolving codebase is left entirely to you.</p><p>We built GRC XL to close that gap: a compliance knowledge framework that translates regulatory requirements into structured, checkable controls tied to specific AWS configurations, with a pipeline that automatically evaluates your Terraform infrastructure against those controls. It starts with HIPAA and SOC 2, but the model is extensible for any regulation, cloud, and programming language.</p><p>If your team ships software that needs to stay compliant with every deployment, not just at audit time, this can help you build that assurance into your development process and stay on top of exactly what gets evaluated and how. In this post, we walk through how the framework is structured, how the pipeline works, and what it surfaced when we ran it against a real open source healthtech codebase.</p><h3>From Regulation to Checkable Control</h3><p>To better understand how to perform that technical translation and how to conduct automated evaluations, we break down regulation into categories, each category into domains, and each domain into controls. This is what each of them is:</p><ul><li>Framework: the regulatory standard the organization is measured against: SOC 2, HIPAA, but extendable.</li><li>Category: a broad area of concern within that framework. For instance, SOC 2 groups its requirements into Security, Privacy, Process Integrity, Availability, and Confidentiality. We try to stick to them when we consider that it’s applicable to code review.</li><li>Domain: a specific subject inside a category: access control, data handling, network security, for instance. The domain is where an abstract requirement becomes a concrete topic an engineer or agent can reason about.</li><li>Control: a single, checkable requirement inside a domain. It states what must be true, why, and how to demonstrate it.</li></ul><p>One real path through the tree:</p><p>SOC 2 &gt; Security &gt; Access Control &gt; CTRL-S-AC-01: IAM Role-Based Access Architecture:</p><p>“IAM role-based access architecture is used for all AWS account access; long-lived IAM user credentials are not used for production workloads”</p><p>But how do we evaluate the controls? Each domain has a runbook: a document that lists all the controls for that domain, provides detailed instructions on how to evaluate them, and documents when to classify them as “fail,” “pass,” or “needs more evidence.”</p><p>An example of a SOC 2 / Privacy / Data Handling control is included at the end of this article.</p><h3>From Controls to Findings: How the Pipeline Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vQmR36hm4LfiFjm4.png" /></figure><p>The pipeline has one job: check the controls defined in the framework against real infrastructure.</p><p>For each control, the system collects the Terraform files that count as evidence: IAM policies, S3 configurations, KMS settings, and any other resources identified by the control’s evaluation logic. That evidence is paired with the control definition and evaluated through a structured, rules-based assessment workflow. The result is a control verdict, supporting reasoning, and references to the exact resources examined.</p><p>The following are the three possible verdicts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IlBUMOsVGZ1Lmo8N.png" /></figure><p>Because the assessment workflow reads static configuration, it can run anywhere the code lives, including on every pull request.</p><h3>Running GRC XL on Open Source Healthtech Infrastructure</h3><p>We ran GRC XL against the public Terraform repository of <a href="https://github.com/the-momentum/healthstack">HealthStack</a> from Momentum, a healthcare software and development company.</p><p>The pipeline returned findings on two transmission security controls: CTRL-H-DH-01 (ePHI Encryption in Transit) and CTRL-H-DH-05 (Minimum TLS Version Policy Enforcement). The reasoning in each finding pointed to the same thing: S3 buckets across three modules had no explicit bucket policy denying unencrypted transport or connections below TLS 1.2.</p><p>AWS’s default allows both HTTP and HTTPS unless a deny policy is present. Without it, plaintext access is not explicitly blocked, even in infrastructure designed for healthcare data.</p><p>The change is two deny statements per bucket:</p><pre># HIPAA §164.312(e) - Transmission Security<br>{<br>  Sid       = &quot;DenyInsecureTransport&quot;<br>  Effect    = &quot;Deny&quot;<br>  Principal = &quot;*&quot;<br>  Action    = &quot;s3:*&quot;<br>  Resource  = [&quot;${aws_s3_bucket.data.arn}&quot;, &quot;${aws_s3_bucket.data.arn}/*&quot;]<br>  Condition = {<br>    Bool = { &quot;aws:SecureTransport&quot; = &quot;false&quot; }<br>  }<br>},<br>{<br>  Sid       = &quot;DenyOutdatedTLS&quot;<br>  Effect    = &quot;Deny&quot;<br>  Principal = &quot;*&quot;<br>  Action    = &quot;s3:*&quot;<br>  Resource  = [&quot;${aws_s3_bucket.data.arn}&quot;, &quot;${aws_s3_bucket.data.arn}/*&quot;]<br>  Condition = {<br>    NumericLessThan = { &quot;s3:TlsVersion&quot; = &quot;1.2&quot; }<br>  }<br>}</pre><p>The Momentum evaluation ran in minutes, surfaced concrete findings, and the PR was merged.</p><h3>What This Means for Your Team</h3><p>GRC XL keeps compliance in the codebase. As your infrastructure changes, the pipeline re-evaluates it automatically. A change that affects a control gets flagged before it merges.</p><p>GRC XL is built and maintained by Xmartlabs. If you want to evaluate your infrastructure against HIPAA or SOC 2 controls, identify where it meets the requirements and where it doesn’t, and integrate compliance checks into your CI workflow, we’d like to help. <a href="https://xmartlabs.com/contact">Talk to the Xmartlabs team</a></p><p>Are you handling compliance as part of your development cycle, or is it something that gets addressed when an audit comes up? We’d like to know</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lDxS0Vhyi6ZZwEToIveKXg.png" /></figure><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/checkable-compliance-deployments/">https://blog.xmartlabs.com/blog/checkable-compliance-deployments/</a> <em>on Aug 7th, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3a0dcf007d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Isn’t Just for Developers:]]></title>
            <link>https://xmartlabs.medium.com/ai-isnt-just-for-developers-9ac104d60095?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ac104d60095</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[claude-code]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 16:19:30 GMT</pubDate>
            <atom:updated>2026-07-23T16:19:30.334Z</atom:updated>
            <content:encoded><![CDATA[<h4>How We’re Bringing AI Beyond Engineering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_LG_5f9F4rLyTMFNePxyQ.png" /></figure><p>Most companies limit AI tools to their engineering teams. We decided to push them to project managers, HR, delivery, and pre-sales. The PM rollout is the furthest along, with five workflows built and in active use. The other roles are still early experiments. Here’s the strategic reasoning, the concrete playbook we used, and an honest account of where we are today.</p><h3>The Strategic Decision</h3><p>After building <a href="https://blog.xmartlabs.com/blog/designing-ia-level-up-ai-workshop-team-adoption/">IA Level Up</a> and measuring the <a href="https://blog.xmartlabs.com/blog/feature-utilization-gap-ai-tool-adoption/">feature utilization gap</a> across our engineering team, a question kept surfacing in our AI Engineering group meetings: if we only optimize developers, what happens to the rest of the pipeline?</p><p>A feature doesn’t start with code. It starts with a PRD. It gets planned in a sprint. It gets communicated to stakeholders. It gets estimated in pre-sales. It gets staffed in delivery. Every one of those steps involves a person doing structured, repetitive work, the exact kind of work that AI tools handle well.</p><p>The logic was straightforward. The same agent that writes functions from a spec can write tickets from a PRD. The same MCP that connects to a GitHub repository can connect to Linear, Notion, or Google Calendar. The same skill that automates a test suite can automate a weekly status report.</p><p>We weren’t trying to make everyone a developer. We were recognizing that the underlying capabilities- reading documents, extracting structured information, connecting to APIs, following multi-step workflows- are role-agnostic. The tool is a developer tool by default. The capabilities are not.</p><p>The risk was obvious: non-technical roles don’t live in the terminal. The onboarding would be different. The resistance would be different. The way you measure value would be different. We decided to run a proof of concept with project managers first, then expand based on what we learned.</p><h3>The Playbook</h3><p>Before targeting a role, we needed a framework that would keep us from guessing. We settled on four evaluation criteria:</p><p><strong>1. Task repetitiveness.</strong> How much of the role involves doing the same structured task weekly or daily? PMs write reports every Friday. Delivery tracks hours and utilization. HR processes candidate pipelines. High repetitiveness means high automation potential.</p><p><strong>2. Tool accessibility.</strong> Does the role already use tools that have MCP integrations available? If a PM lives in Linear, Notion, and Google Calendar, and MCPs exist for all three, the integration surface is ready. If a role depends on a proprietary internal tool with no API, the barrier is higher.</p><p><strong>3. Time savings potential.</strong> What’s the realistic time reclaimed per week if the top 3–5 repetitive tasks are automated? We estimated this before building anything, using simple time logs from the people in each role.</p><p><strong>4. Willingness to experiment.</strong> This one matters more than you’d think. A role with high automation potential but low interest will stall. We started with PMs because several of them had already expressed curiosity about what the engineering team was doing with Claude Code.</p><p>For each role, we followed the same implementation sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vlZnVAXyQo7QGn3G.png" /></figure><ol><li><strong>Shadow:</strong> Observe the person doing their actual work for a week. Document every task, tool used, and time spent.</li><li><strong>Map:</strong> Identify which tasks are candidates for automation. Match them to available MCPs and skills.</li><li><strong>Build:</strong> Create the MCP configurations and custom skills needed for the top 3 workflows.</li><li><strong>Demo:</strong> Run a live session showing the workflows end to end. Let the person drive.</li><li><strong>Iterate:</strong> Adjust based on feedback. Some workflows need more guardrails. Some need fewer steps. Some don’t survive contact with reality.</li><li><strong>Measure:</strong> Track actual time savings after two weeks. Compare against the pre-automation baseline.</li></ol><p>This is the same playbook we’re applying to every role we expand into. The specifics change. The structure doesn’t.</p><h3>PMs: The Proof of Concept</h3><p>This is where the abstract framework becomes concrete. We built a complete demo environment (as of March 2026) with five workflows running on real tools: Linear, Notion, Google Calendar, Slack, and Figma, connected via MCPs and orchestrated through Claude Code.</p><p>We ran these as live demos for our PM team. Every workflow described below was executed in front of an audience, not in a controlled lab. Here’s what we built and what it looks like in practice.</p><h3>Workflow 1: PRD to Linear Project</h3><p><strong>The task:</strong> Take a product requirements document and create a complete project in Linear with epics, user stories, estimates, and priorities.</p><p><strong>Before:</strong> A PM reads the PRD, manually creates each epic in Linear, writes each issue’s description by translating user stories into ticket format, assigns estimates, and sets priorities. For a mid-size feature set, say, 5 epics and 18 issues, this takes 30 to 45 minutes of focused work.</p><p><strong>After:</strong> The PM opens Claude Code, points it at the PRD file, and gives one instruction:</p><p><em>“Read the PRD. Create a new project in Linear called [project name]. Create the epics as labels. Create all issues with title, description based on the user stories, estimation in points, and priority. Assign each issue to its corresponding epic.”</em></p><p>Claude reads the document, understands the structure (epics, user stories, acceptance criteria, priorities), and calls the Linear MCP to create everything. One minute. The PM then reviews in Linear, adjusting anything that needs a human judgment call.</p><p>We tested this with a fictitious delivery app PRD containing 5 epics and 18 user stories with acceptance criteria. The output in Linear matched the PRD structure with accurate descriptions, correct priority assignments, and reasonable point estimates. The PM’s role shifted from data entry to review.</p><h3>Workflow 2: Meeting Transcript to Backlog Updates</h3><p><strong>The task:</strong> After a client meeting, extract new feature requests, priority changes, and scope modifications, then update Linear accordingly.</p><p><strong>Before:</strong> The PM listens to the recording or reads the transcript, takes notes on what changed, goes to Linear, finds the relevant issues, creates new ones, updates priorities on existing ones, and cancels anything removed from scope. For a meeting with three new feature requests, two priority changes, and one scope removal, this takes 30 to 40 minutes.</p><p><strong>After:</strong> We built a custom skill, a Markdown file with natural language instructions, that processes meeting transcripts. The PM runs the skill with the transcript file as input. Claude reads the transcript, identifies changes (new features, priority shifts, scope removals), connects to Linear to check what already exists, and produces a structured proposal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5J2BPjma7l7G7cUd.png" /></figure><p>The critical design decision: Claude proposes but does not execute. It presents the table and waits for confirmation. The PM reviews, adjusts if needed, then approves. Only then does Claude apply the changes to Linear.</p><p>This “propose, then execute” pattern is non-negotiable for PM workflows. Unlike code generation where you can review a diff, backlog changes affect project scope and client expectations. The human stays in the decision loop.</p><h3>Workflow 3: Automated Status Reports to Notion</h3><p><strong>The task:</strong> Generate a weekly status report from project data in Linear and publish it to Notion.</p><p><strong>Before</strong>: Every Friday, the PM opens Linear, reviews each epic’s progress, calculates completion percentages, identifies blockers, writes a summary, formats it, and posts it to Notion. Depending on project size, this takes 20 to 60 minutes.</p><p><strong>After:</strong> We created a custom skill called “Weekly Report.” It’s a Markdown file that tells Claude to connect to Linear, pull the project data, calculate progress by epic, identify blockers, generate a formatted report, and publish it to Notion. The PM types /weekly-report and reviews the output.</p><p>The skill is written in plain language. No code. Any PM could write one by describing the report format they want. Here’s what the skill does in sequence:</p><ol><li>Reads all issues from the active project in Linear</li><li>Groups by epic, calculates percentage completed vs. in-progress vs. not-started</li><li>Identifies blocked issues and extracts blocker descriptions</li><li>Generates an executive summary, progress-by-epic breakdown, blockers section, and next steps</li><li>Publishes to the designated Notion page</li></ol><p>The result is a professional report that would take a human 30+ minutes to assemble. The PM’s job becomes editing the summary and adding qualitative context that Linear data can’t capture: client sentiment, strategic decisions, risk assessments.</p><p>Worth noting: this workflow is built on Linear and Notion, but the pattern applies to any project management and reporting stack. Jira, Asana, Monday, ClickUp. if there’s an MCP or API, the same skill structure works. The skill describes what to do, not which tool to do it with. Swapping the data source is a configuration change, not a rewrite.</p><h3>Workflow 4: Calendar Analysis</h3><p><strong>The task:</strong> Analyze the day’s schedule, identify conflicts, calculate meeting load, and suggest optimization.</p><p><strong>Before:</strong> PMs open their calendar, mentally scan for conflicts, estimate how much focus time they have left, and plan their day. This isn’t a long task, maybe 5 to 10 minutes, but it happens every morning, and the analysis is always shallow.</p><p><strong>After:</strong> Claude reads the Google Calendar MCP, pulls the day’s events, and returns:</p><ul><li>Total meetings vs. available time</li><li>Any scheduling conflicts</li><li>Suggested reorganization for focus blocks</li><li>Recommended time slots for specific tasks based on meeting gaps</li></ul><p>This is the lightest workflow of the five, less about saving time, more about getting analysis that people skip because it feels trivial. A PM who sees “You have 6 hours of meetings and 2 hours of gaps, split across 4 fragments of 30 minutes” makes different decisions than one who glances at a crowded calendar and starts reacting.</p><h3>Workflow 5: PRD to Figma Mockups</h3><p><strong>The task:</strong> Generate initial UI mockups from product requirements, published directly to Figma.</p><p><strong>Before:</strong> The PM writes the PRD, hands it to a designer, waits for first-pass wireframes, gives feedback, iterates. First visual output: 2–5 days.</p><p><strong>After:</strong> Claude reads the PRD, generates HTML/CSS for the key screens (mobile-first), spins up a local server, and uses the Figma MCP to capture and publish the screens. The output is not production design; it’s a conversation starter. A PM can walk into a design kickoff with visual references instead of text descriptions.</p><p>We tested this by generating four screens for a delivery app: restaurant catalog, restaurant detail with menu, cart and checkout, and real-time order tracking. The mockups used a modern glassmorphism style. They’re rough. They’re also dramatically better than nothing, which is what PMs typically bring to design conversations.</p><p>This workflow has a clear limitation: it produces static screens, not interactive prototypes. It doesn’t replace a designer. It gives PMs a way to communicate visual intent faster.</p><h3>The MCP Stack</h3><p>All five workflows depend on this MCP configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JU-4iKTmmfKinpVF.png" /></figure><p>Each MCP is configured once in a JSON settings file. After that, Claude can read and write to the connected tool. The technical setup takes an engineer 15–30 minutes per MCP. After configuration, the PM never touches the setup again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1K8SqVJnyP4bk3M9.png" /></figure><h3>Two Custom Skills</h3><p>Beyond the MCP connections, we built two reusable skills:</p><p><strong>Weekly Report skill:</strong> A markdown file describing the report structure, data sources, and output format. The PM invokes it with /weekly-report. It’s written in natural language; any PM could modify the template to match their team’s reporting format.</p><p><strong>Process Transcript skill:</strong> A markdown file that describes how to read a meeting transcript, extract changes, compare against the existing backlog, and propose updates. Invoked with /process-transcript [file].</p><p>These skills demonstrate something important: non-developers can create automation without code. A skill is a markdown file with instructions. If you can describe what you want in writing, you can build a skill.</p><h3>HR, Delivery, Pre-Sales: The Frontier</h3><p>We need to be direct about this section. Unlike the PM workflows, which are built, demoed, and running, the expansion to other roles is early-stage. We’re sharing what we’re exploring, not what we’ve shipped.</p><p><strong>HR.</strong> The team has started using AI tools for structured, repetitive tasks, processing evaluation data, generating summaries from performance reviews, and drafting communications. These aren’t full automated workflows yet; they’re individual team members using Claude Code to speed up tasks they were already doing manually. The pattern is the same one we saw in engineering: organic adoption of basic capabilities, with deeper workflow automation as the next step.</p><p><strong>Delivery.</strong> Delivery managers have adopted AI tools for sprint evaluation processing, client meeting note processing, and billing workflows. Repetitive reporting, pulling data, calculating utilization rates, and generating status updates are where the time savings are clearest. The integration surface is more fragmented than PM tools (no single MCP covers the typical delivery stack), but skills and automation scripts bridge the gap for now.</p><p><strong>Pre-sales.</strong> The team uses AI tools for data processing, proposal drafting, and estimation support. Estimation and proposal generation involve reading RFPs, estimating effort, assembling team profiles, and drafting proposals; much of this is template-based with project-specific customization, which makes it a natural fit. The open question remains whether AI-generated estimates are accurate enough to meaningfully reduce review overhead, or whether the human review step stays just as long.</p><p>What we can say with confidence: the PM proof of concept validated the approach. The playbook (shadow, map, build, demo, iterate, measure) works for non-engineering roles. The MCP ecosystem is broad enough to cover most common business tools. The barrier isn’t technology; it’s the investment of time to shadow each role, understand its workflows, and build the custom skills.</p><p>We’ll share specifics on HR, delivery, and pre-sales as those experiments mature. For now, these roles represent our active frontier, not our completed work.</p><h3>What We Learned</h3><p>Six months into cross-company AI expansion (as of April 2026), several patterns have emerged that we didn’t anticipate:</p><p>Resistance looks different outside engineering. Engineers worry about AI code quality: “Will it break my production system?” Non-technical roles worry about job security: “Is this going to replace me?” The fear is more personal and more emotional. Addressing it requires a different conversation. We found that framing AI as “handles the repetitive parts so you can focus on judgment calls” lands better than “makes you more productive.” Same truth, different emphasis.</p><p>The terminal is a real barrier. Claude Code runs in the terminal. Engineers don’t think twice about this. For everyone else, the terminal is foreign. The “two commands to install” pitch that works for developers falls flat with PMs. We needed an engineer to handle initial setup and MCP configuration. We also needed to show that the actual interaction is natural language; you type what you want in plain words. Once people see that, the terminal stops being scary. But that first impression matters.</p><p>Skills are the unlock for non-developers. MCPs connect tools. But skills are what make the connection useful. A PM doesn’t want to think about which MCP to call or how to structure a request. They want to type /weekly-report and get their report. Custom skills turn a general-purpose agent into a role-specific assistant. This is where the real value accrues for non-engineering roles. And here’s the part we didn’t expect: PMs are now building their own skills library. They started by using the skills we built for them, then began writing new ones for workflows we hadn’t anticipated. The tool went from “something engineering set up for us” to “something we own and extend.” That’s the adoption inflection point, when the role stops consuming and starts creating.</p><p>Value measurement changes completely. For developers, we measure feature utilization rates, code quality metrics, and development time. For PMs, the metric is time reclaimed from administrative work. For HR, it would be screening throughput. For delivery, it would be reporting accuracy and speed. Each role needs its own success metrics, defined before you start building. If you use engineering metrics to evaluate a PM workflow, you’ll conclude it doesn’t work even when it does.</p><p>The “propose, don’t execute” pattern is essential. In our engineering training (<a href="https://blog.xmartlabs.com/blog/the-mathi-problem-ai-generated-code/">The Mathi Problem</a> taught us this the hard way), we learned that AI needs verification loops. For non-engineering roles, the stakes are different, but the principle is the same. A status report published to Notion with wrong numbers is a trust problem. A backlog update that deletes the wrong issues is a scope problem. Every PM workflow includes a confirmation step. Claude shows what it plans to do and waits for approval.</p><p>Adoption spreads through demonstration, not documentation. We wrote guides. Nobody read them. We ran a live demo where a PM saw her actual weekly report generated in 30 seconds. She started using it the next day. For non-technical roles, seeing is adopting. The demo project we built for PMs wasn’t just a training tool; it was the adoption mechanism.</p><h3>How to Start in Your Company</h3><p>If you’re considering expanding AI tools beyond engineering, here’s the framework we’d recommend based on what worked for us:</p><ul><li><strong>Step 1:</strong> Pick one non-engineering role. Choose the role with the highest combination of task repetitiveness + tool accessibility + willingness to experiment. For most companies, this is PMs or technical writers.</li><li><strong>Step 2:</strong> Shadow before building. Spend a week observing the actual workflows. Don’t assume you know what a PM does all day. Document every task, tool, and time spent. The insights will surprise you.</li><li><strong>Step 3:</strong> Start with three workflows. Don’t try to automate everything. Pick the three most repetitive tasks that use tools with MCP integrations. Build those. Demo those. Get feedback on those.</li><li><strong>Step 4:</strong> Build custom skills, not raw prompts. Non-technical users shouldn’t need to craft prompts. Build skills (markdown templates with instructions) that they invoke with a single command. The skill abstracts the complexity.</li><li><strong>Step 5:</strong> Measure time savings, not “AI usage.” Just like tool adoption doesn’t equal feature utilization, “using AI” doesn’t mean “getting value from AI.” Measure hours reclaimed per week. That’s the number that justifies expanding to the next role.</li><li><strong>Step 6:</strong> Iterate before expanding. Don’t move to the next role until the current one is stable. We spent three months on the PM proof of concept before even discussing HR or delivery. Rushing to breadth kills depth.</li></ul><p>The framework described above- evaluation criteria, the shadow-map-build-demo-iterate-measure sequence, and role-specific success metrics- is what we use for every new role expansion.</p><h3>What’s Next</h3><p>We’re still in the early chapters of this expansion. The PM workflows are the most advanced, but still evolving; PMs are building their own skills and finding new use cases we didn’t anticipate. HR, delivery, and pre-sales are earlier in the process. The playbook exists but is still being validated across roles.</p><p>What we’re confident about: the same tools that transformed how our engineers work have clear applications beyond engineering. The barrier isn’t the technology; it’s the investment in understanding each role deeply enough to build workflows that fit.</p><p>What we’re not confident about: how far this extends. Some roles may not benefit. Some tools may not have adequate MCP integrations. Some workflows may be too nuanced for current AI capabilities.</p><p>We’ll keep building, measuring, and publishing what we learn. The philosophy that runs through everything our AI Engineering group does applies here too: “Try and change. Don’t expect a setup that works for years.” The tools evolve. The roles evolve. Our approach has to evolve with them.</p><p>For the full story of how we got here, from the <a href="https://blog.xmartlabs.com/blog/feature-utilization-gap-ai-tool-adoption/">85% adoption / 0% Skills discovery</a> to building a <a href="https://blog.xmartlabs.com/blog/designing-ia-level-up-ai-workshop-team-adoption/">structured training program</a> to developing a <a href="https://blog.xmartlabs.com/blog/the-mathi-problem-ai-generated-code/">prevention-first approach to AI code quality</a>, start at the hub.</p><h3>Further Reading</h3><ul><li>The Feature Utilization Gap: What 85% AI Tool Adoption Really Looks Like, The data that led to building this workshop</li><li>The Mathi Problem: Why AI-Generated Features Ship Fast and Break Slow, The full story behind Module 6’s core anecdote</li><li>AI Tools Are Not Just for Developers: How We’re Expanding Beyond Engineering, what happened when we applied similar thinking to PMs, HR, and delivery</li><li>Anthropic Skilljar: Claude Code in Action, an external training resource we recommend as a supplement</li><li>Anthropic Prompt Engineering Guide, Foundation for the Prompt Formula, taught in Module 1</li></ul><p><em>This article is part of our AI journey series. Published April 2026. The tools, features, and recommendations described here reflect the landscape as of that date. AI tooling moves fast; verify current capabilities before building on our specific tool recommendations.</em></p><p><em>(Originally published on the Xmartlabs blog at this link:m</em><a href="https://blog.xmartlabs.com/blog/ai-tools-beyond-engineering/">https://blog.xmartlabs.com/blog/ai-tools-beyond-engineering/</a> <em>on July 22nd, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ac104d60095" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate, Yes. Dehumanize, No.]]></title>
            <link>https://xmartlabs.medium.com/automate-yes-dehumanize-no-a8360ed4a039?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8360ed4a039</guid>
            <category><![CDATA[ai-in-recruitment]]></category>
            <category><![CDATA[resume-writing-tips]]></category>
            <category><![CDATA[candidate-experience]]></category>
            <category><![CDATA[hr]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 19:56:15 GMT</pubDate>
            <atom:updated>2026-07-13T19:56:15.448Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dhSDpTA5OHYIpNRuPZIQg.png" /></figure><p>Lately, you’ve probably noticed that across every media outlet, we hear and see more and more about how artificial intelligence is finding its place, or, in some cases, almost entirely taking over, in recruitment processes. From analyzing resumes, responding to applications, and drafting emails, to reaching a first interview where there’s no longer a person on the other side, but an AI. (No, we’re not living inside a futuristic movie; this is today’s job market reality.)</p><p>As someone who has worked in recruitment for more than nine years, I found this unthinkable when I started. And, as happens with almost every field, technology will keep advancing, and we have to adapt to it and make the most of it. But I think the real question isn’t whether we should use AI or not, that’s a given if we don’t want to fall behind and we need to build new skills, the real question is: how far? Which tasks make sense to automate, and which do we still need a person for? Because to me, soft skills are central, and for now, thankfully, those remain uniquely human.</p><p>I consider myself a curious, proactive person when it comes to trying new tools. I enjoy automating processes, finding ways to save time, and eliminating repetitive (and sometimes tedious) daily tasks, which feels valuable too. That’s why I think the discussion shouldn’t be about whether to use AI, but about deciding when it actually adds value.</p><p>This might sound like it’s from the dinosaur era, but it wasn’t that long ago (depending on which generation you belong to). When I started as a recruiter, I called candidates from a landline (yes, almost always to their homes), and all the information lived on paper index cards, handwritten, kept in drawers, and sorted alphabetically. As I mentioned, the way we work has changed enormously in recent years and will surely continue to evolve. As part of today’s evolution, it’s now possible to reach a candidate in a much more practical, agile, and personal way.</p><p>In this article, I’ll focus on my area of expertise: HR. Every company is looking to optimize processes and work more efficiently, and I fully agree. In fact, on our team at Xmartlabs, we also push for that kind of improvement. If an AI can draft a follow-up email using information we’ve already logged and send it with one click, all the better. For example, there are automations I use and value, such as reminders to keep candidates informed and automatic interview transcriptions. We used to spend a large part of our time taking notes; now we can focus much more on listening, asking better questions, and paying attention to the conversation.</p><p>Even resume screening, often seen as the great enemy of job applicants, can be a useful tool when used correctly. It can help prioritize profiles and prevent good candidates from being missed simply because no one got around to reviewing every resume.</p><p>Here are a few valuable tips for building a resume that survives this new process:</p><ul><li><strong>Use a simple format.</strong> Avoid columns, tables, charts, or text embedded in images, since many ATS platforms can’t read them correctly.</li><li><strong>Save it as a PDF</strong> and make sure the text is selectable.</li><li><strong>Include keywords</strong> from the job description and repeat them naturally in your title, profile, experience, and skills sections.</li><li><strong>Describe your experience.</strong> Don’t just list your job title; explain what you did, what tools you used, and what you achieved.</li><li><strong>Add a skills section</strong> with the technologies, tools, and competencies relevant to the role.</li><li><strong>Tailor your resume to each search. </strong>The more aligned it is with the job description, the better your odds of getting past the ATS filter.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z5RHqiqV7sScL3BVjK24Xw.png" /></figure><p>But just because something can be automated doesn’t mean it should be. I still believe that, at least for now, other parts of the process genuinely need a person behind them.</p><p>Not just to get a better sense of cultural fit or to catch things that are hard to detect automatically, but because an interview also means interpreting context. Noticing how someone communicates, understanding when they’re nervous without judging them, and giving them a few minutes of informal conversation to help them feel more at ease before getting started. Some things never show up on a resume or in a transcript. Things that are only perceptible through the humanity that defines us, empathy, above all.</p><p>And it’s just as important to think about the candidate’s experience.</p><p>Let’s not forget that during a hiring process, it’s not only the company evaluating the candidate. The reverse is also true: candidates are evaluating the company. They want to know how the team works, who’s part of it, and what the people they might work alongside every day are like. Often, the interview is their very first impression of the organization. After all, as the name suggests, we’re the human resources of that organization. If not us, then who?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7-uEVg4OrTG12c1S7sa2A.png" /></figure><p>Even though I work “on the other side” in recruitment, I’ll admit that if I were job hunting today and my first interview were with an AI, I probably wouldn’t feel entirely comfortable, speaking personally. That’s my opinion; some companies use this approach, and it works very well for them. But for me, an interview is also a chance to get to know the company, ask questions, and start building a relationship with that workplace. Showing culture and seeing that cultural match with the candidate matter too.</p><p>So, conclusion? I believe artificial intelligence is here to stay and has huge potential to improve how we work. The challenge is to use it thoughtfully, leaning on it to eliminate repetitive tasks and free up time, without losing the human part of the process.</p><p>Even the time we save should go toward improving our work: preparing better for interviews, giving more personalized feedback, understanding each candidate more deeply, and having conversations that actually add value. If AI saves us time, I’d like that time to go back to people. Because at the end of the day, we’re still working with them.</p><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/automate-yes-dehumanize-no/">https://blog.xmartlabs.com/blog/automate-yes-dehumanize-no/</a> <em>on July 13th, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8360ed4a039" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Shipped a Feature in Half a Day with AI. Then Spent Three Days Fixing It.]]></title>
            <link>https://xmartlabs.medium.com/we-shipped-a-feature-in-half-a-day-with-ai-then-spent-three-days-fixing-it-9c761e3d1eae?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c761e3d1eae</guid>
            <category><![CDATA[code-quality]]></category>
            <category><![CDATA[ai-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 13:48:52 GMT</pubDate>
            <atom:updated>2026-06-29T13:48:52.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1ZlKAOFma66HXJcvFv6Rw.png" /></figure><p>Part of the <a href="https://blog.xmartlabs.com/blog/feature-utilization-gap-ai-tool-adoption/">Xmartlabs AI Journey series</a>. Previously: <a href="https://blog.xmartlabs.com/blog/designing-ia-level-up-ai-workshop-team-adoption/">Designing IA Level Up</a>.</p><h3>TL;DR</h3><p>One of our team members shipped a feature in half a day using AI-assisted development. Then, I spent three days fixing the bugs it introduced. This is not an outlier; it is the most common failure mode of AI-assisted development. The good news is that it is entirely preventable. This post breaks down what happened, why the industry data is alarming, and the concrete prevention framework we now apply to every AI-assisted task.</p><h3>The Incident</h3><h3>Summary</h3><p>A developer on our team (Mathi) used AI tooling to implement a feature. The AI-generated working code fast. By midday, the feature was done: implemented, apparently functional, ready for review. What followed was three days of debugging, subtle bugs, edge cases the AI had not considered, and logic errors that only surfaced under real conditions.</p><h3>Timeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MrzPHh1_9SdR_Vwk.png" /></figure><h3>What happened</h3><p>The AI did what it was asked to do. It produced code that compiled, ran, and appeared to work for the primary use case. But it lacked understanding of the broader system context: edge cases, existing patterns, and implicit requirements that an experienced developer would have internalized.</p><p>This is not a story about blame. Mathi did nothing wrong; this is what happens when a capable tool meets a workflow that was not designed for it. The AI wrote code the way a confident but inexperienced engineer would: quickly, plausibly, and without the judgment that comes from maintaining a codebase over months.</p><p>The ratio tells the story: half a day building, three days fixing. A 6:1 ratio of debugging to development. The feature was not faster. It was faster to appear done.</p><h3>Why This Keeps Happening</h3><p>Our experience is not unique. The industry data on AI-generated code quality is sobering:</p><ul><li>~45% of AI-generated code contains security vulnerabilities. Veracode’s 2025 GenAI Code Security Report tested 100+ LLMs across 80 code completion tasks and found that AI models chose the insecure coding path <a href="https://www.securitymagazine.com/articles/101801-ai-introduces-security-vulnerabilities-within-code-in-45-of-cases">45% of the time</a>.</li><li>1.7x more defects in AI-generated code compared to human-written code. CodeRabbit’s analysis of <a href="https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report">470 open-source PRs</a> found that AI-authored code averaged 10.83 issues per PR, compared with 6.45 for human-only code.</li><li>Pull requests are 18% larger when developers use AI coding assistants, according to a <a href="https://jellyfish.co/blog/ai-assisted-pull-requests-are-18-larger/">Jellyfish study</a> tracking millions of PRs across 500+ companies.</li><li>Delivery stability decreases measurably with AI adoption. Google’s <a href="https://dora.dev/research/2024/dora-report/">DORA 2024 report</a> found that a 25% increase in AI adoption was associated with a 7.2% decrease in delivery stability, and its 2025 follow-up confirmed the trend hasn’t reversed.</li></ul><p>These numbers paint a grim picture. If you stopped reading here, you would conclude that AI coding assistants make things worse, not better.</p><h3>The “But” That Changes Everything</h3><p>Here is the critical detail that most coverage of these statistics omits: these numbers come from teams using AI tools without proper configuration.</p><p>No project-level instructions. No verification workflows. No constraints on what the AI should or should not do. No structured review process for AI-generated output.</p><p>In other words, these teams gave an eager but inexperienced assistant the keys to the codebase and said, “Go.” No onboarding, no guardrails, no code review process adapted for AI output.</p><p>That is the environment that produces a 1.7x defect rate. That is the environment where incidents spike. And that was, honestly, the environment we were operating in when the Mathi Problem happened.</p><p>The question is not “should we use AI for coding?”, that ship has sailed. The question is: how do we configure the environment so AI-generated code meets the same standards as human-written code?</p><h3>The Root Cause</h3><p>During our <a href="https://app.notion.com/p/AI-light-training-3-4-Hour-Workshop-31ffbac623cd80eb8f66d006a734cc7a?pvs=21">IA Level Up training program</a>, we landed on a mental model that reframed everything:</p><p>Think of AI as a junior developer, fast, capable, and without judgment.</p><p>This is not dismissive. Junior developers are valuable. They learn fast, produce volume, and, given the right mentorship and review process, ship good work. But nobody gives a junior developer unsupervised access to production on their first week.</p><p>The problem with the Mathi Problem was not that AI wrote bad code. The problem was that the AI operated without:</p><ul><li>Project context: It did not know our conventions, architectural patterns, or existing abstractions.</li><li>Constraints: It was not told what not to do. It did not know our testing requirements, our security considerations, or our deployment patterns.</li><li>A review framework: The code review process was optimized for human-authored code, not for the specific failure modes of AI-generated code.</li></ul><p>The root cause was not AI. It was the absence of a verification and prevention framework adapted to how AI writes code.</p><h3>Prevention &gt; Verification</h3><p>After the Mathi Problem, we developed a principle that now drives how we work with AI: prevention is more valuable than verification.</p><p>Verification means catching bugs after they are written. Prevention means configuring the environment so the AI writes fewer bugs in the first place. Both matter, but prevention has a dramatically better return on time invested.</p><p>Here is the concrete framework we use, with real-time investments:</p><h3>1. <a href="http://claude.md/">CLAUDE.md</a> / <a href="http://agents.md/">AGENTS.md</a>; 30 Minutes Setup, Hours Saved</h3><p>The single highest-leverage action you can take is writing a project instruction file. In Claude Code, this is CLAUDE.md. In Cursor, it is .cursorrules or similar. The name varies by tool, but the concept does not.</p><p>This file tells the AI how your project works before it writes a single line of code. It is the equivalent of the onboarding document you give a new team member: here is how we structure things, here is how to run tests, here are the patterns we follow, here are the mistakes people make.</p><p>Time investment: 30 minutes to create, 5 minutes to update as the project evolves.</p><p>What it prevents:</p><ul><li>Code that ignores your existing abstractions (the AI reinvents utilities you already have)</li><li>Test files in the wrong location or with the wrong framework</li><li>Import patterns that violate your conventions</li><li>Architecture decisions that contradict your system design</li></ul><p>Anthropic’s <a href="https://docs.anthropic.com/en/docs/claude-code/claude-md">CLAUDE.md best practices guide</a> is a good starting point. Your file should cover at minimum: project overview, how to run tests, code conventions, and known gotchas, based on your actual project, not a generic template.</p><p>The research supports this approach. Studies on project instruction files consistently show that curated, human-written instructions yield substantial benefits, whereas AI-generated instructions offer negligible improvement. McKinsey’s 2026 research found that <a href="https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai">57% of top-performing organizations</a> invested in hands-on AI workshops and coaching versus only 20% of bottom performers — the gap is about intentional setup, not just having the tools. The investment of writing project instructions yourself, based on actual project knowledge, is what makes them work.</p><h3>2. Skills Setup: 15 Minutes, Consistent Workflows</h3><p>Skills are reusable workflow definitions that encode expert knowledge into repeatable processes. Instead of re-explaining your testing approach, deployment steps, or code review criteria in every conversation, define them once and refer to them by name.</p><p>Time investment: 15 minutes per skill, most projects need 3–5.</p><p>What it prevents:</p><ul><li>Inconsistent approaches to recurring tasks (one conversation uses Vitest, the next switches to Jest)</li><li>Lost context between sessions (the AI “forgets” your testing patterns)</li><li>Scope creep on routine tasks (the AI adds features you did not ask for)</li></ul><p>However, a caveat: our internal data showed that 0% of our team was using Skills before our training program. And research from Vercel found that models fail to invoke Skills approximately 56% of the time; AGENTS.md-style project context consistently outperformed Skills-based approaches in their testing. Skills are powerful but require deliberate setup and team education to be effective. They are not a plug-and-play solution.</p><h3>3. Plan Mode — 5 Minutes Per Task, Avoid Scope Creep</h3><p>Plan Mode forces a two-step workflow: first, the AI proposes a plan; then, only after you review and approve, it executes.</p><p>Time investment: 5 minutes per task to review the plan before execution.</p><p>What it prevents:</p><ul><li>The AI modified files you did not intend it to touch</li><li>Scope creep (it “helpfully” refactors adjacent code)</li><li>Architectural decisions made implicitly during implementation</li><li>The scenario where you realize 30 minutes into execution that the approach is wrong</li></ul><p>This is the tool that directly addresses the Mathi Problem. If the plan had been reviewed before execution, the structural issues in the AI-generated feature would have been visible before a single line of code was written.</p><p>Our training data suggests Plan Mode can reduce feature development time by 60%, not because it writes code faster, but because it eliminates the rework cycle. Before our training program, only 62% of our team used Plan Mode. We set a target of 90%. We haven’t yet run the formal follow-up survey, but anecdotally, Plan Mode has become the default for multi-file changes across most teams.</p><h3>The Prevention Stack in Practice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PGrEFZJ7VHnM-BQ_.png" /></figure><h3>The Verification Loop</h3><p>Prevention reduces the frequency of bugs. Verification catches the ones that get through. Both are necessary.</p><p>After the Mathi Problem, we built a systematic verification process for AI-generated code. This is not a replacement for standard code review: it is an additional layer that targets the specific failure patterns of AI output.</p><p>Here is the verification framework we use:</p><h3>Step 1: Boundary Check</h3><p>Before reviewing the code itself, verify that the AI stayed within scope.</p><ul><li>Did it modify only the files you expected?</li><li>Did it add dependencies you did not request?</li><li>Is the PR size proportional to the task? (Remember: AI-assisted PRs tend to be 18% larger; some of that is unnecessary.)</li></ul><h3>Step 2: Security Scan</h3><p>AI-generated code has a disproportionately high rate of security vulnerabilities. Check for:</p><ul><li>Hardcoded credentials or secrets</li><li>Missing input validation</li><li>SQL injection or XSS vectors</li><li>Overly permissive access controls</li><li>Dependencies with known vulnerabilities</li></ul><p>This is not paranoia. When the data says ~45% of AI code contains security flaws, a dedicated security pass is not optional.</p><h3>Step 3: Logic Review</h3><p>The AI often produces code that works for the happy path but fails on edge cases. Focus on:</p><ul><li>Null/undefined handling</li><li>Empty collections and boundary conditions</li><li>Error paths and exception handling</li><li>Race conditions in async code</li><li>Off-by-one errors (a classic, and AI is not immune)</li></ul><h3>Step 4: Test Audit</h3><p>Check that the tests actually test the right things:</p><ul><li>Do tests cover edge cases, or only the happy path?</li><li>Are assertions meaningful? (AI sometimes writes tests that always pass.)</li><li>Do tests use the project’s testing patterns and utilities?</li><li>Is the test-to-code ratio reasonable?</li></ul><h3>Step 5: Pattern Conformance</h3><p>Verify the code follows your project’s established patterns:</p><ul><li>Does it use your existing abstractions or reinvent them?</li><li>Does it follow your naming conventions?</li><li>Does the error handling match your project’s approach?</li><li>Are the imports organized the way your team expects?</li></ul><p>This is where a well-written <a href="http://claude.md/">CLAUDE.md</a> pays dividends; if the AI has the right instructions, pattern-conformance issues should be rare. If they are frequent, your project instruction file needs to be updated.</p><h3>Step 6: Integration Check</h3><p>Verify the code works within the broader system:</p><ul><li>Does it handle the data contracts correctly (API schemas, database types)?</li><li>Does it interact correctly with adjacent services?</li><li>Are there side effects that the AI might not have anticipated?</li></ul><h3>Results</h3><h3>Before the Prevention Framework</h3><p>The Mathi Problem was our wake-up call, but it was not an isolated incident. Before we implemented structured prevention:</p><ul><li>AI-generated features had an unpredictable debugging tail</li><li>Code reviews caught issues late in the process</li><li>Developers treated AI output with the same trust level as their own code</li><li>There was no systematic approach to AI-specific failure modes</li></ul><h3>After</h3><p>Once we rolled out the Prevention &gt; Verification framework through our IA Level Up training:</p><ul><li>Every project has a <a href="http://claude.md/">CLAUDE.md</a> file, written by the team, updated as the project evolves</li><li>Plan Mode is the default for any task that touches more than one file</li><li>The Verification Loop checklist is part of our PR review process</li><li>AI-generated code is treated as what it is: output from a fast, capable tool that lacks project context and judgment</li></ul><p>We don’t have clean before/after metrics yet; measuring code quality across an entire team is harder than measuring tool adoption. What we can say: teams that adopted the full prevention stack (<a href="http://claude.md/">CLAUDE.md</a> + Plan Mode + verification checklist) report fewer surprise debugging sessions. The “half-day feature, three-day fix” pattern has become a cautionary tale the team references, not a recurring experience. We’ve also observed an increase in the number of PRs being shipped across projects, suggesting the framework isn’t slowing teams down; it’s giving them confidence to move faster.</p><p>The shift is not about using AI less. We use it more than ever. The shift is about using it within a framework that matches the tool’s strengths (speed, volume, pattern implementation) while compensating for its weaknesses (judgment, context, edge case awareness).</p><h3>What You Can Do Today</h3><p>If the Mathi Problem resonates, if you have seen the half-day feature that becomes a three-day debugging session, here is where to start:</p><p><strong>This afternoon (30 minutes):</strong><br>Write a <a href="http://claude.md/">CLAUDE.md</a> for your current project. Start with your project overview, how to run tests, code conventions, and known gotchas. Anthropic’s <a href="https://docs.anthropic.com/en/docs/claude-code/claude-md">CLAUDE.md best practices guide</a> is a good reference.</p><p><strong>This week (1 hour):</strong><br>Create 2–3 Skills for your most common workflows (testing, PR preparation, code review). Enable Plan Mode as your default for multi-file changes.</p><p><strong>This month:</strong><br>Adopt a verification loop in your PR review process: boundary check, security scan, logic review, test audit, pattern conformance, and integration check. Track how many issues it catches. Refine your <a href="http://claude.md/">CLAUDE.md</a> based on patterns you see.</p><p>The tools are not the problem. The absence of a configured environment is. Prevention is a one-time investment with compounding returns.</p><h3>Further Reading</h3><ul><li>The Feature Utilization Gap: What 85% AI Tool Adoption Really Looks Like, The data that led to building this workshop</li><li>The Mathi Problem: Why AI-Generated Features Ship Fast and Break Slow, The full story behind Module 6’s core anecdote</li><li>AI Tools Are Not Just for Developers: How We’re Expanding Beyond Engineering, What happened when we applied similar thinking to PMs, HR, and delivery</li><li>Anthropic Skilljar: Claude Code in Action, an external training resource we recommend as a supplement</li><li>Anthropic Prompt Engineering Guide, Foundation for the Prompt Formula, taught in Module 1</li></ul><p>This article is part of our AI journey series. Published April 2026. The tools, features, and recommendations described here reflect the landscape as of that date. AI tooling moves fast; verify current capabilities before building on our specific tool recommendations.</p><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/the-mathi-problem-ai-generated-code/">https://blog.xmartlabs.com/blog/the-mathi-problem-ai-generated-code/</a> <em>on Jun 25, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c761e3d1eae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Ways to Test Whether Your Medical AI Is Dangerously Overconfident]]></title>
            <link>https://xmartlabs.medium.com/5-ways-to-test-whether-your-medical-ai-is-dangerously-overconfident-8245da1007b3?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/8245da1007b3</guid>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 14:54:42 GMT</pubDate>
            <atom:updated>2026-06-25T14:54:42.638Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4883QA8Hu_xFavKq.png" /></figure><p>Let’s talk about a scary thought:</p><p><strong>What if a chatbot is too good at sounding sure of itself?</strong></p><p>In the world of AI, there’s a sneaky phenomenon called <em>Automation Bias</em>. It’s that instinct we all have to trust a computer’s answer blindly, just because it sounds “smart”.</p><p>But in medical software, that kind of blind trust can be dangerous.</p><p>Today, I want to take you under the hood of a <strong>fictional project</strong>: <em>MediBot,</em> an AI triage assistant that uses NLP and Machine Learning to help patients decide what to do when they feel unwell.</p><p>While MediBot isn’t real, the risks and the testing strategies absolutely are.</p><p>This post was inspired by concepts from the ISTQB AI Testing syllabus and how they can be applied to real-world scenarios like this one.</p><h3>Meet MediBot: Your Virtual Triage Assistant</h3><p>MediBot’s job is simple: you describe your symptoms, and it gives you a recommendation.</p><p>Behind the scenes, a system like this would typically rely on:</p><ul><li><strong>NLP</strong> → to understand the user’s input</li><li><strong>ML Classifier</strong> → to estimate probabilities (Is it flu? COVID? something more serious?)</li><li><strong>Decision Logic</strong> → to decide whether you need rest… or urgent care</li></ul><p>Sounds straightforward, right?</p><p>Well… not quite.</p><h3>The “Oops” Moment</h3><p>Imagine a user says:<br><em>“My chest hurts when I walk fast.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0CL6NZ1LR1BNA7Tm.png" /></figure><p>A system like MediBot might internally estimate something like:</p><ul><li>60% → acid reflux</li><li>10% → heart-related issue</li></ul><p>And then respond:</p><p><em>“It’s likely indigestion. Try resting and drinking water.”</em> ❌</p><p>Wait… what?</p><p>That’s a potentially serious cardiac symptom being dismissed as something minor.</p><p>And here’s the real danger:</p><p>Because the chatbot sounds calm and confident, a user might actually trust it and stay home.</p><p>That’s <strong>Automation Bias in action</strong>.</p><h3>How Should We Fix It? 5 Pro-Testing Moves</h3><p>Inspired by ISTQB principles for AI testing, here are five approaches teams should apply to make systems like MediBot safer and more reliable.</p><h3>1. The “Data Diet” (Testing for Bias)</h3><p>One of the first things to question is the training data.</p><p>In a system like MediBot, it’s easy to imagine a dataset where:</p><ul><li>Most cases are mild (anxiety, reflux, flu)</li><li>Very few represent critical conditions</li></ul><p>That imbalance creates bias.</p><p>👉 The model learns what is <em>common</em>, not what is <em>critical</em>.</p><p><strong>What should be done:</strong></p><ul><li>Balance the dataset with more high-risk scenarios</li><li>Ensure rare but dangerous conditions are well represented</li></ul><p><strong>Why this matters for QA:</strong></p><p>Even if a condition is statistically rare, it must be treated as a high priority in testing. In medical AI, <strong><em>severity matters more than frequency</em></strong>.</p><h3>2. Defect Prediction: Testing Smarter, Not Harder</h3><p>Instead of guessing where bugs might appear, a Defect Prediction approach can help teams focus on the most fragile parts of the system.</p><p>Think of it as a <strong>“weather forecast” for bugs</strong>.</p><p>In a system like MediBot, a model could be fed with:</p><ul><li>Code complexity</li><li>History of previous defects</li><li>Recent changes in medical logic</li></ul><p><strong>The output might look like this:</strong></p><ul><li>NLP Engine → 20% risk</li><li>Rule Engine → 15% risk</li><li>ML Symptom Classifier → 65% risk 🚩</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d58nax6yj2I6mdd8.png" /></figure><p>That insight should directly shape the testing strategy.</p><p>👉 Instead of testing everything equally, teams should prioritize the <strong>highest-risk component</strong>.</p><p>In this case, the Symptom Classifier, where misclassification could directly impact patient safety.</p><p>Testing here should go deeper:</p><ul><li>Edge cases</li><li>Critical symptom combinations</li><li>Adversarial scenarios</li></ul><p><strong>Why this matters for QA:</strong></p><p>Not all parts of an AI system fail equally. Defect Prediction helps teams focus on what actually matters, so they’re not just testing more, but testing <em>smarter</em>.</p><h3>3. Mixing It Up (Synthetic Data)</h3><p>Let’s be real: people don’t talk like medical textbooks.</p><p>A doctor might say <em>“angina pectoris”</em>, but a user might say:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rIJ_aXTyEZWD_Iyk.png" /></figure><p>If the AI only understands formal language, it will fail in the real world.</p><h3><strong>What should be done:</strong></h3><ul><li>Generate synthetic variations of the same intent</li><li>Test across different tones, styles, and expressions</li></ul><p>This allows teams to scale testing from a few examples… to thousands.</p><p><strong>Why this matters for QA:</strong></p><p>Without linguistic diversity, the model doesn’t truly understand users; it’s just matching keywords.</p><p><strong>4. The “Trick” Questions (Adversarial Testing)</strong></p><p>To really validate robustness, you need to think like an attacker or a biased user.</p><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UIeDOZIUotUVve6M.png" /></figure><p>A poorly tested system might latch onto “pizza” and downplay the real issue.</p><p>👉 This is a classic <strong>adversarial scenario</strong>:</p><p>A distracting detail leads the model away from a critical symptom.</p><p><strong>What should be done:</strong></p><ul><li>Test with misleading, biased, or manipulative inputs</li><li>Ensure high-risk symptoms override contextual noise</li></ul><p><strong>Why this matters:</strong></p><p>AI should not blindly agree with the user. Safety must take priority over being “helpful”.</p><h3>5. Managing the “Vibe” (Automation Bias Evaluation)</h3><p>This one is subtle but critical.</p><p>Compare this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8zMi4N49McKczgJk.png" /></figure><p><strong>What should be done:</strong></p><ul><li>Avoid overly confident language</li><li>Introduce uncertainty when appropriate</li><li>Encourage safe actions in ambiguous cases</li></ul><p><strong>Why this matters:</strong></p><p>Confidence directly influences user decisions. Reducing overconfidence helps mitigate <strong>Automation Bias</strong>.</p><h3>From Theory to Practice: A Real-World Example (Redi)</h3><p>While <em>MediBot</em> is a fictional example, these challenges show up in real systems too, just in different ways.</p><p>At Xmartlabs, we built a product called <em>Redi</em> that tackles a similar challenge from a different angle. Unlike MediBot, it doesn’t diagnose or estimate probabilities. Instead, it helps users prepare for a medical consultation by asking about their symptoms and generating a summary with suggested questions for their doctor.</p><p>At first glance, it’s a much simpler system:</p><ul><li>It relies on an LLM and prompt design</li><li>There’s no custom-trained classifier or complex decision engine</li><li>The logic is embedded directly in how the model is guided</li></ul><p>But the <strong>safety challenge</strong> <strong>remains.</strong></p><p>👉 What happens if a user describes an emergency?</p><p>This is where a key concept comes in: <strong>hard stops</strong>.</p><p>If a user mentions symptoms that could indicate a critical condition, such as a stroke, the system should not continue the normal flow. It should interrupt the experience and clearly direct the user to seek immediate help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ikvFNnW3ksZJ95LK.png" /></figure><p>For example:</p><ul><li>Suggest calling emergency services</li><li>Show nearby clinics or hospitals</li><li>Stop generating non-critical guidance</li></ul><p>And this is not just a design decision, it’s something that must be <strong>thoroughly tested</strong>.</p><p>As part of QA, we need to validate that the hard stop is consistently triggered across different ways of expressing the same emergency.</p><p>For example, instead of testing just:</p><ul><li><em>“I think I’m having a stroke”</em></li></ul><p>We should also test variations like:</p><ul><li><em>“My face feels numb, and I can’t move my arm”</em></li><li><em>“I suddenly can’t speak properly”</em></li><li><em>“One side of my body feels weak”</em></li></ul><p><strong>Why this matters:</strong></p><p>Users don’t describe emergencies in predictable ways. If the system only catches one phrasing, it’s not truly safe.</p><p>In this case, the “decision logic” isn’t a separate module; it’s part of the prompt and the guardrails around the LLM.</p><p>Even in simpler AI systems, safety cannot be optional. The implementation might look different, rules engine vs prompt design, but the responsibility is the same.</p><p>There’s no single formula for testing AI systems.</p><p>Each architecture requires a different strategy, but the goal is always consistent:</p><p><strong>Ensure the system behaves safely when it matters most.</strong></p><h3>The Bottom Line</h3><p>As QAs, we’re not just testing features anymore.</p><p>We’re testing:</p><ul><li>decisions</li><li>assumptions</li><li>and real-world consequences</li></ul><p>In AI systems, being “smart” is not enough.</p><p><strong>A smart chatbot is useful. But a safe chatbot is essential.</strong></p><p>Because in the end, the real question isn’t: <em>“Does it work?”</em> It’s: <strong><em>“Does it fail safely when it matters most?”</em></strong></p><p>What do you think? Have you ever trusted an AI a bit too much, just because it sounded confident?</p><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://deploy-preview-310--blog-xl.netlify.app/blog/automation-bias-medical-ai-chatbot/">https://deploy-preview-310--blog-xl.netlify.app/blog/automation-bias-medical-ai-chatbot/</a> <em>on Jun 11, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8245da1007b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Spaghetti Data Crisis Every HealthTech Startup Eventually Faces]]></title>
            <link>https://xmartlabs.medium.com/the-spaghetti-data-crisis-every-healthtech-startup-eventually-faces-3cdff80e2139?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cdff80e2139</guid>
            <category><![CDATA[digital-health]]></category>
            <category><![CDATA[fhir]]></category>
            <category><![CDATA[healthtech-development]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 14:50:48 GMT</pubDate>
            <atom:updated>2026-06-25T14:50:48.344Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tjaWjaP3cUsREN8_.png" /></figure><p>After working across multiple healthcare projects, I kept seeing the same pattern: different teams, different products, different markets — same crisis. We started calling it the <em>Spaghetti Data Crisis (not copyrighted)</em>, and this post is our attempt to put a name to something that’s quietly slowing down healthtech teams.</p><p>Let’s set the scene. You’re three months post-launch. Your health-tech startup just closed a pilot with a mid-sized clinic, or you’re already running with a few hundred users. Congrats. Seriously. That’s a significant milestone.</p><p>Everything is moving fast. Your backend, for example, has a patient object that started as a JSON document in a MongoDB collection, grew into a relational table, and was then extended with five ad hoc fields because every new feature needed <em>just one more thing.</em></p><p>The client is now asking you to integrate with their EHR. You pull up the codebase and realize: there’s no standard. No schema that anyone outside your team would recognize. No clear boundary between what a “patient” is, what an “encounter” is, or how a “medication” relates to either. It’s all custom. All yours. All unmaintainable.</p><p>Welcome to the <em>Spaghetti Data Crisis</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*-yGgtRhPMXQnCMNG.gif" /></figure><h3>The Inflection Point: When Custom Chaos Becomes a Risk</h3><p>Here’s the thing — if you’ve found yourself in this situation, it doesn’t mean you failed. Anyone who’s built in this space knows that the “right” data architecture isn’t just about <em>how</em> you store data. It’s about <em>where</em>, <em>what</em>, security, cost, compliance, and a dozen other constraints that don’t have clear answers when you’re a small team with limited runway. How many pilots actually go to production overnight?</p><blockquote><em>The good news? You’re not too late. You’re just at the inflection point.</em></blockquote><p><em>This is the moment that separates healthtech companies that scale from those that spend the next months/years paying down technical debt instead of expanding.</em></p><p>The point isn’t a single event. It usually looks like one of these:</p><ul><li>A health system asks for an FHIR API during procurement, and your team realizes you’d have to rebuild your entire data layer to support it.</li><li>You try to pull analytics across patients and encounters, and nobody can agree on what fields mean what because three engineers built three slightly different versions of the same concept.</li><li>A compliance audit reveals inconsistencies between how your app stores data and what HIPAA requires you to report.</li><li>You lost a deal because your competitor had a FHIR-native integration, and you didn’t.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JsxwIXq5-1Zvq8I8.png" /></figure><p>At this point, teams typically do one of two things: they double down on the custom model (“we’ll clean it up later”) or they face the music and start thinking about standards. The ones who delay usually face a much harder refactor 12 months later — when they can least afford it.</p><h3>FHIR in Practice: You Probably Need Less Than You Think</h3><p>Before we get into why we always recommend moving toward this approach, it’s worth pausing to clarify: the right entry point depends on where you currently stand.</p><ol><li><strong>If you’re just getting started</strong>, whether with your own team or alongside a development partner (like <a href="https://xmartlabs.com/services">Xmartlabs</a> ❤️), do everything you can to adopt FHIR as your standard from day one. Later in this post, we’ll show how, even for engineers with no FHIR experience, the short-term learning curve pays off quickly. It’s a win-win.</li><li><strong>If you’re already in production</strong> and feeling the pressure of the crisis we described above, the good news is that you can start taking meaningful action at a relatively low operational and budget cost. You don’t need a full rewrite to break through the inflection point; you need a strategy, and we’ll share a few on this blog later.</li></ol><p><strong>Now, back to the core question: why FHIR specifically?</strong></p><p>Beyond being the most widely adopted standard in healthcare today, FHIR has something that most technical standards don’t — a documentation ecosystem rich enough for both engineers and non-technical stakeholders to engage with on their own terms. That shared language matters more than people expect. It means your clinical advisor, your compliance team, and your backend engineer can sit in the same room and actually understand each other.</p><p><strong>Theory is useful. But here’s what this actually looks like in practice.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3CLOu-ckJJYWYchC.png" /></figure><p><strong>Real case #1 — Building from scratch</strong> <em>VR-powered consultations in challenging environments</em></p><p>A client needed to manage medical consultations delivered via VR in physically or logistically demanding settings, such as remote locations, high-stress environments, and areas with limited connectivity. Because we mapped FHIR resources to the consultation model from day one, the entire data layer was built around standard entities. With just a handful of resources, the product interoperated with external services without any structural rework. No custom translation layers. No retrofitting. The standard did the heavy lifting before the product even reached its first users.</p><p><strong>The takeaway:</strong> Starting with FHIR adds some upfront weight, a steeper learning curve, and a few more decisions to make early on, but it removes an entire category of future problems. The kind that show up at the worst possible moment, when you start to scale.</p><p><strong>Real case #2 — Already in production</strong>, <em>Turning voice into structured clinical data</em></p><p>A health services provider was generating hours of clinical voice conversations with no standardized output — just raw audio and informal notes. The goal was to turn those conversations into automated, structured transcripts that could plug into existing clinical workflows. The product already had traction but lacked a standard data model. Rather than a full rewrite, we ran a modular rollout: incrementally introducing FHIR-aligned structures by service area, using out-of-the-box tools where possible. Each module became interoperable without breaking what was already working.</p><p><strong>The takeaway:</strong> You don’t need to stop shipping to fix your foundation. A modular approach to FHIR adoption lets you reduce technical debt without losing momentum.</p><blockquote><em>🗒️ </em><strong><em>A practical note:</em></strong><em> In the vast majority of cases, especially for focused, single-problem products (which tend to be the ones that win), you don’t need to implement the full FHIR specification. Not even close. Getting fluent in just a handful of core resources: </em><strong><em>Patient, Practitioner, Encounter, Observation, Condition,</em></strong><em> and </em><strong><em>MedicationRequest</em></strong><em> will cover roughly 90% of your data model needs.</em></blockquote><blockquote><em>Start there, and then do the whole transition if needed.</em></blockquote><h3>Now That You’re Convinced: The Pragmatic FHIR Stack</h3><p>The most common mistake teams make after deciding to adopt FHIR is trying to boil the ocean. They download the full spec, open the <a href="https://www.hl7.org/fhir/overview-dev.html">HL7 documentation</a>, and close the tab three minutes later.</p><p>Here is what worked for us, broken into three layers.</p><p><strong>1. Knowledge Boost: Learn the Standard</strong></p><ul><li><a href="https://www.hl7.org/fhir/overview-dev.html">FHIR for Developers (HL7 official)</a> — The official developer overview. Read it once, top to bottom. It is shorter than you think.</li><li><a href="https://smilecdr.com/docs/">Smile CDR’s FHIR Learning Center</a> — One of the clearest practical introductions available, written by people who build FHIR infrastructure for a living.</li><li><a href="https://medblocks.com/training/courses/fhir-fundamentals">FHIR Fundamentals Course (Medblocks)</a> — A beginner-friendly, hands-on course covering FHIR resources, REST APIs, transactions, search, and cloud FHIR servers. Great if you prefer structured learning over reading the specification directly.</li><li><a href="https://simplifier.net/">Simplifier.net</a> — A massive repository of Implementation Guides, profiles, and examples used by healthcare organizations worldwide.</li><li><a href="https://hl7.org/fhir/us/core/">US Core Implementation Guide</a> — Essential if you are going to write Implementation Guides or define custom profiles.</li><li>The <a href="https://chat.fhir.org/">#fhir Zulip chat</a> — An active community where you can ask very specific implementation questions and get answers from people who maintain the spec itself.</li></ul><blockquote><em>💡</em><strong><em>Tip</em></strong><em>: Assign a team member as the internal FHIR SME lead. Not a permanent role, just someone who goes deeper and becomes the go-to reference for the rest of the team for the first few months.</em></blockquote><p>Fair warning: this one is for the engineers on the team. Once you have enough shared context, the next question is tooling. The good news is that the ecosystem has matured significantly. You do not need to build everything from scratch.</p><p><strong>2. FHIR Servers (managed or self-hosted):</strong></p><ul><li><a href="https://hapifhir.io/">HAPI FHIR</a> — Open source, Java-based, battle-tested. If you want full control and are comfortable running your own infrastructure, this is the most widely adopted open source option.</li><li><a href="https://learn.microsoft.com/en-us/azure/healthcare-apis/">Azure Health Data Services</a> / <a href="https://cloud.google.com/healthcare-api">Google Cloud Healthcare API</a> / <a href="https://aws.amazon.com/healthlake/">AWS HealthLake</a> — Managed FHIR servers if you are already on one of the major clouds and want to avoid infrastructure overhead early on. We leaned on these early precisely because we did not want to maintain a server while also building a product.</li><li><a href="https://www.medplum.com/">Medplum</a> — A newer, developer-friendly FHIR platform with a strong DX focus. Worth evaluating if you want a modern API-first approach without heavy enterprise overhead.</li><li><a href="https://github.com/anthropics/healthcare/tree/main/fhir-developer-skill">FHIR Developer Skill (Anthropic)</a> — A practical AI-assisted development guide for building FHIR APIs. Covers resource validation, OperationOutcome errors, correct HTTP status codes, SMART on FHIR authorization, and common coding systems such as LOINC, SNOMED CT, RxNorm, and ICD-10. Particularly useful when implementing FHIR endpoints with tools like Claude Code or Cursor.</li></ul><blockquote><em>We have used both </em><strong><em>Google Cloud Healthcare API</em></strong><em> and </em><strong><em>Medplum</em></strong><em> in real-world projects and can confidently recommend both. Google Cloud Healthcare API is an excellent choice if you are already invested in Google Cloud and want a fully managed FHIR infrastructure layer. Medplum, on the other hand, is much more than a FHIR server: it is an open-source healthcare development platform that includes core EHR capabilities such as patient management, scheduling, forms, communications, and authentication out of the box. This makes it particularly attractive when developer experience, rapid prototyping, and launching a production-ready healthcare application are the priority. We are also actively exploring </em><strong><em>AWS HealthLake</em></strong><em> as another managed option within the Amazon Web Services ecosystem.</em></blockquote><p><strong>3. Management: Project and Data Governance</strong></p><p>Technical execution and information governance go hand in hand. Not a groundbreaking insight for anyone who has shipped software before. But in healthtech, the cost of getting this wrong is higher than in other domains. So alongside the technical stack, we always pair a set of non-technical practices that save us from future headaches:</p><p><strong>Define your core resource inventory first.</strong> Before writing any profiles or standing up a server, map out which FHIR resources are actually relevant to your product. Most early-stage healthtech products require fewer than 10 resource types to get started. Patient, Observation, Encounter, Condition, MedicationRequest, Practitioner, and Organization cover most clinical workflows. Write this down. Make it visible. It becomes your team’s shared vocabulary.</p><p><strong>Establish a FHIR version policy and review it periodically.</strong> Pick a version upfront (R4 is the current standard for most US regulatory and interoperability requirements) and document it explicitly. But do not treat it as a one-time decision. Schedule a lightweight quarterly review to assess how your resource mappings have evolved, whether any new profiles are relevant to your use case, and whether R5 adoption in your target integrations is accelerating. Mappings drift silently over time. A periodic review is the cheapest way to catch that drift before it becomes a migration project.</p><p><strong>Involve a clinical informatics person early, even informally.</strong> If you do not have one on staff, find an advisor or consultant who can review your resource mappings. Clinical data has semantic nuance that engineering instincts alone do not always catch. A Condition is not the same as a Diagnosis. An Observation value has units that matter. Getting this wrong early means having to clean up data integrity issues at the worst possible time.</p><h3>What’s Next?</h3><p>Hope this was useful, wherever you are in your journey. Whether you are staring down the inflection point or already past it, the goal here was to share some of what we have learned, without pretending there is one right answer for every startup.</p><p>This is the first of three posts on this topic. Partly because trying to fit everything into one piece felt like bad advice in itself. And partly because we, or at least I, know how frustrating it is to read something that does a good job of naming a problem, throws out a few tips, and then leaves you without a path forward. I have been on the receiving end of that too many times as a developer. The next two posts will go deeper into the actual solutions: concrete implementation, real tradeoffs, and the things we wish someone had told us earlier.</p><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/why-fhir-is-the-foundation-healthcare-products-need/">https://blog.xmartlabs.com/blog/why-fhir-is-the-foundation-healthcare-products-need/</a> <em>on Jun 3, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cdff80e2139" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Small Language Models Might Win in Healthcare]]></title>
            <link>https://xmartlabs.medium.com/why-small-language-models-might-win-in-healthcare-6ad049b6968e?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ad049b6968e</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[health-technology]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 16:00:39 GMT</pubDate>
            <atom:updated>2026-06-23T20:01:26.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_QJxNjy6H_31zt0q.png" /></figure><p>Is it possible to get the reasoning of a +100B parameter model packed into a 400MB binary that runs on a phone? Yes. In this post, I’ll show you how I integrated the reasoning capabilities of a frontier model like Gemini 3.1 Pro into a small model like Qwen with just 500 million parameters, using an SFT (Supervised Fine-Tuning) strategy on real-world data. The result? Minimal latency, zero cost, and complete privacy.</p><h3>1. The Bias of “AI Obesity.”</h3><p>In agent development, technical inertia pushes us to always use the most powerful model available: “For any problem, the natural instinct is to reach for GPT-5.5 or Claude 4.6 Sonnet.” However, in production environments (and even more so in HealthTech), this approach runs into three hard walls:</p><ol><li><strong>Latency:</strong> For a chat interface, waiting 5–10 seconds for Time To First Token (TTFT) destroys the user experience.</li><li><strong>Privacy:</strong> In healthcare, privacy isn’t a “feature”; it’s a legal requirement. Sending patient data to third-party cloud services complicates compliance with regulations such as HIPAA and GDPR.</li><li><strong>Scalability and Vendor Lock-in:</strong> Building on top of third-party APIs ties your product to a critical dependency on external providers (vendor lock-in). At high volumes, variable operational costs become prohibitive, threatening the agent’s long-term financial viability. Local execution transforms a marginal per-token cost into a fixed, controllable infrastructure cost.</li></ol><p><strong>The necessary insight:</strong> We don’t need a model that can write COBOL and explain quantum physics for a task that specifically requires structuring medical information under a protocol. This is where <strong>Distillation</strong> comes in.</p><h3>2. The Concept: Model Distillation (Teacher-Student)</h3><p>Distillation isn’t just “copying” the large model. It’s a process in which a <strong>Teacher Model</strong>, in our case, Gemini, generates high-quality “rationales” or responses that serve as labels for training a much smaller <strong>Student Model</strong>.</p><h3>The Use Case: Redi (Medical Preparation Agent)</h3><p>Our agent helps patients synthesize their condition before seeing a doctor, using the <strong>SBAR</strong> framework (<em>Situation, Background, Assessment, Recommendation</em>).</p><h3>Infrastructure Evolution</h3><p>We moved from a complex architecture on <strong>AWS Bedrock</strong> (with VPCs, Private Links, and fixed management costs) to a model that can live in a lightweight container or even on the <em>edge</em>.</p><h3>3. The Data Strategy (Supervised Fine-Tuning): Building the “Golden Dataset.”</h3><p>The success of fine-tuning depends not on quantity, but on <strong>information density</strong>. Many developers fail when they try to fine-tune on thousands of noisy examples. We bet on quality: <strong>327 perfect examples</strong>.</p><h3>The Data Pipeline (Data Engineering)</h3><p>Before training, we transformed logs from real (anonymized) conversations into a training dataset. The flow was:</p><ol><li><strong>Observability:</strong> Capturing real AI-User interactions.</li><li><strong>Cleaning:</strong> Chronological ordering by conversation_id and role normalization (human -&gt; user).</li><li><strong>Rationalization Filtering:</strong> This is where the magic happens. We used the “Teacher” to clean the noise.</li></ol><h3>The Distillation Prompt (The “Brain” of the Process)</h3><p>We didn’t ask the small model to learn from raw logs. We used a rewriting prompt to have Gemini act as an <strong>expert curator</strong>.</p><p>This is the script we used to generate our <strong>JSONL, compatible with </strong><strong>apply_chat_template</strong>:</p><pre>import json<br>from typing import Optional, Dict</pre><pre>def distill_conversation(json_line: str) -&gt; Optional[Dict]:<br>    &quot;&quot;&quot;<br>    Transforms a raw conversation into a &#39;Gold Standard&#39; example<br>    using a Teacher LLM to align the output with the SBAR framework.<br>    &quot;&quot;&quot;<br>    base_prompt = &quot;&quot;&quot;<br>    You are an expert in Data Engineering for LLMs and a specialist in SBAR medical communication.<br>    <br>    MISSION:<br>    You will receive a real conversation between a user and the assistant &#39;Redi&#39;.<br>    Your task is to rewrite the assistant&#39;s response to make it a perfect training example.<br>    <br>    GOLDEN RULES:<br>    1. SBAR Framework: The response MUST include: Situation, Background, Assessment, Recommendation.<br>    2. No Diagnosing: Validate symptoms and prepare the consultation, but never give a medical diagnosis.<br>    3. Consistency: Empathetic, professional, and structured tone.<br>    4. Quality Filter: If the conversation is trivial (e.g., just a &#39;hello&#39;), respond with &#39;SKIP&#39;.<br>    <br>    OUTPUT: Return EXCLUSIVELY a JSON with the structure:<br>    {&quot;messages&quot;: [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;...&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;...&quot;}, {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;...&quot;}]}<br>    &quot;&quot;&quot;<br>    <br>    try:<br>        # Call to the Teacher model (Gemini 3.1 Pro)<br>        full_prompt = f&quot;{base_prompt}\n\nORIGINAL CONVERSATION:\n{json_line}&quot;<br>        response = model.generate_content(full_prompt)<br>        text = response.text.strip()</pre><pre>        if &quot;SKIP&quot; in text: return None</pre><pre>        # Clean markdown if the model responds with code blocks<br>        clean_json = text.replace(&quot;```json&quot;, &quot;&quot;).replace(&quot;```&quot;, &quot;&quot;).strip()<br>        return json.loads(clean_json)<br>        <br>    except Exception as e:<br>        print(f&quot;Distillation error: {e}&quot;)<br>        return None</pre><p>An example of the resulting training record:</p><pre>{<br>  &quot;messages&quot;: [<br>    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a medical intake assistant.&quot;},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;I have frequent headaches.&quot;},<br>    {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;How long have you been experiencing them?&quot;}<br>  ]<br>}</pre><h3>Why is this approach superior?</h3><ul><li><strong>Strict Alignment:</strong> We force the small model to learn the SBAR format from day one.</li><li><strong>Hallucination Elimination:</strong> By using real data rewritten by a superior model, the small model learns safe and accurate response patterns.</li><li><strong>Cost-Effective:</strong> We pay for Gemini API calls only once per dataset.</li></ul><h3>4. Technical Implementation: From Theory to 95 Seconds of Training</h3><p>To train a model faster than it takes to make a cup of coffee, we used an optimized “heavy artillery” stack: <strong>Unsloth + QLoRA</strong>.</p><h3>The Heart of the Pipeline: QLoRA and Adapters</h3><p>We don’t update all 500 million parameters of the model. That would be inefficient. Instead, we use <strong>LoRA (Low-Rank Adaptation)</strong>.</p><p><strong>The technical intuition:</strong> Think of the base model as a general medicine textbook. Instead of rewriting the whole book (full fine-tuning), we add “sticky notes” (Adapters) on the key pages that teach it to structure information in SBAR format. Mathematically, we factorize the weight update matrix into two much smaller matrices (A and B), drastically reducing the computational load.</p><h3>Training Configuration</h3><p>These are the “magic numbers” that allowed the model to converge rapidly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*CJQQyZXLcEW2ZVXl.png" /></figure><h3>Responses-Only Training</h3><p>One of the keys to our precision was enabling --train_responses_only. We didn&#39;t want the model to learn <strong>to imitate the user&#39;s speech, but rather to learn how the medical assistant should respond</strong>. This concentrates all the gradient on the output policy.</p><h3>Code Snippet for Reproduction</h3><pre>from unsloth import FastLanguageModel<br>from trl import SFTTrainer<br>from unsloth.chat_templates import train_on_responses_only</pre><pre># Load the model in 4-bit to democratize training (runs on 8GB GPUs)<br>model, tokenizer = FastLanguageModel.from_pretrained(<br>    model_name = &quot;unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit&quot;,<br>    max_seq_length = 2048,<br>    load_in_4bit = True,<br>)</pre><pre># Insert adapters into the Attention and MLP modules<br>model = FastLanguageModel.get_peft_model(<br>    model,<br>    r = 16,<br>    target_modules = [&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;, &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;],<br>    lora_alpha = 16,<br>    lora_dropout = 0,<br>)</pre><h3>5. Results: How Good Is a 0.5B Model?</h3><p>The biggest risk with fine-tuning on such small models is <strong>“catastrophic forgetting,”</strong> or the model simply learning to repeat the format without understanding the underlying logic. To avoid confirmation bias, we subjected the model to a two-level evaluation process.</p><h3>A. Internal Metrics (Optimization Health)</h3><p>First, we verified that the learning process was stable. We weren’t just looking for the loss to drop; we wanted it to drop smoothly, indicating the model is generalizing patterns rather than noisily memorizing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/0*jo9D4JuPjMP_Kkk-.png" /></figure><blockquote><strong><em>Evidence:</em></strong><em> As shown in the </em><strong><em>MLflow</em></strong><em> logs (image above), we achieved a 46% reduction in loss with a stable gradient. This confirms that the “Student” (Qwen) has healthily absorbed the structure from the “Teacher” (Gemini).</em></blockquote><h3>B. External Benchmark (HealthBench)</h3><p>For the test, we didn’t use data that the model had already seen. We used a test battery based on the <strong>HealthBench</strong> framework (inspired by protocols from OpenAI and Google for healthcare).</p><p><strong>Why this benchmark?</strong> Because it doesn’t just measure “whether the sentence sounds nice.” It measures critical dimensions:</p><ol><li><strong>Clinical Accuracy:</strong> Is the medical information correct?</li><li><strong>Safety:</strong> Does it avoid giving dangerous or incorrect diagnoses?</li><li><strong>Key Omissions:</strong> Did it forget to ask about allergies or key medical history?</li><li><strong>SBAR Compliance:</strong> Did it follow the professional communication structure?</li></ol><p>We compared three models under equal conditions: the <strong>original Qwen 2.5 0.5B</strong>, our <strong>Qwen FT (Fine-Tuned)</strong> version, and the “upper bound” (market reference), <strong>Claude 4.6 Sonnet</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LgkWOk0_M3M8sUVN.png" /></figure><p><strong>The big surprise:</strong> Although Claude remains the king of pure reasoning, our model with just <strong>0.5B parameters</strong> managed to outperform the base model on every quality metric, and came dangerously close to frontier-model performance on the specific SBAR task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VPVuD4NeU3JEMqpj.png" /></figure><h3>. The Quality Leap (Baseline vs FT)</h3><p>The fine-tuned model doesn’t just “speak better” it is safer and more precise than its base version:</p><ul><li><strong>Clinical Accuracy:</strong> Increased from 5.90 to <strong>6.06</strong>.</li><li><strong>Safety Check:</strong> Improved from 8.54 to <strong>8.74</strong> (critical in healthcare).</li><li><strong>Conciseness:</strong> Jumped from 7.02 to <strong>7.56</strong>, eliminating the unnecessary “verbosity” common in generic LLMs.</li></ul><h3>2. The Latency and Cost Victory</h3><p>This is where the SLM (Small Language Model) shines against giants like Claude 4.6 Sonnet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2hEVj1zJXQ6ymzRg.png" /></figure><h3>Inference Conclusion</h3><p>The resulting model weighs just <strong>400MB in GGUF format</strong> (Q4_K_M quantization). This means it can run locally in <strong>Ollama</strong>, in a lightweight Docker container, or even on a Raspberry Pi, while maintaining a level of medical reasoning that competes with models 200 times larger.</p><h3>6. From Training to Reality: Local Inference with Ollama</h3><p>Having a trained model in a PyTorch adapter is great for experimentation, but production requires portability. This is where we transform our work into something any user can run with a single command.</p><h3>The Export Process (GGUF)</h3><p>Using Unsloth to convert the fine-tuned model to <strong>GGUF</strong> format. This format not only packages the model and adapter into a single file but also allows aggressive quantization without losing precision.</p><ul><li><strong>Chosen format:</strong> Q4_K_M (4-bit).</li><li><strong>Final weight:</strong> ~400 MB.</li></ul><h3>How to Integrate It in Ollama in 3 Steps</h3><p><strong>1. Create a </strong><strong>Modelfile:</strong> This file tells Ollama how the model should behave and what its &quot;personality&quot; is.</p><pre>FROM ./qwen-redi-sft.gguf<br>PARAMETER temperature 0.3<br>SYSTEM &quot;&quot;&quot;You are Redi, a medical assistant specialized in consultation preparation.<br>Your goal is to gather patient information using strictly the SBAR format.&quot;&quot;&quot;</pre><p><strong>2. Build the model:</strong></p><pre>ollama create redi-med -f Modelfile</pre><p><strong>3. Run it:</strong></p><pre>ollama run redi-med</pre><p><strong>Result:</strong> Inference with a latency of <strong>&lt;50ms per token on a standard CPU</strong>, with no dedicated GPU required.</p><h3>7. Final Thoughts: Quality &gt; Quantity</h3><p>This experiment shows that the era of “giant models for everything” is giving way to the era of <strong>specialized models</strong>.</p><h3>Key Takeaways</h3><ul><li><strong>The “Golden Dataset” is real:</strong> 300 examples curated by a “Teacher” model (Gemini 3.1 Pro) are worth more than 100,000 rows of raw logs.</li><li><strong>SLMs are the future of the Edge:</strong> A well-trained 500M parameter model can handle niche tasks with surprising effectiveness, eliminating API bills and privacy concerns.</li><li><strong>Distillation is accessible:</strong> Thanks to tools like <strong>Unsloth</strong>, the cost of training these models has dropped from thousands of dollars to just a few cents of electricity.</li></ul><h3>What’s Next?</h3><p>If you’re a developer, my recommendation is to stop seeing LLMs as a black-box API and start seeing them as infrastructure you can own. Follow us on our social media for more content like this.</p><h3>References and Further Reading</h3><ul><li><strong>Hu, J. J., et al. (2021).</strong> <em>LoRA: Low-Rank Adaptation of Large Language Models.</em></li><li><strong>Dettmers, T., et al. (2023).</strong> <em>QLoRA: Efficient Finetuning of Quantized LLMs.</em></li><li><strong>Hsieh, C. Y., et al. (2023).</strong> <em>Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data.</em></li><li><strong>Yang, A., et al. (2024).</strong> <em>Qwen2.5 Technical Report.</em></li><li><strong>OpenAI (2024).</strong> <em>HealthBench: A benchmark for evaluating LLMs in healthcare.</em></li></ul><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/distilling-intelligence-llm-training/">https://blog.xmartlabs.com/blog/distilling-intelligence-llm-training/</a> <em>on May 22, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ad049b6968e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing an AI Workshop That Engineers Actually Use]]></title>
            <link>https://xmartlabs.medium.com/designing-an-ai-workshop-that-engineers-actually-use-519cb7fb12b4?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/519cb7fb12b4</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai-adoption]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 15:55:56 GMT</pubDate>
            <atom:updated>2026-06-23T20:04:42.616Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hoFVYfIK7xFQDa0U.png" /></figure><p>We built a 6-module, 3-hour hands-on workshop to close the gap between AI tool adoption and actual feature utilization. It starts with a mindset shift, builds through context management, planning, and tooling, and ends with a verification loop grounded in a real failure story. Here’s why we chose this format over the alternatives, the logic behind every design decision, what we’d change, and a full outline you can steal.</p><p><em>This is Spoke 2 of our AI journey content series. If you haven’t read the data behind the problem, start with The Feature Utilization Gap.</em></p><h3>The Problem We Needed to Solve</h3><p>Our internal survey showed: 85% of our engineering team used Cursor, 62% used Plan Mode, 23% used MCPs, and 0% used Skills. We had near-universal tool adoption and near-zero use of advanced features. The gap wasn’t a knowledge deficit you could fix with a PDF. It was a compound problem: people didn’t know the features existed, didn’t understand when to use them, and lacked a mental model for integrating them into real work.</p><p>Meanwhile, the industry wasn’t waiting. New models are shipped every few weeks. MCPs went from niche protocol to essential infrastructure. Skills and custom commands launched. Claude Code added Plan Mode and background agents. Every week, the tools gained capabilities that our team wasn’t using. The velocity of change meant any training we built would face a shelf-life problem; it had to be repeatable and updatable, not a one-shot event.</p><p>We needed something that met three criteria:</p><ol><li><strong>Measurable:</strong> we had to know if it worked, not hope it worked</li><li><strong>Repeatable:</strong> new hires and new tools meant we’d run this again</li><li><strong>Depth-focused:</strong> the problem wasn’t awareness of AI, it was utilization of features people already had access to</li></ol><h3>Options We Considered</h3><p>We evaluated three approaches before committing to a design.</p><h3>Self-paced online course</h3><p>We designed a full version: 8 units, 8–12 hours, with quizzes and a capstone project. On paper, it covered everything. In practice, self-paced courses have a well-documented completion problem. Coursera’s own data shows completion rates between 5–15% for most MOOCs. Our team already had the tools installed and working- motivation wasn’t high enough to sustain 12 hours of async content. We built the full course materials anyway (they live in Notion as a reference), but we didn’t lead with them.</p><h3>External training</h3><p>Anthropic’s Skilljar courses and various community tutorials cover individual features well. The problem is general. External content teaches Plan Mode, MCPs, and Skills in isolation. It doesn’t teach how they compound in the context of a real project using your stack, conventions, and codebase patterns. Our team uses Next.js 14, TypeScript, Prisma, and SQLite. Generic training with Python examples and toy projects wouldn’t transfer to their Monday morning.</p><p>The other issue is control. When Anthropic ships a new feature or deprecates an old one, we can’t update someone else’s course. We needed content we owned.</p><h3>Internal hands-on workshop (chosen)</h3><p>A 3-hour, facilitator-led, hands-on workshop built around a real demo app in our stack. This is what we chose, for specific reasons:</p><ul><li><strong>Conceptual first, tools second.</strong> The workshop teaches how to think about AI-assisted work — mental models, context management, verification discipline — not just which buttons to press. This was a deliberate design choice: tools change every month, but the concepts remain the same. It also meant the workshop could serve the entire company, not just engineering. A PM doesn’t need to know Cursor shortcuts, but they absolutely need to understand context management and the “director, not coder” mindset. By keeping the content conceptual, we built a program that scales across roles.</li><li><strong>3 hours forces prioritization.</strong> We couldn’t cover everything, which meant we had to identify the highest-leverage features and teach only those. That constraint turned out to be a design advantage — it eliminated the “firehose” problem of dumping every feature on people at once.</li><li><strong>Hands-on means retention.</strong> Each module ends with an exercise on the same codebase. Participants don’t watch someone else code — they code. The exercise progression builds on itself, so by the end, they’ve extended a real app using every technique taught.</li><li><strong>Facilitator-led means questions get answered.</strong> The gap between “I read how Plan Mode works” and “I used Plan Mode on a real task and got stuck” is where most learning happens. A facilitator catches those moments.</li><li><strong>Internal means updatable.</strong> When the landscape changes — and it changes weekly — we update the modules and rerun them. The philosophy we built into the training materials says it directly: “This could change day by day, be encouraged to keep up the pace.”</li></ul><h3>The Design: 6 Modules + Bonus Track</h3><p>The workshop has 6 modules and a bonus track, all built around a single demo app called TaskFlow — a task management application in Next.js 14, TypeScript, Prisma, and SQLite. Participants start with a working CRUD app and progressively extend it through each module’s exercise. By the end, they’ve added priority fields, categories, due dates, Markdown descriptions, search, and an activity feed — all using AI-assisted techniques that build on each other.</p><p>Here’s the module sequence and the reasoning behind the order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sFk7_P7Oym1Ul75A.png" /></figure><h3>Module 1: The Mental Shift (20 min)</h3><p><strong>Why it goes first:</strong> If participants walk into the workshop thinking of AI as fancy autocomplete, nothing else sticks. The first module reframes their role: you’re no longer a coder, you’re a director. Your job is to ensure clarity of requirements, architectural decisions, quality control, and code review. The AI writes the code. You direct the work.</p><p>This module introduces the Prompt Formula — Context + Task + Constraints + Examples + Output Format — and immediately demonstrates the difference between a vague prompt (“add login”) and a specific one with architectural context, technology constraints, and output expectations. The exercise has participants add a priority field to TaskFlow tasks using a well-structured prompt rather than writing the code themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zhD4rygxb04A04d0.png" /></figure><p>We tried putting context management first in an early outline. It didn’t work. Without the mindset shift, participants treated context management as a technical trick rather than a fundamental change in how they work.</p><h3>Module 2: Context is Everything (30 min)</h3><p><strong>Why, before Plan Mode:</strong> Context is the substrate that makes everything else work. Plan Mode produces bad plans without good context. MCPs provide context from external sources. Skills encode context into reusable workflows. Teaching context management second means every subsequent module benefits from it.</p><p>This module covers context windows (the AI’s “working memory”), context rot (when irrelevant information accumulates and degrades output quality, symptoms include the AI repeating rejected suggestions, contradicting itself, or giving generic responses), strategic file inclusion (what to always include, sometimes include, and never include), and chunking (dividing large tasks into context-manageable pieces).</p><p>We teach 7 ways to provide context: the prompt itself, conversation history, project rules (.cursorrules / CLAUDE.md), attached files, Skills, MCPs, and agent memory. The exercise has participants practice structured context provision while adding categories to TaskFlow.</p><h3>Module 3: Plan Mode (30 min)</h3><p><strong>Why third:</strong> Plan Mode is the first feature where the new mental model pays off. You describe what you want, the AI proposes a plan, you review and edit the plan, and then the AI executes. It’s the “director” role in action. But it only works well if participants already understand how to provide context (Module 2) and have shifted their mental model (Module 1).</p><p>We teach when to use Plan Mode (for tasks longer than 30 minutes, multi-file changes, unfamiliar code, and architecture decisions) and, critically, how to read plans. Red flags in AI-generated plans include vague steps, unnecessary refactoring, and incorrect assumptions about the codebase. The exercise uses Plan Mode to add due dates with overdue highlighting to TaskFlow, a multi-file change that benefits from planning.</p><p>The stat we share: Plan Mode can reduce feature development time by 60% when used for the right tasks. We’re careful to scope that claim, “appropriate tasks” means complex, multi-step work, not every one-line change.</p><h3>Module 4: MCPs (30 min)</h3><p><strong>Why fourth:</strong> MCPs (Model Context Protocol) solve a problem participants now understand viscerally from Module 2; the AI’s knowledge is static, but libraries and frameworks update constantly. MCPs are plugins that give the AI access to live documentation, external tools, and real-time data. After a 10-minute break, this module introduces the MCP ecosystem: Context7 (live documentation), Figma, Supabase, GitHub, Jira/Linear, and Pencil.</p><p>The exercise has participants configure Context7 in their editor and use it to add Markdown descriptions to TaskFlow tasks, pulling live library docs instead of relying on the AI’s potentially outdated training data. We also cover security considerations: what data MCPs can access, permission models, and audit logging.</p><h3>Module 5: Skills and Commands (30 min)</h3><p><strong>Why fifth, not earlier:</strong> Skills are expert knowledge encoded into reusable workflows. They’re powerful, but they’re also the feature with 0% adoption in our survey. Teaching them too early, before participants understand context, planning, and MCPs, would make them feel abstract. By Module 5, participants have enough foundation to understand why encoding a workflow matters.</p><p>We introduce the Superpowers Framework: core skills (brainstorming, writing plans, executing plans, TDD, systematic debugging, verification, code review), lifecycle skills (git worktrees, parallel agents, finishing branches, subagent-driven development), and meta skills (using superpowers, writing skills).</p><h3>This module includes the research that influenced our design decisions:</h3><ul><li><strong>The AGENTS.md finding:</strong> Research on project configuration files showed that AI-generated AGENTS.md Files provide marginal benefit. Curated, human-written configuration files provide substantially better results. This told us we needed to teach people to write these files themselves rather than generate them.</li><li><strong>The Skills effectiveness finding:</strong> Similarly, curated Skills deliver measurable benefits, while AI-generated Skills yield negligible improvement. Quality of the encoded knowledge matters more than quantity.</li><li><strong>The Vercel study:</strong> Vercel’s research found that AGENTS.md-style project context outperformed Skills-based approaches because the model failed to invoke Skills correctly 56% of the time. This shaped our recommendation: invest heavily in project context files (CLAUDE.md, AGENTS.md), and use Skills selectively for well-defined, repeatable workflows.</li></ul><p>The exercise has participants use a commit skill and explore the skill discovery workflow in their TaskFlow project.</p><h3>Module 6: The Verification Loop (20 min)</h3><p><strong>Why last:</strong> Verification builds on everything. You need the mental model (Module 1) to understand why review matters. You need context management (Module 2) to understand what the AI might have missed. You need Plan Mode (Module 3) to understand how planning prevents issues upstream. Verification is the capstone that ties the workshop together.</p><p>This module opens with the mindset: “Think of AI as a junior developer, capable and fast, but without judgment.” Then we tell the story that makes it concrete.</p><p><strong>The Mathi Problem:</strong> One of our team members shipped a feature in half a day using AI. Then, I spent 3 days fixing the bugs it introduced. The feature was done fast. The code was not done well. This is the most common failure mode of AI-assisted development, and it’s entirely preventable. (We cover this incident in depth in The Mathi Problem: Why AI-Generated Features Ship Fast and Break Slow.)</p><p>The module teaches Prevention over Verification, the idea that the best verification is not needing to verify as much:</p><ul><li>CLAUDE.md /AGENTS.mdsetup (30 min investment, hours saved): Project context that prevents the AI from making wrong assumptions</li><li><strong>Skills setup</strong> (15 min investment, consistent workflows): Encoded knowledge that ensures repeatable quality</li><li><strong>Plan Mode discipline</strong> (5 min per task, avoids scope creep): Planning before execution catches bad approaches before code is written</li></ul><p>We also present sobering industry statistics: Veracode found that approximately 45% of AI-generated code contains security flaws, CodeRabbit measured 1.7x more defects in AI code than in human-written code, Jellyfish found that PRs are 18% larger with AI assistance, and Google’s DORA report showed measurable decreases in delivery stability with AI adoption. But this is the critical pivot: those statistics come from teams without proper configuration. The entire workshop is about proper configuration. The tools aren’t the problem. The absence of setup, process, and verification is the problem.</p><h3>Bonus Track: Choosing the Right Model (10 min)</h3><p>A brief closer on model tiering: not every task needs the smartest model. We teach a 3-tier framework (Flagship for planning and debugging, Mid-Range for daily coding, Fast for boilerplate) and the budget rule (20% / 60% / 20%). The guiding principle: “Plan with the best, execute with the rest.”</p><h3>Key Design Decisions</h3><p>Beyond the module sequence, several structural decisions shaped the workshop.</p><p><strong>“Director, not Coder” as the throughline.</strong> Every module reinforces this concept. Module 1 introduces it. Module 3 (Plan Mode) is in practice. Module 6 (Verification) is the director’s role in quality control. We chose a single concept to anchor the entire workshop because workshops that try to convey 15 concepts dilute them all.</p><p><strong>Exercise progression on a single codebase.</strong> Participants don’t start a new project in each module. They extend TaskFlow from CRUD to a full-featured app with priority, categories, due dates, markdown descriptions, search, and an activity feed. This creates compound learning: the category filter you added in Module 2 serves as the context for the Plan Mode exercise in Module 3. It also mirrors real work, where you’re always building on an existing codebase.</p><p><strong>The Prompt Formula as a concrete tool.</strong> We didn’t teach “write better prompts” as vague advice. We gave participants a formula: Context + Task + Constraints + Examples + Output Format. Every exercise uses it. By the end, it’s muscle memory, not theory.</p><p><strong>Research citations, not opinions.</strong> When we say curated Skills outperform AI-generated ones, we cite the research. When we say AGENTS.mdoutperforms Skills in certain contexts, we cite Vercel&#39;s study. When we present statistics on AI code quality, we cite the sources. This isn&#39;t a workshop built on vibes—it&#39;s built on evidence. That also means the workshop changes when the evidence changes.</p><p><strong>Stack-specific, not generic.</strong> TaskFlow is a Next.js 14 application built with TypeScript, Prisma, and SQLite. Our team works in this stack. The exercises use this stack. The CLAUDE.mdexamples reference this stack. This is a deliberate trade-off (see below), but it means every exercise is directly transferable to participants&#39; real projects the next day.</p><h3>Trade-offs and What We’d Change</h3><p>This is the section that separates an honest retrospective from a press release. Every design decision involved sacrificing something.</p><p><strong>3 hours is not enough time.</strong> We cut aggressively to fit the format. The full self-paced course is 8–12 hours long for a reason — there’s 4x more material than we could cover live. What got cut: deep dives into MCP security models, advanced Skills authoring, multi-agent workflows, git worktree strategies for parallel AI work, and any meaningful coverage of TDD with AI. Participants leave the workshop knowing these things exist, but not knowing how to use them. The Notion self-training materials cover what the workshop can’t, but we know self-paced completion rates are low. This is our biggest unsolved tension.</p><p><strong>One stack means limited universality.</strong> TaskFlow is a Next.js application. If your team works in Python, Go, or Rust, the exercises don’t transfer directly. The concepts transfer, context management, Plan Mode, verification, but the muscle memory doesn’t. To run this workshop for a different stack, you’d need to rebuild the demo app and all seven exercises. That’s 2–3 days of work. We chose depth in one stack over breadth across many, but it limits who can use the workshop without modification.</p><p><strong>Hands-on means slower coverage.</strong> Every module with an exercise takes 2–3 times longer than a lecture covering the same material. We could have covered 12 topics in a lecture-based format. We covered 6 with hands-on exercises. We believe the retention trade-off is worth it — people remember what they do, not what they hear — but it means the workshop’s breadth is narrow by design.</p><p><strong>No pair programming or collaborative exercises.</strong> Every exercise is individual. We considered pair programming formats (one person prompts, one reviews), but cut them for time. This means participants miss the collaborative review dynamic that’s central to real AI-assisted development. A future version should address this.</p><p><strong>The workshop doesn’t cover when NOT to use AI.</strong> We teach how to use AI features well. We don’t spend meaningful time identifying tasks where AI is the wrong tool — tasks requiring deep domain expertise, security-critical code paths that need manual review regardless, or creative architecture decisions where AI suggestions can anchor you to conventional approaches. We acknowledge this in Module 6, but don’t dedicate a module to it.</p><p>We haven’t run a formal facilitator retrospective yet, so we’re being transparent about what we know and what we’re still learning. The trade-offs listed above come from the design process, not from post-delivery data. As we run more cohorts — the workshop was mandatory for ~70 engineers and optional for the full company of ~100 people — we’ll have richer data on what did and didn’t land. We plan to publish a follow-up with those findings.</p><h3>Results</h3><p>We defined clear metrics with targets before running the workshop:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nieQJgbvA3UsyghQ.png" /></figure><p>We also defined success metrics at three time horizons:</p><ul><li><strong>Immediate (workshop completion):</strong> Every participant can write a structured prompt, has Context7 configured, and has used Plan Mode at least once</li><li><strong>Short-term (1 week):</strong> Plan Mode usage increases, at least 3 MCPs configured across the team</li><li><strong>Long-term (1 month):</strong> Skills usage above 0% (target 50%), reduced bug reports from AI-generated code</li></ul><p>We’re still collecting formal post-training data, but the early signals are encouraging. Claude Code adoption across the team is now ~91%. More telling than any single number: engineers now share skills, MCP configurations, and tips organically — in Slack channels, in PR comments, in team meetings. The conversation shifted from “should I use this?” to “here’s what I built with this.” The mandatory workshop reached ~70 engineers, with optional access for the full company of ~100 people.</p><p>We’ll publish formal before/after comparison data once we run the follow-up survey. For now, the strongest evidence is cultural: the team talks about these tools differently than they did six months ago.</p><p>The one-week challenge we set for every participant: “Use Plan Mode for every task longer than 30 minutes.” This creates a forcing function for the most impactful single behavior change from the workshop.</p><h3>Steal This: The Workshop Outline</h3><p>The workshop outline — all 6 modules plus the bonus track, with learning objectives, key concepts, exercise descriptions, and timing — is designed to be customized.</p><p>The outline is designed to be customized. The concepts are stack-agnostic even though our implementation uses Next.js. The template notes tell you what to swap out for different stacks, team sizes, and skill levels.</p><p>If you build on it, we’d genuinely like to hear what you changed and why. The training evolves, “try and change, don’t expect to build a setup that works for years”, and your adaptations could improve ours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Iv7YymIKQuymV5iR.png" /></figure><h3>Further Reading</h3><ul><li>The Feature Utilization Gap: What 85% AI Tool Adoption Really Looks Like, The data that led to building this workshop</li><li>The Mathi Problem: Why AI-Generated Features Ship Fast and Break Slow, The full story behind Module 6’s core anecdote</li><li>AI Tools Are Not Just for Developers: How We’re Expanding Beyond Engineering, What happened when we applied similar thinking to PMs, HR, and delivery</li><li>Anthropic Skilljar: Claude Code in Action, an external training resource we recommend as a supplement</li><li>Anthropic Prompt Engineering Guide, Foundation for the Prompt Formula, taught in Module 1</li></ul><p><em>This article is part of our AI journey series. Published April 2026. The tools, features, and recommendations described here reflect the landscape as of that date. AI tooling moves fast — verify current capabilities before building on our specific tool recommendations.</em></p><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/designing-ia-level-up-ai-workshop-team-adoption/"><em>https://blog.xmartlabs.com/blog/designing-ia-level-up-ai-workshop-team-adoption/</em></a><em> on May 13, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=519cb7fb12b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Thought 85% AI Adoption Meant Success. We Were Wrong.]]></title>
            <link>https://xmartlabs.medium.com/we-thought-85-ai-adoption-meant-success-we-were-wrong-68b48ab66815?source=rss-35da6de3ebbf------2</link>
            <guid isPermaLink="false">https://medium.com/p/68b48ab66815</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Xmartlabs]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 15:31:34 GMT</pubDate>
            <atom:updated>2026-06-23T20:06:09.512Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GSlt-4B9hllHvkry.jpeg" /></figure><p>This is the first article in a four-part series about how Xmartlabs went from organic AI tool adoption to a structured, cross-company program. The short version of the journey:</p><ul><li><strong>We thought we were ahead.</strong> 85% of our engineers adopted Cursor on their own, no mandate, no rollout plan.</li><li><strong>A survey told us we were wrong.</strong> Tool adoption was high. Feature utilization was near zero. The gap was structural, not individual.</li><li><strong>That gap forced a response.</strong> We created a dedicated AI Engineering group, built a 3-hour hands-on training program called IA Level Up, and extended it beyond engineering to PMs, HR, delivery, and pre-sales.</li></ul><p>This article covers the data — how we measured the gap, what we found, and how to run the same diagnostic on your own team. The rest of the series walks through what we did about it: the training program we designed, the most expensive AI debugging lesson we learned, and how we pushed these tools into non-technical roles.</p><h3>The Survey</h3><p>In early 2026, as part of forming our AI Engineering group, we ran an internal survey across our engineering team. The goal was straightforward: stop guessing how people use AI tools and start measuring it.</p><h3>What We Asked</h3><p>The survey covered six dimensions of AI-assisted development tooling:</p><ol><li><strong>Tool adoption</strong> — Do you use Cursor (or equivalent) in your daily work?</li><li><strong>Plan Mode</strong> — Do you use Plan Mode to scope work before execution?</li><li><strong>Rules files</strong> — Do you maintain .cursorrules, CLAUDE.md, or AGENTS.md files in your projects?</li><li><strong>MCPs (Model Context Protocol)</strong> — Do you use MCP integrations to connect AI tools to external services?</li><li><strong>Commands</strong> — Do you use custom commands or slash commands in your workflow?</li><li><strong>Skills</strong> — Do you use Skills (reusable, encoded expert workflows)?</li></ol><p>Each question targeted a specific capability layer — from basic tool presence to advanced workflow automation. We weren’t asking people to self-assess their skill level. We were asking binary usage questions: do you use this, yes or no.</p><h3>Methodology and Limitations</h3><p>We should be honest about what this survey is and isn’t.</p><p>This was an internal survey of our engineering team at Xmartlabs — a software consultancy with a culture of early technology adoption. The sample represents one company’s engineering organization at one point in time. We didn’t run a controlled study. We didn’t track usage telemetry. We asked people what they use, and people sometimes overestimate their own tool usage.</p><p>The survey also captures a snapshot. AI tooling moves fast enough that the specific features we measured may look different by the time you read this. The categories matter more than the specific tool names.</p><p>What makes this data useful isn’t statistical rigor — it’s that we asked about feature depth, not tool presence. Most internal surveys stop at “Do you use the tool?” We went further.</p><h3>The Data</h3><p>Here are the raw numbers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*yseDJ5NRXVxjMMm8.png" /></figure><p>Read that last line again. Zero percent. Not low adoption — zero.</p><h3>The Utilization Funnel</h3><p>The pattern becomes clearer when you visualize it as a funnel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ppTKvZ_hc6UBskSM.png" /></figure><p>Each layer represents increasing sophistication in how someone uses AI-assisted development:</p><ul><li><strong>Layer 1 — Tool presence (85%):</strong> The tool is installed and used for basic tasks. Autocomplete, inline suggestions, and code questions.</li><li><strong>Layer 2 — Guided execution (62%):</strong> The user structures their interaction. Plan Mode scopes work before execution. Rules files provide the AI with persistent context for the project.</li><li><strong>Layer 3 — External integration (23%):</strong> The tool connects to the outside world. MCPs pull in live documentation, project management data, design specs, and database schemas.</li><li><strong>Layer 4 — Custom automation (8%):</strong> The user has created reusable workflows. Commands encode common operations into repeatable shortcuts.</li><li><strong>Layer 5 — Encoded expertise (0%):</strong> Expert knowledge is packaged into Skills that can be invoked by name — complex, multi-step workflows that capture how a senior developer approaches specific problem types.</li></ul><p>The drop from Layer 2 to Layer 3 is where the funnel collapses. Going from 62% to 23% represents the boundary between “using AI as a better text editor” and “using AI as a connected development partner.”</p><h3>What the Numbers Mean</h3><p>At 85% tool adoption, we looked like an AI-forward engineering team. If you’d asked us whether we used AI tools, we’d have said yes with confidence. And we would have been telling the truth, but only a shallow version of it.</p><p>Most of our team used AI as a glorified autocomplete. Type a few characters, accept the suggestion, move on. That’s useful. It’s also about 20% of what these tools can do.</p><p>The features that compound — the ones that turn AI from a typing assistant into an architectural collaborator — sat unused. Plan Mode was used by 62% of the team. That sounds decent until you realize it’s where AI-assisted development transitions from reactive (the AI responds to what you type) to proactive (the AI helps you think before you build). The 38% who weren’t using it skipped the planning step entirely.</p><p>MCPs were worse. Only 23% connected their AI tools to external data sources, such as live documentation, project management systems, and design tools. Without MCPs, the AI operates on stale training data and whatever files are open in the editor.</p><p>And Skills — the ability to encode expert workflows into reusable, invocable patterns — had zero adoption. Not low. Zero.</p><h3>Introducing the Feature Utilization Gap</h3><p>We started calling this the <strong>feature utilization gap</strong>: the difference between the percentage of a team that has adopted a tool and the percentage that uses its advanced capabilities.</p><p>It’s a useful concept because it reframes the question of adoption. Most engineering leaders measure tool adoption as a binary: does the team use it or not? That metric is misleading. A team at 85% Cursor adoption and 0% Skills usage isn’t an AI-adopting team. It’s a team that adopted the easy parts and missed the leverage.</p><p>The feature utilization gap is not specific to AI tools, but AI tools make it especially visible because the capability gap between basic and advanced features is large. The difference between autocomplete and a properly configured Skill with MCP integrations and Plan Mode discipline is not incremental — it’s a different way of working.</p><p>We define the gap as:</p><blockquote><strong><em>Feature Utilization Gap = Tool Adoption Rate — Advanced Feature Adoption Rate</em></strong></blockquote><p>For us, on Skills: 85% — 0% = 85 percentage points. On MCPs: 85% — 23% = 62 percentage points.</p><p>A healthy team might have a 20–30-point gap (some lag in advanced adoption is natural). A gap of more than 50 points signals a structural problem — the team adopted the tool but missed the paradigm shift it enables.</p><p>Industry data confirms this isn’t unique to us. A Jellyfish study of 20 million pull requests found that only 44% of companies had even tried autonomous AI agents, and those agents accounted for less than 0.2% of merged PRs. Stack Overflow’s 2025 survey showed 80% of developers use AI tools, but only 52% say agents have affected how they work. The gap between “has the tool” and “uses the tool deeply” is a widespread industry pattern.</p><h3>Why This Happens</h3><p>The gap isn’t about laziness or resistance. Our team actively chose to adopt Cursor — they were enthusiastic. The problem is structural.</p><h3>Advanced features require setup</h3><p>Basic Cursor usage requires nothing beyond installation. Open the editor, start typing, accept suggestions. MCPs require configuration: setting up a server, configuring authentication, and defining which external tools the AI can access. Skills require authoring — you need to write the workflow, test it, and iterate on it. The activation energy for each layer increases significantly.</p><p>In our survey data, you can see this directly: the adoption rate drops in proportion to setup complexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*tisaw-aBFLML0ZuJ.png" /></figure><p>This isn’t a coincidence. Each step adds friction, and friction compounds.</p><h3>Advanced features require a mental model shift</h3><p>Using Plan Mode well means changing how you think about development. Instead of “let me write this function,” you think “let me describe what I need and review the AI’s implementation plan before any code is written.” That’s a different cognitive pattern. Without someone showing you why that shift matters and what it looks like in practice, Plan Mode feels like an unnecessary extra step.</p><p>The same applies to MCPs. Connecting your AI tool to Linear, Notion, or Context7 only makes sense once you understand that the AI’s effectiveness is directly proportional to the context it has access to. That’s not obvious if you’ve been treating the tool as autocomplete.</p><h3>Nobody teaches the advanced path</h3><p>AI tool vendors ship features and write documentation. But they don’t provide the connective tissue — the “here’s how Plan Mode, MCPs, and Skills work together to change your workflow” narrative. Setup guides are per-feature, not per-workflow. Individual engineers are expected to connect the dots on their own, and most don’t.</p><p>Organic adoption works for the easy layers because the value is self-evident. You install Cursor, it autocompletes, and you see the benefit immediately. Advanced features require someone to explain their value before they’re visible and to provide the setup path to make them accessible.</p><h3>What We Did About It</h3><p>The survey data was the catalyst for the creation of a dedicated AI Engineering group at Xmartlabs. The group’s first major output was <strong>IA Level Up</strong> — a structured training program designed to systematically close the feature utilization gap, building capabilities layer by layer in the order the funnel reveals them. Context management and Plan Mode before MCPs. MCPs before Skills. Verification is the capstone.</p><p>We set concrete targets based on the survey baseline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*3Sm6IDWy2P2wxX7N.png" /></figure><p>As of April 2026, we don’t have a formal post-training survey yet. What we do have is observable change: Claude Code adoption across the team is now at ~91%. Engineers share skills, MCP configurations, and workflow tips in Slack daily — something that didn’t happen before. The conversations shifted from “what is this tool?” to “here’s what I built with it this week.” The culture moved. We’ll run a formal follow-up survey to get hard numbers, but the qualitative signal is strong.</p><p>The full design of the IA Level Up program — what we chose, what we rejected, and why we sequenced the modules the way we did — is covered in Designing IA Level Up: A 3-Hour Workshop That Changed How Our Team Uses AI.</p><h3>How to Measure This in Your Team</h3><p>You don’t need our exact survey. You need the principle: measure feature depth, not tool presence.</p><p>Here’s a concrete approach you can run this week:</p><h3>Step 1: Map Your Tool’s Feature Layers</h3><p>Before you survey anyone, identify the capability layers for whatever AI tool your team uses. The layers should follow increasing sophistication:</p><ol><li><strong>Presence</strong> — Is the tool installed and used at all?</li><li><strong>Configuration</strong> — Has the user customized the tool for their workflow? (rules files, settings, preferences)</li><li><strong>Structured interaction</strong> — Does the user plan before executing? (Plan Mode, multi-step prompting)</li><li><strong>External integration</strong> — Is the tool connected to external data sources? (MCPs, plugins, API integrations)</li><li><strong>Custom automation</strong> — Has the user created reusable workflows? (commands, scripts, shortcuts)</li><li><strong>Encoded expertise</strong> — Are complex workflows packaged as invocable patterns? (Skills, custom agents)</li></ol><h3>Step 2: Survey for Binary Usage</h3><p>Ask yes/no questions for each layer. Don’t ask people to rate their proficiency — this introduces bias. Ask: “Do you use [feature] in your daily/weekly work?”</p><p>Keep the survey short. Six to ten questions. If it takes more than three minutes, completion rates drop.</p><h3>Step 3: Build Your Funnel</h3><p>Plot the adoption percentages by layer. The shape of your funnel tells you where the gap is:</p><ul><li><strong>Gradual decline</strong> (80% → 70% → 55% → 40% → 30% → 20%): Healthy. Natural lag in the adoption of advanced features; no structural problem.</li><li><strong>Cliff drop</strong> (85% → 62% → 62% → 23% → 8% → 0%): This was us. A sharp boundary where the team stops advancing. Everything below the cliff needs intervention.</li><li><strong>Flat low</strong> (85% → 20% → 15% → 12% → 10% → 5%): The team adopted the tool, but nothing else was done. Either the tool is being used as a simple replacement for an older tool, or the more advanced features don’t fit the team’s workflow.</li></ul><h3>Step 4: Identify the Cliff</h3><p>The most actionable insight is where the cliff is. For us, it was between Layer 2 (Rules, 62%) and Layer 3 (MCPs, 23%). That told us the intervention needed to focus on the transition from “configuring the tool” to “connecting the tool to external systems.” That’s exactly what our training program focuses on.</p><h3>Step 5: Intervene Structurally</h3><p>Once you know where the cliff is, you can design targeted interventions instead of generic “AI upskilling” programs. If the cliff is between Presence and Configuration, your team needs help with initial setup. If it’s between Structured Interaction and External Integration (like ours), they need hands-on walkthroughs of MCP configuration and use cases.</p><p>The key insight: don’t try to move people from Layer 1 to Layer 6 in one push. Move them one layer at a time. Our training follows this progression deliberately.</p><p>The survey approach described above — binary usage questions across capability layers — is something you can adapt and run with your team this week. The key is measuring depth, not presence.</p><h3>What’s Next in This Series</h3><p>Measuring the gap was the easy part. Closing it is where the real work starts. The next three articles cover what we did about it:</p><ul><li><strong>Designing IA Level Up: A 3-Hour Workshop That Changed How Our Team Uses AI</strong> — the training program we built to close the gap, module by module, with the logic behind every design decision and what we’d change.</li><li><strong>The Mathi Problem: Why AI-Generated Features Ship Fast and Break Slow</strong> — a real postmortem. One of our engineers shipped a feature in half a day with AI, then spent three days fixing the bugs it introduced. Here’s the prevention framework that emerged from it.</li><li><strong>AI Tools Are Not Just for Developers: How We’re Expanding Beyond Engineering</strong> — the same approach, applied to PMs, HR, delivery, and pre-sales, what worked, what didn’t, and the playbook we’re using.</li></ul><p>Subscribe or follow along to get the rest as they drop. If you’re running this diagnostic on your own team and want to compare notes, we’d like to hear what you find.</p><blockquote><em>The data reflects our team’s status as of early 2026. AI tooling evolves rapidly — the specific features and adoption rates described here are a snapshot, but the framework for measuring utilization depth applies regardless of which tools your team uses.</em></blockquote><p><em>(Originally published on the Xmartlabs blog at this link: </em><a href="https://blog.xmartlabs.com/blog/feature-utilization-gap-ai-tool-adoption/"><em>https://blog.xmartlabs.com/blog/feature-utilization-gap-ai-tool-adoption/</em></a><em> on April 28, 2026.)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68b48ab66815" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>