<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vishwa Jayawickrama on Medium]]></title>
        <description><![CDATA[Stories by Vishwa Jayawickrama on Medium]]></description>
        <link>https://medium.com/@pmvishwajayawickrama?source=rss-83bd980d9a31------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OrcaZISVKF40yX3o4GWewQ.jpeg</url>
            <title>Stories by Vishwa Jayawickrama on Medium</title>
            <link>https://medium.com/@pmvishwajayawickrama?source=rss-83bd980d9a31------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:08:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pmvishwajayawickrama/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agentic Workflow to Generate Visual Documentation.]]></title>
            <link>https://medium.com/@pmvishwajayawickrama/agentic-workflow-to-generate-visual-documentation-5219732d9a3d?source=rss-83bd980d9a31------2</link>
            <guid isPermaLink="false">https://medium.com/p/5219732d9a3d</guid>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[connector]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Vishwa Jayawickrama]]></dc:creator>
            <pubDate>Sun, 24 May 2026 19:40:48 GMT</pubDate>
            <atom:updated>2026-05-27T01:49:24.179Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wqAt8bpq99T3rDUR58HGVg.png" /></figure><p>The WSO2 Integrator is an open-source integration platform powered by Ballerina. It lets developers create enterprise integrations in a low-code environment using a block-style visual approach. It also provides connectors that make it easier to communicate with external services.</p><p>The problem was simple: the WSO2 Integrator had more than 400 connectors with little to no documentation.</p><p>If documentation only involved code examples, automation would have been straightforward. But in this case, users needed guidance on how to use the low-code mode inside WSO2 Integrator. That meant the documentation needed visual references and screenshots showing how to configure and use each connector.</p><p>Manually creating those screenshots for 400+ connectors was not feasible.</p><p>So our goal became:</p><blockquote><em>Given a connector or trigger name, automatically generate the complete documentation for that connector or trigger, including screenshots and setup instructions, and optionally publish it to the documentation site, all without human intervention in the middle.</em></blockquote><p>Humans could still review the generated documentation before publishing, but the entire generation pipeline itself needed to be automated.</p><h3>The Initial Idea</h3><p>Our first idea was fairly straightforward.</p><p>We wanted an AI agent to:</p><ol><li>Open WSO2 Integrator</li><li>Configure a connector</li><li>Use remote function calls from that connector</li><li>Capture screenshots during major steps</li><li>Use the execution logs and screenshots to generate documentation that users could follow themselves</li></ol><p>At first glance, this sounded simple enough.</p><p>But once we started designing the workflow, several problems appeared immediately:</p><ul><li>How can an AI agent use WSO2 Integrator?</li><li>How can we take screenshots automatically?</li><li>Can the whole workflow run headlessly?</li><li>How do we make the agent reliably navigate the UI?</li></ul><h3>Making WSO2 Integrator Agent-Friendly</h3><p>Most agent harnesses already provide tools like Bash, Grep, and WebFetch. But interacting with an application like WSO2 Integrator required more than standard CLI tools.</p><p>To give those capabilities to agents, we needed MCP-based tooling.</p><p>Since WSO2 Integrator is essentially a VS Code fork bundled with WSO2-specific extensions, we explored two approaches.</p><h3>Option 1 — Desktop-Level Visual Automation</h3><p>The first option was to use desktop-level MCP tools with vision models.</p><p>The idea was simple:</p><ul><li>Analyze the screen visually</li><li>Identify UI components</li><li>Click based on screen position</li></ul><p>This approach technically worked, but the results were unreliable.</p><p>The agent spent too much time analyzing pixels, and even small UI changes or screen-size differences confused the workflow. The screenshots and generated flows also weren’t consistent enough for production-quality documentation.</p><h3>Option 2 — Browser-Based Semantic Automation</h3><p>The second option was significantly more promising.</p><p>Instead of visually analyzing pixels, we used Playwright MCP to interact with UI components semantically through structured accessibility snapshots. This is similar to working with the browser’s DOM hierarchy in the sense that the agent sees a structured representation of the page instead of raw pixels, but Playwright MCP exposes that structure through the accessibility tree.</p><p>That meant the agent could:</p><ul><li>Inspect components structurally</li><li>Click elements by meaning instead of position</li><li>Navigate reliably</li><li>Run headlessly</li></ul><p>But there was one problem:</p><blockquote><em>WSO2 Integrator is a desktop application.</em></blockquote><p>So we needed a way to run VS Code in the browser.</p><h3>Running VS Code in the Browser</h3><p>This turned out to be possible because VS Code itself is built using TypeScript and web technologies. While the desktop version is packaged with Electron, VS Code can also run in browser-based environments such as vscode.dev because it is built on web technologies.</p><p>We experimented with several approaches: WSO2 Devant, GitHub Codespaces, and CodeServer.</p><p>We abandoned the first two because of authentication and CDN-related complications.</p><p>Eventually, we settled on <strong>CodeServer</strong>.</p><p>CodeServer allowed us to:</p><ul><li>Run VS Code entirely in the browser</li><li>Install the WSO2 Integrator extension</li><li>Execute everything headlessly</li><li>Use Playwright MCP reliably</li></ul><p>At that point, we had a workable automation environment.</p><h3>Switching from Vision to Playwright MCP</h3><p>We initially tested the workflow using vision-based agents.</p><p>While the idea sounded appealing, the practical results were poor:</p><ul><li>execution was slow</li><li>screen analysis was inconsistent</li><li>pixel-based reasoning was unreliable</li><li>UI scaling issues caused failures</li></ul><p>After those experiments, we fully shifted to browser-based automation.</p><p>The final setup used CodeServer running VS Code in the browser, the WSO2 Integrator extension installed, Playwright MCP for semantic interaction, and a headless execution pipeline.</p><p>This gave much more reliable results.</p><h3>Designing the Execution Prompt</h3><p>Even with browser automation working, agents still needed proper guidance.</p><p>Although WSO2 Integrator is fairly intuitive for developers, agents still struggled with connector-specific flows, integration patterns, required configuration steps, and documentation expectations.</p><p>To solve this, we introduced an <strong>execution prompt generation phase</strong>.</p><p>Instead of directly sending a task to the agent, we first generated a detailed execution prompt containing:</p><ul><li>connector-specific instructions</li><li>WSO2 Integrator navigation guidance</li><li>screenshot requirements</li><li>documentation structure expectations</li><li>mandatory sections for the final output</li></ul><p>The system <a href="https://github.com/ballerina-platform/ballerina-library/blob/main/example-doc-generator/modules/prompts/system_prompt.bal">prompt</a> in our source code defines how these execution prompts should be generated.</p><p>At runtime, we combine connector-specific goals, internal instructions, and documentation templates, and then use an LLM to generate the final execution prompt dynamically.</p><p>That generated prompt is then passed to the agent for execution.</p><p>We later extended this beyond connector examples to support <strong>triggers</strong>, which act as listeners that automatically start an integration workflow when an external event occurs (like a new GitHub issue or a Webhook alert). For these triggers, we can run a trigger-specific workflow by passing the trigger flag, while keeping the same overall automation model for navigation, screenshots, execution logs, and documentation generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksBlKkVraaYmUMNUfb31Kw.png" /></figure><p>This is the core execution loop. The workflow starts with the connector name and optional additional instructions, which are converted into an execution prompt. The AI agent uses that prompt to decide which browser actions to perform. Those actions go through Playwright MCP as tool calls to WSO2 Integrator running in CodeServer, while browser snapshots come back to the agent as structured context. The agent uses that context, along with captured visuals, to produce the final documentation.</p><h3>Choosing the Right Agent and Model</h3><p>The next major challenge was selecting an agent and model combination capable of reliably completing the workflow.</p><p>We primarily tested:</p><ul><li>Claude Code with Sonnet 4–6</li><li>Claude Code with Opus 4–5</li><li>Codex with GPT 5.3</li><li>Gemini CLI with Gemini 3 Pro</li></ul><p>After multiple runs, Claude Code consistently produced the best results.</p><p>Codex performed reasonably well, but:</p><ul><li>It took significantly longer</li><li>execution involved many more tool calls</li><li>Workflows sometimes became inefficient</li></ul><p>Gemini CLI with Gemini 3 Pro struggled the most and often failed to complete the full workflow.</p><p>One interesting observation was that Sonnet and Opus produced nearly identical results for this task. Because of that, we chose Sonnet 4–6 due to the lower operational cost.</p><h3>Orchestrating the Workflow with Ballerina</h3><p>We built the orchestration pipeline using Ballerina.</p><p>The pipeline handled:</p><ul><li>pre-flight verification</li><li>CodeServer startup</li><li>Claude Code invocation</li><li>LLM API calls</li><li>execution coordination</li></ul><p>Initially, we directly submitted execution prompts to Claude Code.</p><p>But this created another issue:</p><blockquote><em>The execution process behaved like a black box.</em></blockquote><p>We had no visibility into:</p><ul><li>intermediate execution state</li><li>tool usage progress</li><li>failure points</li></ul><p>To solve this, we switched to using the Claude Agent SDK (formerly Claude Code SDK).</p><p>This gave us:</p><ul><li>synchronous execution logs</li><li>better debugging visibility</li><li>execution monitoring</li><li>restart capabilities when failures occurred</li></ul><p>However, the SDK was only available for Python and JavaScript.</p><p>Since our orchestration pipeline was written in Ballerina, we created a lightweight Python server that exposed the SDK through REST APIs. The Ballerina pipeline communicated with this service during execution.</p><p>The same orchestration layer also supports switching between connector and trigger workflows based on the trigger flag.</p><p>The following recording shows the agent execution from the pipeline’s point of view. It captures how the agent moves through WSO2 Integrator in the browser, using Playwright MCP interactions while the orchestration layer keeps the execution observable.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdD-25TeiVZ8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdD-25TeiVZ8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdD-25TeiVZ8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c02c4aa9bf608acf55e2d3a4d37de1d/href">https://medium.com/media/9c02c4aa9bf608acf55e2d3a4d37de1d/href</a></iframe><h3>Post-Processing the Documentation</h3><p>Even after the documentation was generated successfully, the formatting was often inconsistent.</p><p>The structure usually drifted away from our expected template.</p><p>To enforce consistency, we added another post-processing stage where the generated documentation was sent through an LLM again with style guides, formatting instructions, strict templates, and documentation conventions.</p><p>Finally, we added:</p><ul><li>screenshot cropping</li><li>source-code snippets</li><li>“Open in WSO2 Integrator” buttons</li><li>final formatting cleanup</li></ul><p>before publishing the generated output.</p><p><strong>Using this pipeline, we have generated 150+ documentation pages for the </strong><a href="https://wso2.com/integration-platform/docs/connectors/catalog"><strong>WSO2 Integrator connector catalog</strong></a><strong>. The example pages in the connector catalog are generated through this workflow.</strong></p><p>The implementation, source code, and running instructions are available in the <a href="https://github.com/ballerina-platform/ballerina-library/tree/main/example-doc-generator">example-doc-generator</a> directory of the Ballerina library repository.</p><h3>Final Thoughts</h3><p>What initially looked like a straightforward documentation automation task became a fairly complex agentic workflow involving browser-based IDE automation, MCP tooling, prompt generation, orchestration pipelines, headless execution, and post-processing systems.</p><p>One of the biggest lessons from this project was that visual low-code environments introduce very different automation challenges compared to traditional code-generation workflows.</p><p>Vision-based approaches sounded promising initially, but semantic browser automation through Playwright MCP ended up being dramatically more reliable.</p><blockquote><strong>Another important takeaway was that the quality of the execution prompt mattered just as much as the model itself. Giving the agent proper context, navigation guidance, and documentation expectations significantly improved execution quality.</strong></blockquote><p>In the end, we managed to create a workflow capable of generating visual connector documentation with minimal human involvement, something that would have been extremely difficult to scale manually across hundreds of connectors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5219732d9a3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>