<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prachi Parmar on Medium]]></title>
        <description><![CDATA[Stories by Prachi Parmar on Medium]]></description>
        <link>https://medium.com/@prachiparmar04?source=rss-4fdba4900c3d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*A6joisSJcBJxxTYH</url>
            <title>Stories by Prachi Parmar on Medium</title>
            <link>https://medium.com/@prachiparmar04?source=rss-4fdba4900c3d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:59:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prachiparmar04/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Smart Auditor: Agentic AI for Enterprise Risk Management (ERM)]]></title>
            <link>https://medium.com/@prachiparmar04/smart-auditor-agentic-ai-for-enterprise-risk-management-erm-c34b0404b299?source=rss-4fdba4900c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c34b0404b299</guid>
            <category><![CDATA[audit]]></category>
            <category><![CDATA[smart-audit]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[audit-automation]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Prachi Parmar]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 03:45:32 GMT</pubDate>
            <atom:updated>2026-11-07T08:48:57.366Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How an agentic layer on IBM watsonx Orchestrate is automating evidence-based compliance audits at enterprise scale, what broke on the way to production, and where the architecture goes next.</em></p><p><strong>Co-authors: </strong>Buvana Iyer, Tannavi Snehal, Prachi Parmar, Aman Verma</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PO_Bx5y6LB6sjcdo0yy58A.png" /></figure><p>Ask anyone who has sat through an audit cycle at a service-based technology company and you will hear the same story. The audit is raised in a portal, someone emails the delivery manager, the delivery manager digs through SharePoint or Emails for evidence, the auditor reads the account contract documents to check what was promised, the evidence is validated and the result is typed back into the portal by hand. Multiply that by dozens of accounts, hundreds of projects and more than a hundred audit types, and audit season starts to feel like a second job.</p><p>This article explains why audits are still so manual, how we built an agentic “Smart Auditor” that automates the repetitive parts while keeping auditors in control, and what we learned taking it to production. It walks through one Audit end to end, shows the architecture running today, the challenges we hit at production scale, and the proposed end goal: a truly agentic architecture.</p><blockquote><em>The solution was build for a large service-based IT company that runs roughly 400 audit engagements a year across 100+ audit controls, with 40+ auditors involved. Current Manual Time Taken for a Audit is around ~15–30 Days.</em></blockquote><h3>The problem: a large audit universe, run by hand</h3><p>Service-based technology companies manage hundreds of client engagements and projects, each with its own contractual, regulatory and operational audit requirements. The applicable audits, which can exceed 100 types, vary by client, location, industry and project scope, and run at defined intervals or on demand. Which audits apply to a project is set by the client’s Contract Documents, Statements of Work, Annexures, Schedules, and others so scope has to be worked out Account by Account.</p><p>Most companies manage this in an Enterprise Risk Management (ERM) or GRC platform such as ServiceNow, SAP, Archer or IBM OpenPages. Audits typically fall into two categories:</p><p>• <strong>Physical audits</strong>, which require an auditor on site, for example asset management or workstation and device compliance.</p><p>• <strong>Evidence-based compliance audits</strong>, covering business continuity, network security, operational security, compliance, HR and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NStNsXrwX_yxWEpPTbx3cA.png" /><figcaption><em>.Manual Process : platforms, audit types, accounts, projects and the evidence sources behind each audit.</em></figcaption></figure><p>For every audit cycle, auditors follow a defined test procedure and collect evidence from many places: SharePoint, email, ERM records, contract repositories and other enterprise tools. Then they judge each audit themselves. <strong>This manual, repetitive process makes audit execution slow and hard to scale across hundreds of projects.</strong></p><h3>Auditor’s pain points: why the manual process hurts</h3><ul><li><strong>Time-consuming preparation</strong>: Auditors spend around 70% of their effort gathering context and validating evidence rather than auditing.</li><li><strong>Complex scope identification</strong>: Auditors manually review 200+ pages of contract documents to determine which of the 100+ controls apply to each project.</li><li><strong>Scattered evidence</strong>: Evidence is spread across SharePoint, emails, ERM records, contract repositories and monitoring tools.</li><li><strong>Manual evidence validation</strong>: Auditors manually review screenshots, scanned documents, PDFs and large spreadsheets, increasing the risk of human error.</li><li><strong>Inconsistent audit outcomes</strong>: Different interpretations of the same control procedures can lead to inconsistent results across 40+ auditors.</li></ul><h3>What auditors actually need</h3><p>Put together, these pain points describe what automation has to deliver, and what the Smart Auditor was designed around:</p><p>• Evidence gathered <strong>automatically</strong> from where it already lives, not chased by email.</p><p>• Massive Contracts Documents will be ingested in VectorDB, with the relevant clause or value pulled out defining which audits are applicable for the account.</p><p>• A verdict that <strong>explains itself</strong>, has an audit trail with links to the exact evidence and test step behind it. The results are always consistence based as the steps are standardized.</p><p>• Auditors kept <strong>in the loop</strong> for exceptions and judgment calls, not removed from the process completely.</p><h3>The Smart Auditor, Harnessing the Power of Agentic AI</h3><p>The idea is simple. When an auditor triggers a audit in ERM platform, an agent picks it up, works out what the audit needs (Test procedure of the audit), gathers the relevant evidence, validates the check and writes the result back &amp; auditors review the outcome.</p><ul><li><a href="https://www.ibm.com/products/watsonx-orchestrate?utm_source=chatgpt.com"><strong>IBM watsonx Orchestrate</strong></a>: Orchestrates a supervisor agent and specialist auditor agents, with each audit executed as a tool call.</li><li><strong>IBM watsonx.ai</strong>: Powers vision models to extract and interpret evidence from complex scanned documents and images.</li><li><strong>IBM watsonx.data</strong>: Uses a Milvus vector database to index contract documents and metadata for retrieval-augmented generation (RAG).</li></ul><h3>Solution Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aw3Lz6_fNuT1ZiK_90-ylg.png" /></figure><p>Six parts do the work, in the order an audit flows through them:</p><p>1. <strong>Trigger (ServiceNow).</strong> When an audit is raised, ServiceNow calls the Orchestrate agent with engagement_id, account_id, control_id and the control’s test procedure.</p><p>2. <strong>Supervisor agent (watsonx Orchestrate).</strong> Reads the control type and delegates it to the right collaborator. It never runs an audit itself.</p><p>3. <strong>Collaborator agents.</strong> The <strong>Physical Controls Agent</strong> handles on-site audits: it pulls the inspection report from ServiceNow and a <strong>Conclusion Agent</strong> drafts the write-up for the auditor. The <strong>Compliance Auditor Agent</strong> handles evidence-based audits by calling that control’s tool. The <strong>MSA Agent</strong> interprets contract clauses after RAG retrieves them, and any agent can call it.</p><p>4. <strong>Control tools (IBM Cloud Code Engine).</strong> One containerised async API per control. Tools pull evidence from ServiceNow, SharePoint, iCertis, Qualys, AlgoSec and Panorama, etc run OCR and business rules, and produce the verdict.</p><p>5. <strong>Knowledge and models.</strong> Contracts are indexed in Milvus on watsonx.data; watsonx.ai vision models read scanned pages and hard images.</p><p>6. <strong>Write-back.</strong> The verdict, reasoning and evidence links land on the ServiceNow control record for the auditor to review.</p><h3>Architectural Decisions That Made Agentic AI Audit Automation Production-Ready</h3><p>In audit work, “mostly right” isn’t good enough. A single missed procedure can turn into a compliance gap, a financial exposure, or a failed review. When we set out to bring agentic AI into the audit process, the client was clear about one non-negotiable: <strong>100% accuracy and consistent, repeatable results.</strong></p><p>That requirement shaped everything. Before writing a single line of code, we made three architectural decisions that would define how the system behaves in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kaGncFLBnYOxl8LANzaA6Q.png" /></figure><h4>Deterministic First, Agentic Next: One Dedicated Tool per Audit</h4><p>Our first decision was to build a <strong>separate tool for each audit type</strong>, each following a fixed, deterministic flow.</p><p>Instead of letting an LLM plan and execute audit steps on its own, we translated each audit’s existing manual procedure, step by step, into a dedicated tool. Every control the system checks maps directly to a step an auditor performs today, so the output can be verified against a process auditors already know.</p><p>The agentic layer acts as the <strong>orchestrator</strong>: it interprets the request, identifies the right agent to trigger, routes &amp; triggers it to the right tool.</p><h4>Why this was the right call</h4><blockquote>When you introduce agentic AI into a critical process, trust has to be earned in production, not assumed. So we chose a phased approach to increase the adoption of AI gradually in production.</blockquote><p>Audit is a high-risk process, and this was the organisation’s first production rollout of audit automation. An agent planning its own steps might skip a check or behave differently across runs, and in audit, that’s a finding waiting to happen. Deterministic tools give the same steps and the same output every time, with a clear trail when results are questioned.</p><p>So we chose a phased path:</p><ul><li><strong>Phase 1 (now):</strong> deterministic execution with agentic orchestration.</li><li><strong>Later phases:</strong> agents gradually take on planning and execution as the solution proves itself in production.</li></ul><h4>Right Engine, Right Image: Our Hybrid OCR Approach</h4><p>Much of our audit evidence is screenshots: device dashboards, configuration pages, review reports, emails and network diagrams. Before any control can be checked, key fields like device names, admin logins and timestamps must be extracted. Some images are simple to read; others are dense dashboards with small text and multiple panels.</p><h4>Why We Needed a Hybrid OCR</h4><p>The vision model available to us on watsonx.ai is <strong>mistral models</strong>. It’s capable, but routing every image through it isn’t the right fit. Much of our evidence is plain text on a screen, which open-source OCR can read just as accurately at a fraction of the cost. Spending LLM tokens on these images limits how far the inferencing budget can stretch.</p><p>The goal was simple: <strong>the right engine for the right evidence.</strong></p><ul><li><strong>Open-source OCR (RapidOCR, Docling)</strong> handles images where the task is reading text, such as configuration pages, tables and log panels.</li><li><strong>The vision LLM</strong> is reserved for cases that need real interpretation, such as understanding a network diagram or checking whether boxes are ticked in a form, and for any fields open-source OCR fails to extract.</li></ul><p>This keeps token usage low without compromising accuracy where it matters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XqLdAF_v7mM1nKO4mxJ8_g.png" /></figure><h4>Approach 1: open source first, vision as fallback</h4><p>Each image goes to RapidOCR first. If all key fields are extracted, the result returns directly. If not, only the missing fields go to the vision model.</p><p>This worked for simple evidence, but struggled with complex images. Reading a full dashboard in one pass, open-source OCR missed many fields, so around <strong>8 out of 10 calls</strong> still reached the LLM.</p><h4>Approach 2: crop first, then read</h4><p>Open-source OCR performs much better on small, focused images, and evidence from the same system follows a consistent layout. So for complex images, we:</p><ol><li><strong>Crop</strong> the image into fixed-template regions</li><li><strong>Run RapidOCR</strong> on each crop</li><li><strong>Send only crops still missing fields</strong> to the vision model</li><li><strong>Merge</strong> the results into one output</li></ol><h4>RAG Pipeline: Teaching Retrieval to Read Contracts Like an Auditor</h4><p>A single client engagement is rarely captured in one contract. It’s typically spread across:</p><ul><li><strong>Master Services Agreement (MSA):</strong> the overarching legal terms — there are multiple version of this document that is maintained.</li><li><strong>Statements of Work (SOWs):</strong> scope and deliverables for specific projects</li><li><strong>Schedules:</strong> Deep dive into some sections of the MSA like Scedule for Business Continuity, Asset Management, etc.</li><li><strong>Annexures:</strong> later additions or amendments to any of the above</li></ul><p>Many of these exist in multiple versions (v1, v2) and arrive as a mix of scanned and digital files.</p><h4>The problem</h4><p>To answer a single audit question, the system often needs to read across several of these documents at once. A clause in an SOW may say “as defined in Annexure 3.” A newer MSA may quietly override a condition in the older one: v1 requires customer approval, v2 doesn’t. Standard RAG retrieves whatever text looks most similar, with no sense of which version is current or what a clause depends on. In audit, citing an outdated or incomplete clause produces a confidently wrong finding.</p><h4>How the current pipeline works</h4><ol><li><strong>Extraction:</strong> Scanned pages go through <strong>watsonx.ai Vision</strong> to capture text, layout and tables. Digital files go straight to pre-processing.</li><li><strong>Pre-processing :</strong> Each document is tagged with its type, version, and other metadata information. Tables are rebuilt, headers and footers removed, content <strong>chunked by clause</strong> rather than fixed length, and cross-references detected and stored as metadata.</li><li><strong>Storage:</strong> Clause chunks are stored as vectors in <strong>Milvus on watsonx.data</strong>, each carrying its document type, version, date, page and references.</li></ol><h4>Where it still falls short</h4><p>The pipeline handles versions and references well, but a few complexities remain:</p><ul><li><strong>Oversized clauses:</strong> Clauses longer than the chunk size still get split, and since RAG returns only the most similar chunks, part of the clause can be lost.</li><li><strong>Scanned documents:</strong> Parsing quality varies from file to file, which affects everything downstream.</li><li><strong>Custom logic to maintain:</strong> Versioning and cross-referencing are handled by tooling we built ourselves, which adds maintenance effort as document types grow.</li><li><strong>Domain-heavy context:</strong> Contract language is dense and specialised, so the LLM doesn’t always interpret retrieved chunks correctly, even when retrieval is accurate.</li></ul><h3>Future Scope: from fixed flows to Orchestrate-planned audits</h3><p>Phase 1 works, but every audit still has its own audit-specific tool with its steps and business rules hard-coded. Adding a audit means building and testing a new tool. With trust and a baseline in place, the next phase changes who decides the steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-EIh9rDuh7SQecBHljE9Cg.png" /></figure><p><em>Figure 8. Today (Phase 1) versus the to-be architecture: Orchestrate plans its own steps.</em></p><p>In the to-be architecture, Orchestrate reads the control’s <strong>audit description</strong> (objective, test steps, expected evidence, pass and fail criteria) and runs a plan, execute, validate loop:</p><p>• <strong>Steps:</strong> no longer predefined per control; Orchestrate plans them from the audit description.</p><p>• <strong>Tools:</strong> audit-specific tools are replaced by light-weight, reusable utilities and parsers on ROSA, shared across controls.</p><p>• <strong>Execution:</strong> Orchestrate native workflows call those shared tools, instead of rules and heavy processing living in one tool per control.</p><p>• <strong>Validation:</strong> instead of fixed rule checks, Orchestrate decides how each step is checked and re-plans when it finds a gap.</p><blockquote>The heavy lifting stays on the cluster, so the production lessons above still hold. What moves into watsonx Orchestrate is the reasoning, enabling <strong>Orchestrate-driven autonomous audit automation</strong> to plan which steps to run and how to evaluate the results.</blockquote><h3>Case study: Compliance Audit (CO), Account Compliance Focal identification and communication</h3><p>This compliance audit verifies that each account has a designated Account Compliance Focal (ACF) / ITRO, with documented responsibilities, regular client communication, and applicable certification or training requirements assessed.</p><p><strong>Manual process:</strong> Auditors review Account Security Plan and Management(ASPM)/Incident Response Plan (IRP) documents on SharePoint to identify the ACF, cross-check the name against the organisation chart, verify client communication through business review invites, minutes or emails, and check the pre-assessment questionnaire for training or dedicated Data Protection Officer (DPO) requirements. They then document the verdict and observation in ServiceNow.</p><h3>Current implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*srCuZzdVliMuJLSsnW0bxg.png" /><figcaption><em>CO Audit today: the supervisor delegates to the ERM auditor agent, which calls a deterministic CO3 tool.</em></figcaption></figure><p>In production today, ServiceNow triggers the audit with the engagement, account and control IDs. The Orchestrate supervisor agent routes to <strong>ERM Auditor collaborator agent</strong>, which calls a deterministic CO audit tool. The tool runs the decision flow on the right of Figure 6:</p><p>• If the ACF / ITRO is not identified, or roles and responsibilities are not documented, the control is <strong>Ineffective</strong>.</p><p>• If there are fewer than two instances of ACF communication with the client, the control is <strong>Ineffective</strong>.</p><p>• If the questionnaire marks certification / training or a dedicated DPO as applicable, the result is <strong>In-Progress</strong>, with a note telling the auditor exactly what to verify by hand.</p><p>• If all mandatory checks pass and both conditional items are not applicable, the control is <strong>Effective</strong>.</p><p>The result, the evidence links and the auditor note go back through the agent to the supervisor, which writes them to the ServiceNow control record.</p><h3>Challenges and Lessons Learned on the Journey to Production ( WatsonX Orchestrate )</h3><p>The solution deterministic tools to the Orchestrate Agent worked well in isolation. Running dozens of controls across real engagements, with real file sizes and real deadlines, exposed limits that only show up at production scale. At a high level, we hit multiple product challenges in four areas:</p><p><strong>1. Large file handling</strong></p><ul><li><strong>Problem:</strong> Large Size file (eg. 240 MB Excel file) far exceeded watsonx Orchestrate’s 0.1 MB tool payload limit.</li><li><strong>Fix:</strong> tools hosted on containized platform now download and process files directly, returning only structured results and metadata to Orchestrate.</li></ul><p><strong>2. Multiple file formats</strong></p><ul><li><strong>Problem:</strong> Evidence arrives as Excel, CSV, PDFs, images and emails, each needing different handling.</li><li><strong>Fix:</strong> Custom tools normalise every format into a common structure before validation, moving provessing of files &amp; large size files to processing layer than orchestrate.</li></ul><p><strong>3. Tool timeouts from Orchestrate.</strong></p><ul><li><strong>Problem:</strong> Orchestrate’s 120-second synchronous window and limited compute couldn’t support long-running jobs.</li><li><strong>Fix:</strong> Heavy processing now runs as asynchronous Kubernetes workloads, independent of the Orchestrate session wherever required.</li></ul><p><strong>4. High model inference costs</strong></p><ul><li><strong>Problem:</strong> Sending all evidence through vision models would exhaust the inference budget quickly.</li><li><strong>Fix:</strong> Open-source OCR is now the primary path, with vision models used only when needed as explained in the hybrid OCR approach</li></ul><p><strong>5. Inconsistent LLM output across platforms</strong></p><ul><li><strong>Problem:</strong> The same model, given the same input, produced different extractions on watsonx Orchestrate and watsonx.ai.</li><li><strong>Fix:</strong> For Consistent results, we moved these tasks to Mistral Small on watsonx.ai for repeatable output.</li></ul><h3>Delivery lessons</h3><p>Not every challenge was technical. Delivery ran in three phases, with each audit going through understand, BRD, develop, UAT , Business User Testing and production, co-created with the client’s auditors. Four lessons stood out:</p><p>• <strong>Data readiness first.</strong> Partial datasets drove repeated clarification sessions. Get representative evidence for every control before building.</p><p>• <strong>Lock requirements early.</strong> Requirement documents went to second and third versions before sign-off; test procedures for a single control can change several times in a few weeks.</p><p>• <strong>Plan for UAT iterations.</strong> Logic, format and UI changes came up in UAT, and some controls were rebuilt.</p><p>• <strong>Right-size timelines.</strong> Compressed schedules meant long weeks for the team. Build buffer into the plan.</p><h3>What changes for the business</h3><p>The ERM platform has already digitised the audit record. The next step is to automate the audit work itself. An agentic layer delivers shorter audit cycles, consistent testing on every project, auditors focused on judgment, less disruption to delivery teams, and a path to continuous assurance, because once audits are automated they can run far more often than a yearly or quarterly calendar allows.</p><p>Audit will always need people. It does not need people to spend most of their time searching folders. Putting an agentic layer between the ERM platform and the evidence is a practical way to give that time back.</p><blockquote>This Project targeted &gt;50% Automation Level on an average for a audit.</blockquote><h3>Sources</h3><p>• <a href="https://www.ibm.com/products/watsonx-orchestrate">IBM watsonx Orchestrate product page</a></p><p>• <a href="https://www.techtarget.com/ai/news/366623635/IBM-targets-agentic-AI-orchestration">TechTarget: IBM targets agentic AI orchestration</a></p><p>• <a href="https://sprinto.com/blog/servicenow-alternatives/">Sprinto: ServiceNow alternatives for IRM and GRC</a></p><p>• <a href="https://www.cybersierra.co/blog/alternatives-to-servicenow-grc">Cyber Sierra: ServiceNow GRC alternatives</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c34b0404b299" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>