There’s a growing chorus in the tech world declaring the death of the Graphical User Interface. The argument goes something like this: AI agents are so capable now that we don’t need traditional applications anymore. Just talk to an agent, get a markdown file, a chat response, or a generated document — and you’re done.

I believe this thinking is dangerously backward.

Not because AI isn’t powerful — it absolutely is. But because the people making this argument are confusing flexibility in creation with consistency in operation. They’re solving for the first while completely ignoring the second.

A Doctor, a Patient, and a Markdown File

Let me make this concrete. A traditional Electronic Health Record system has structured forms, required fields, validation rules, consistent layouts, audit trails, and role-based access. Every patient record follows the same structure. A nurse in one clinic can read a record created by a doctor in another. Regulators can audit it. Insurance companies can process claims against it.

That’s not an accident of old technology. That’s a requirement of the domain.

Now imagine replacing that with an AI agent that generates a different markdown file for each patient interaction. The structure shifts depending on how the doctor phrased the prompt. Required fields get omitted because the model didn’t think to include them. Tuesday’s output looks nothing like Thursday’s. One doctor’s records are paragraphs of prose; another’s are bullet points.

Where is the governance? Where is the consistency? Where is the deterministic output that every downstream system — billing, compliance, analytics, continuity of care — depends on?

It’s gone. You traded it for the illusion of simplicity.

What a GUI Actually Enforces

The “GUI is dead” crowd tends to be individual power users — developers exploring codebases, researchers summarizing papers, writers drafting content. For them, free-form AI output is genuinely liberating. You don’t need determinism when you need creativity.

But enterprise software exists precisely because most business processes require determinism. A well-designed interface isn’t just pixels on a screen. It’s a governance system. Consider what it quietly enforces:

Data integrity. A dropdown with five options ensures one of five valid values gets stored. An AI might hallucinate a sixth. A required field on a form guarantees completeness. An AI agent might skip it entirely because the prompt didn’t mention it.

Workflow governance. A GUI enforces sequence — step 2 happens after step 1, approvals are collected before proceeding, certain roles see certain screens. An AI agent generating free-form output has no inherent concept of “you are not allowed to skip this step.”

Auditability and compliance. When a structured application records an action, you know exactly what happened, who did it, and when. When an AI generates a markdown summary, you have a blob of text with no structured metadata behind it. Try running a SOC 2 audit on that.

Consistency across users and time. If fifty people use a well-designed application, they all produce comparable, queryable, aggregatable data. If fifty people use an AI agent for the same task, you get fifty different interpretations.

Trainability at scale. A consistent interface can be taught, documented, and supported across a 500-person organization. “Talk to the AI and it’ll figure it out” is not a training plan.

Where the Narrative Breaks Down

The “replace the GUI” argument works in exactly one scenario: non-repeatable, single-user, low-stakes tasks. Exploratory analysis. Brainstorming. Ad-hoc research. Creative drafts. One-off queries.

The moment a task becomes repeatable, multi-user, regulated, or high-stakes, you need structure. You need defined fields, enforced workflows, role-based access, and predictable outputs. You need, in other words, an application — whether it’s a traditional GUI, a next-generation conversational interface, or something we haven’t invented yet.

The format matters less than the principle: a designed system with defined behavior will always beat raw AI output for operational work.

The Real Future: AI as Architect, Not as Output

Here’s what I think the intelligent path forward actually looks like: AI should design and build the systems, not replace them.

Instead of an AI agent that generates a different markdown file every time a doctor sees a patient, imagine an AI that designs the entire EHR application — the data models, the required fields, the validation rules, the workflows, the APIs, the UI components — and then generates the code for that system. The doctor then works inside a governed, consistent application that was architected by AI but operates with the determinism the domain demands.

The AI is the architect. The human operates within a building that has walls, doors, fire exits, and plumbing — not inside an AI-generated essay about what the building might look like.

This is the approach we’ve taken at Trillo AI. Our platform generates comprehensive software blueprints — specifications, data models, APIs, workflows, UI layouts — before a single line of code is written. The AI does the heavy lifting of design. But the output is a structured, deterministic application, not a different document every time.

The pitch is simple: every time you ask an AI agent to generate a report, you get a different report. When you use AI to build the application that generates reports, you get the same governed, auditable, consistent output every time — and the AI did the hard work of designing that system for you.

The Two Camps

The market is splitting, whether it realizes it or not:

Camp 1: “Let the AI generate everything on the fly.” This is the vibe coding, chat-as-interface, markdown-as-output world. It works beautifully for prototypes, personal projects, and demos. It falls apart at scale, in teams, and under governance requirements. It’s where a lot of excitement — and a lot of venture capital — is concentrated right now.

Camp 2: “AI designs the system; humans operate within a governed system.” This is harder to demo in a tweet. It’s less flashy. But it’s what actually works when you need fifty people using the same application, compliance officers auditing the output, and downstream systems consuming structured data.

I’m not anti-AI. I’m pro-AI in the right layer of the stack.

The GUI isn’t dead. It’s being reborn — designed by AI, built by AI, but operated by humans who need consistency, governance, and trust in the systems they use every day.

A Question Worth Asking

The next time someone tells you that AI agents will replace traditional applications, ask them one question:

“When fifty different people use your AI agent for the same task, do they all get the same structured, auditable, governed output — or do they get fifty different markdown files?”

The answer will tell you everything about whether they’re building for demos or for the real world.

I am Anil Sharma, CEO and Founder of Trillo AI, a Google Cloud Premier Partner company building a blueprint-first platform for AI-powered application generation. This post was developed through a conversation with Claude (Anthropic), which helped refine the arguments and structure. The core thesis — that replacing governed interfaces with free-form AI output sacrifices the determinism that enterprise software exists to provide — is the author’s.

The conventional process to create a blueprint with similar detail would take four months.

It consists of 44,000 lines of server-side code and 63,000 lines of UI code. Overall, the AI-generated code is 50%–60% of the production application code.

Our assessment is that the server side has 40%–50% stubbed code that needs to be completed. The incomplete part is algorithmic and related to external systems integration and knowledge graphs.

The blueprint surfaced the knowledge graph in the specification. By analyzing the artifacts using AI, we were able to deduce what the knowledge is and how to extract it from documents.

Our approach was to blueprint first. The blueprint was created using Gemini 3.0 and a human-in-the-loop interface.

Using blueprint artifacts, it took Claude Code 20 hours to generate the code autonomously.

The following diagram shows the process we followed.

Vibe-Coding vs Blueprint for Application Generation

The blueprint approach works for generating a large body of code with a textbook structure. It doesn’t mean that vibe-coding is dead or some such claim. We will later see that vibe-coding works well once structure is available.

Could we have built an application of such complexity using vibe-coding? Definitely not. A developer can’t carry the mental map of a complex application in their head and translate it into code using Claude Code.

By analogy, you can’t build a thousand-room hotel one room at a time. Add fifty amenities, optimal use of the land, and city building code to the mix — it becomes a problem you can’t solve by prompting AI (vibe-code style). You need a blueprint of the application.

Similarly, you can’t build a large enterprise-class application using the bottom-up approach of vibe-coding — prompting AI to generate code incrementally, one feature at a time, without an upfront architectural design. You need a blueprint of the application.

There is a second problem with vibe-coding: it is developer-centric. It does not involve business experts, analysts, product managers, architects, and other roles in the process.

The following diagram says it all about vibe-coding vs. blueprint.

Vibe-coding can’t produce either normative or canonical output at scale, because there’s no single authoritative specification governing the generation.

The interesting finding is that vibe-coding works well once you generate the structure of the code using a blueprint. Blueprint does not eliminate AI coding tools. It elevates them by grounding them in a system-level design.

The rest of the blog answers the following questions:

• Is it possible to generate a blueprint of a large application using AI?
• Is the process repeatable?
• Can the blueprint be used to generate code at scale using tools like Claude Code?
• Will the blueprint and generated code be accurate — or will it produce plausible-looking output that is fundamentally wrong?
• How will different roles participate in and verify the output?

You will find that the answers to the above questions are as follows:

• AI can generate a structured blueprint by orchestrating several smaller steps in a logical order.
• The output is accurate if human actors review and refine it by conversing with AI.
• Human participation leads to convergence.
• That blueprint can drive deterministic, large-scale code generation.
• The resulting system is more coherent, auditable, and understandable across roles.

What is a blueprint of a software system?

A software blueprint is a collection of structured documents that collectively provide a normative definition of a software system. Externally, to a user, these are presented as readable documents, including visuals, to understand and comprehend. Internally, they may be modeled using JSON/XML schemas, RDFs, or knowledge graphs.

Blueprint-driven development is a superset of spec-driven development. Rather than focusing on individual specifications, a blueprint unifies them — connecting object models, APIs, workflows, and UI components into a single, cohesive architecture.

An AI-driven blueprint should fulfill the following requirements.

1. AI should generate it from a requirements document, with a human-in-the-loop interface that operates using natural language.
2. Internally, it should adhere to a formal schema so an AI system can consume it for code generation.
3. Externally, it should be easy to understand so people in different roles can review and validate it.
4. Together with human-in-the-loop input, an orchestrator should converge on a desired blueprint.

List of blueprint artifacts

The following diagram shows a list of AI artifacts. They resemble steps of SDLC (Software Development Life Cycle).

Requirements: This is input to the workflow.

Requirements Analysis: This agent analyzes requirements to determine gaps using AI. It prompts the user with questions, providing one or more possible answers. The user can edit or select the right answer.

Solution Discovery: This agent proposes solution options based on factors such as cloud services, deployment models, and available out-of-the-box services. The user selects the preferred approach.

Software Specification: This agent generates functional and non-functional requirements of the application. It also generates technical specifications for various aspects of the application — object model, user interface scenarios, microservices, workflows, external integration, MCP tools, AI agents, deployment specifications, and cloud services mapping.

Architecture: This agent prepares an architecture diagram of the application with a short description of each component.

The remaining agents generate detailed designs for each layer of the application.

In the past, a comprehensive specification of artifacts was time-consuming. Therefore software development teams rarely prepared a normative definition of artifacts and often relied on agile methodologies to refine them throughout the development process.

AI can generate a comprehensive definition of artifacts in minutes and validate them by cross-checking. Due to the enormous speed gain, iterations of agile methodologies take on a different dimension.

How AI generates blueprint artifacts

Each artifact is generated by one or more AI agents. AI agents produce a normative specification and are deterministic.

A workflow orchestrates AI agents in the order shown in the diagram. The input to each AI agent depends on the output of one or more previous agents.

The input to the workflow is a business requirement document.

Let us examine a few agents — how they work.

The first AI agent is the “Requirements Analysis” agent. It analyzes the requirements and identifies gaps. It iteratively interviews the user presenting gaps and their possible resolutions. It repeats the process until no more gaps exist.

The second agent takes the refined requirements from the ‘Requirements Analysis’ step and proposes solution options. Each option is based on factors such as cloud service choices, hybrid on-premise and cloud deployment, or available out-of-the-box services. The user can guide AI in a particular direction or propose solutions based on different criteria. Finally, the user selects one of the options.

The ‘Software Specification’ agent takes requirements and the user-selected solution approach as input. It generates a functional, non-functional, and technical specification from them.

‘Software Specification’ leads to an architecture diagram and so on.

The agents for access control rules, object model, UI, APIs of microservices, workflows, external interfaces, MCP tools, and AI agents generate detailed designs for artifacts. Each of these agents is designed to work with minimal context. Some agents perform validation and cross-validation.

The process described above has the following characteristics:

1. Gradually refine the output of each previous step to arrive at detailed specifications and design.
2. The AI does the heavy lifting — generating output, validating, and proposing solutions for gaps.
3. The user has control over the direction of artifacts at each step.

Based on our experience of MRO, the above process produces a detailed blueprint of an application. It is highly accurate (>90% based on tracing to requirements). Based on user input, agents were able to produce accurate results in one to three iterations.

Requirements Analysis and Software Specification are two important steps where all roles (domain experts, PMs, architects, etc.) can participate. A solution architect participates in all activities. At the design step, architects, senior software engineers, and UI/UX experts participate in the process.

The output of artifacts was detailed and structured enough that we were able to orchestrate large-scale code generation using Claude Code. It is covered in more detail below.

To summarize our findings, an accurate blueprint of a large application can be generated if a workflow follows these steps and rules.

1. Starting from requirements, it should proceed in steps towards a detailed blueprint.
2. Human-in-the-loop is critical for accuracy and governance. Human interaction should be simple and friendly.
3. AI output should follow a structure so it can be guided using the best principles of software design.
4. Workflow orchestration and agent design should ensure cohesive output. It will require cross-checking outputs of different artifacts.
5. You can generate a detailed technical specification and architecture of an application from requirements in less than five minutes. Review, refine, and iterate over it a few times. This process is many times faster than doing it manually. The coverage and accuracy are higher than in manual processes.

Holistic analysis of application

A blueprint serves as a detailed knowledge base of the application. Starting from the requirements, AI expands it in a stepwise manner. The steps are logically ordered so they can evolve in a cohesive manner. A user can analyze the output of each step and update it using a conversational, chat-like interface.

Once the knowledge base of the entire application is available, users can analyze it to gain insight into the application. Based on analysis, users can regenerate the blueprint or develop a future roadmap. They can also use it to generate user documentation, marketing materials, whitepapers, etc.

Generating code stepwise

We were able to generate a detailed blueprint that served as the foundation for generating large-scale application code (>100,000 lines). We used Claude Code through a carefully crafted orchestrator. You can think of our orchestrator as an outer loop of the Claude Code loop.

The orchestrator mimics how developers work. Like a development team, it generates code in layers starting with the bottommost layer. For example, it generates object model code first, API code next, and so on. It executes validation and testing loops at every step. After the server code is generated, it tests it using curl and other triggers. The orchestrator follows similar steps to generate UI code. Finally, it does UI and server integration testing.

If you examine the entire process, we generate structured blueprints using stepwise refinement. Using complementary steps and Opus 4.6 through Claude Code, we then generate application code.

Role of Vibe-coding

Once code was generated, the server-side code worked well 98% of the time. But the UI had 10%–20% broken links or API calls. We used vibe-coding to fix such issues. It took us about a week.

Once the structure of the code was in place, we were able to extend modules using vibe-coding.

Here is an example. A developer encountered a duplicate key error, described it in a prompt, and Claude Code fixed it by analyzing the generated code structure.

Issue: Duplicate React Key in Threshold Tuning Form

Error:

Warning: Encountered two children with the same key, `2`. 
Keys should be unique so that
components maintain their identity across updates. 
Non-unique keys may cause children to
be duplicated and/or omitted.
  at select (<anonymous>)
  at ThresholdTuningForm (threshold-tuning-form.tsx:60:3)

Vibe-Code Prompt:

The following error is coming under 
Reliability Engineer -> Hypothesis Testing -> clicking on Threshold Tuning.

Fix:

The `modelOptions` array merges `predictiveModels` and `cvModels` 
using `String(m.id)` as the key. 
Both model types use auto-incrementing database IDs, 
so a PredictiveModel and a ComputerVisionModel can share the same 
numeric ID (e.g., both have `id: 2`), producing duplicate keys.

Prefixed each ID with its model class to make keys unique:

```tsx
// Before — duplicate keys when both tables have id=2
const modelOptions = [
  ...predictiveModels.map((m) => ({ id: String(m.id), ... })),
  ...cvModels.map((m) => ({ id: String(m.id), ... })),
];

// After — unique keys with model class prefix
const modelOptions = [
  ...predictiveModels.map((m) => ({ id: `pm-${m.id}`, ... })),
  ...cvModels.map((m) => ({ id: `cv-${m.id}`, ... })),
];
```

Updated the mutation functions to strip the prefix before 
sending to the API:

```tsx
const numericId = parseInt(data.modelId.replace(/^(pm|cv)-/, ''), 10);
```

**Files Changed:**
- `src/components/containers/hypothesis-testing-container.tsx`

Well-structured code makes vibe-coding more effective.

Conclusion

MRO is a real customer application. Once we generated an architectural blueprint in an hour, it created excitement. Someone said, “Beautiful — this is what we imagined a system based on AI agents should look like.” Within a week, we had a POC deployed on the cloud.

The reason we mention this — the blueprint approach is not merely a research exercise. It works.

The real breakthrough isn’t that AI writes code — it’s that AI can manufacture structure. That makes App Factories possible. Once structure exists, code generation becomes predictable, scalable, and governable.

We believe that the future will see many such factories. Some of them will be tailored to a particular domain. Human-in-the-loop will be very important. Factories will provide interfaces that make it easy for humans to interact with AI and comprehend its output.

It may make you anxious or excited.

When the wheel was invented, the powerful men of the community, pulling rocks uphill, were concerned. Soon they were wheeling more rocks and building castles.

We Built a 100K-Line Enterprise App Using AI — Here’s Why Vibe-Coding Couldn’t was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this blog, you’ll learn how to produce high-quality specifications and architecture documents using AI — generating the first draft in minutes, then refining it into a clear blueprint of any application, however complex.

In the next blog, we go further: generating server and UI code directly from these artifacts using Claude Code. Not scaffolding. Production-grade code built on best practices — as good as writing it yourself.

The Problem

You have software requirements in informal or formal form — emails, meeting transcripts, notes, partial documents. Historically, you take one of two paths:

Path 1: The Blank Page Trap

You start with a clean slate and wrestle with blank page syndrome. You spend weeks researching, discovering, and producing a draft specification and architecture.

Path 2: Jump Straight to Code

Based on past experience, you visualize the application in your mind and skip specification work. You jump straight to prototyping — and you start down a winding road where technical debt accumulates from day one.

How AI Changes This

With AI, you generate a software specification and architecture in minutes. You review, analyze, and refine by conversing with AI over a day or two. At the end, you have genuine comprehension of the system — not a vague sketch, but technical understanding you can articulate and defend.

The cost: under $50 for the first draft plus 2–3 revisions using a tool like Trillo AI used for demonstration in this blog.

Even if you already use modern IDEs with AI code generation, a solid specification and architecture makes you dramatically more effective. You work at the right level of abstraction. You stop solving the wrong problems.

Light Bulb Moment

Here’s what this looks like in practice. Someone handed me a complex set of requirements. Instead of spending hours reading and parsing it, I put it through Trillo AI. Minutes later, I had the architecture diagram below and a structured specification. I started developing a good understanding of the system instantly. After that, reading the original document and understanding user intent became straightforward.

What Rest of This Blog Covers

As discussed in AI & the Future of Software Engineering, developing with AI will be a critical skill. This blog shows you how in tutorial style. The steps marked with (*) are interesting.

1. Collecting Requirements
2. Ingesting Requirements
3. Generating Software Specifications Draft
4. Reviewing Software Specifications by Conversing with AI (*)
5. Generating Architecture (*)
6. Cloud Services Costs
7. Architecture Diagrams for GCP, AWS, Azure

Collecting Requirements

We’ll use Trillo AI, a tool with features that let us explore AI’s capabilities discussed in this blog. Trillo AI accepts input as a PDF file or through its built-in editor.

If you have formal requirements, export them as a PDF file. Otherwise, cut and paste informal requirements from multiple sources — emails, call transcripts, notes — directly into the Trillo AI editor. Or consolidate them into a single PDF.

No need to format or reorder. Let Trillo AI do the heavy lifting.

Ingesting Requirements

Once your requirements are ready, follow these steps:

1. Sign up or log in at trillo.ai
2. From the application dashboard, create a new application by giving it a name and selecting your target cloud (AWS, Azure, or GCP)
3. Click on the application title to begin ingesting your requirements
4. You’ll land on the following screen
5. Drag and drop your PDF, or enter requirements manually
6. For this blog, we used a Hospital Management System requirements document. You can view it by clicking on the link.
7. Once your file uploads, the system will prompt you to proceed with processing. Confirm.

Within seconds, Trillo AI extracts your requirements and prepares them for the next step.

Generating Software Specifications Draft

Once your document is ingested, Trillo AI takes over. You direct the process by confirming its actions or by conversing with it.

You’ll see the following screen. Click “Ask AI to Generate Software Specification” and confirm the default values in the dialog box.

Within a minute, Trillo AI generates a structured specification organized by functional requirements, business entities, user interface, backend workflows, and more.

Reviewing Software Specifications by Conversing with AI

This is the most important step for human-in-the-loop review.

Once the software specification is generated, you’ll land on the screen shown below. Open the conversation sidebar by clicking the Chat icon. Notice the buttons to add, accept, delete, and edit requirements and issues. You can toggle the project phase — MVP, POC, GA — as needed. All changes remain temporary and can be undone until you click “Merge All Changes.”

Using the sidebar chat, you can converse with AI to analyze the specification. For example:

• “What are the most important PHI and PII attributes in the system that I need to protect? List them in two categories: most critical and major.”
• “Why is GKE needed for deployment? How does it help?”

You can also use the chat to update the document. For example:

• Ask: “What are the compliance requirements to deploy this application in Europe?”
• The system responds with a detailed answer.
• Follow up: “Add these requirements to the software specifications.”
• The system updates your document accordingly.

Trillo AI maintains version history, so you can always go back to a prior version and restore it.

If you diligently review and refine the specification at this stage, every subsequent step — architecture, cost estimation, code generation — will work with higher accuracy.

Generating Architecture

Similar to the software specification, you can generate the application architecture with a single click, as shown below.

Use the sidebar to analyze the architecture. For example:

• “My application UI (for patient, doctor, and admin portals) consists of a React bundle. Can I use Firebase Hosting?”

To update the architecture, regenerate it with comments. In the regeneration dialog, provide instructions like:

• “Regenerate architecture by replacing Cloud Run with Firebase Hosting for Patient, Doctor, and Admin portals.”

The system generates a new diagram and updates the description to match. Previous versions are preserved in the version history.

Cloud Services Costs

Similar to the software specification and architecture, cloud service costs can be generated using AI.

The estimate breaks down costs by service category — compute, database, networking, security, monitoring, and more — giving you a realistic view of operational expenses before you write a line of code.

The figures are approximate and may not be perfectly accurate, but they provide a useful indication of scale and a solid starting point for budget planning.

Architecture Diagrams for GCP, AWS, Azure

Trillo AI generates architecture for your target cloud. To demonstrate portability, here is the same Hospital Management System designed for all three major cloud providers.

Notice how the core layers — Presentation, API, Application Services, Data — remain consistent across clouds, while the specific services adapt to each platform’s strengths:

• GCP: Firebase Hosting for static content, Cloud Run for containerized services, Cloud SQL and Memorystore for data
• AWS: S3 and CloudFront for content delivery, ECS for container orchestration, RDS and DynamoDB for data, Step Functions for workflows
• Azure: App Service for both hosting and microservices, API Management for gateway, SQL Database and Cosmos DB for relational and NoSQL needs

The architectural decisions — layered design, managed services, separation of concerns — transfer across clouds. Only the implementation details change.

Architecture Diagram for GCP

Architecture Diagram for AWS

Architecture Diagram for Azure

Conclusion

In this blog, you went from informal requirements to a complete cloud architecture — specification, diagrams, cost estimates — in a fraction of the time it would traditionally take.

The key insight isn’t just speed. It’s that starting from an AI-generated draft fundamentally changes how you work. You’re no longer staring at a blank page, struggling to structure your thoughts. You’re reviewing, questioning, refining. The system becomes clear faster because you have something concrete to react to.

For under $50, you get a structured specification, a defensible architecture, and a realistic cost estimate — artifacts that would normally take weeks and cost thousands in consulting time.

In the next blog, we take these artifacts further: generating production-grade server and UI code directly from the specification and architecture. Not scaffolding — working code built on best practices.

If you’re an architect, a developer, a product manager, or an engineering lead, you’ve probably felt this. That mix of excitement and dread when you watch AI generate in seconds what used to take you hours. The quiet worry that maybe you’re witnessing your own replacement.

It is a valid fear. The speed at which AI operates can feel overwhelming. As a software company, we are not immune to this uncertainty; we have faced the same dilemma and asked the same hard questions.

So let me give you a direct answer: No. AI will not make you obsolete. But it will change how you work — profoundly. Those who adopt AI-powered software engineering will innovate faster, deliver higher quality, and explore domains beyond their traditional expertise.

To understand this change, we must look at history:

When the wheel was invented, the powerful men of the community, pulling rocks uphill, were concerned. Soon they were wheeling more rocks and building castles.

We are currently at that “wheel” moment.

What Does This Change Look Like?

Let’s move from theory to reality. Consider the architecture diagram below.

This is the complete, HIPAA-compliant Cloud Architecture generated from a single sentence: “I want to build a hospital management application that is HIPAA compliant”

In a traditional workflow, reaching this level of clarity — defining the Presentation Layer, API Layer, and Application Services Layer would take a team of experts days or weeks of white boarding. With AI, you can generate this baseline in seconds.

This delivers immediate gratification. It eliminates the intimidation of the blank page and lets you start editing immediately.

But There’s a Catch

The power of AI comes with a dangerous caveat. That architecture diagram above looks perfect, but if generated without rigor, it might just be a pretty hallucination.

If you feed an AI a massive, undefined request, like “build me a hospital app”, the context becomes too vast. The AI loses the plot. Instead of converging on a workable solution, it diverges, roaming in a zigzag direction.

Think of it this way: It is the difference between asking an architect to “design a building” versus asking them to “draft the foyer dimensions based on this specific occupancy code.” AI fails at the former (the broad, vague request) but excels at the latter (the specific, constrained task).

To turn that “pretty diagram” into working, reliable software, we cannot just chat with the bot. We need a process.

The Solution: The AI-Powered SDLC

Traditional SDLC failed due to a lack of rigor simply because maintaining that rigor was too expensive, time-consuming, and often humanly impossible. We must apply the rigor of a Software Development Life Cycle (SDLC), but one reimagined for the AI age.

To make AI effective, we must break the problem space into small, managed steps orchestrated by an SDLC workflow. This workflow moves sequentially through distinct phases to lock in the context and prevent drift:

• Requirements: Integrating ideas and user stories.
• Specifications: Defining user scenarios and architecture.
• Design: Creating data models and API definitions.
• Code Generation: Artifacts produced by AI-Powered SDLC platform (such as Trillo AI) and Claude Code can generate working code of an application. The blog Architecting Cloud Applications with AI covers this topic in detail.

By breaking the project down, we ensure the AI is never “guessing” about the whole world. It is only solving one specific puzzle at a time.

The rest of this blog walks through a few phases. But first, let’s address the most important question: Where do you fit in this process?

You Are the Architect, Not the Passenger

Automation does not mean losing control. In this model, AI is the engine but you are the Chief Architect.

Your role is to guide the AI toward convergence: reviewing outputs, catching blind spots, enforcing constraints the AI doesn’t know about. The tool’s job is to make that guidance easy, visual, and intuitive.

We call this the Human-in-the-Loop workflow. Let’s see it in action.

SDLC Step 1 — Generating the Software Specification with Human-in-the-Loop

The SDLC begins with Requirements. These can range from a one-liner to a hundred-page RFP. Your first goal is to convert that raw input into a structured Software Specification.

Here’s how the Human-in-the-Loop workflow operates at this stage:

1. Input & Generation You provide the requirements, and the AI generates a baseline draft. This isn’t just a summary, it’s a comprehensive specification including user personas, functional/non-functional requirements, data entities, workflows, integrations, and Cloud Service requirements.

2. The Review Loop This is where you take the wheel. You review the baseline and choose one of four paths:

• Clarify: Use the sidebar conversation (like Google Docs comments) to ask the AI questions about dense sections. It answers using the specification as its locked context.
• Refine: Use UI or instruct the AI to adjust, add, or delete content incrementally.
• Regenerate: If the draft missed the mark, note your constraints and regenerate from scratch.
• Revise Input: If outputs are consistently off, the source requirements are likely too vague. Revise them and restart.

The Result: Convergence in Hours, Not Weeks

Based on our case studies, an architect can reach a highly accurate Software Specification in a few iterations, typically less than a day.

This offers a strategic advantage: because this step often happens before a project is funded, generating high-fidelity specifications instantly lets you present a mature proposal to stakeholders without weeks of unpaid work.

The following screenshots show this workflow in practice. Notice the sidebar conversation on the right. This is your cockpit for steering the AI.

SDLC Step 2 — Generate Architecture Diagram using AI

With the specification locked in, the next phase is architecture.

Here, AI uses the Software Specification as context to generate a system architecture diagram. If you’ve refined the specification to high accuracy in the previous step, the generated architecture will reflect that accuracy. But it may not match your vision exactly.

This is expected. Note your feedback, supply it as input, and regenerate. The AI will adjust.

Regenerate Dialog

Converging on the architecture typically takes two or three regeneration cycles with comments. During regeneration, you can choose to retain past execution history as context — useful for incremental refinements where you want to preserve earlier decisions.

In other cases, you may want to start fresh. Uncheck “Retain Past Execution Memory as Reference” to discard prior context and regenerate from a clean slate.

Rest of Activities and Their Organization

The diagram below shows all activities in the SDLC pipeline. They appear in the logical order they’re executed, moving from specification through design to prototype. We will cover them in future blogs.

The activities are classified in three categories (progressively including previous perspectives).

Specification Perspective Activities

This phase transforms raw requirements into structured artifacts: the Software Specification, Architecture, Estimates, and Cloud Service mappings. Each artifact feeds the next, building context as you progress.

Design Perspective Activities

With specifications locked, the Design phase produces the detailed blueprints: data models, API definitions, UI structures, workflows, and external system integrations. These become the source of truth for code generation.

Prototype Perspective Activities

This phase generates working software: test data, UI code, and a deployable prototype on cloud infrastructure. What would traditionally take weeks of development happens in hours.

Holistic Analysis Feature

At any point, you can run a cross-artifact analysis to surface inconsistencies, gaps, or misalignments between phases. Think of it as a built-in reviewer that catches what human eyes might miss.

In Trillo AI, this feature is accessible via the green ‘Analysis’ button on the top bar.

Concluding Remarks

We started with a fear: that AI might make us obsolete. We end it with a realization: AI is the lever that allows us to lift heavier rocks.

By adopting this AI-powered process, you are stepping up. You are moving away from the fatigue of “blank page syndrome” and the drudgery of manual drafting. You are embracing a role where you innovate faster, deliver higher quality, and explore areas beyond your traditional expertise.

The future of software engineering isn’t about writing code from scratch. It is about orchestration, judgment, and vision using platforms that support rigors of software engineering.

It is time to stop pulling rocks and start building castles.

Let us take a deep breath and for the sake of brainstorming, see how such an AI based solution could help in the future. This blog makes an attempt to explain the use of AI agents in air traffic control in simple terms. It is safe to assume that AI agents are not replacing human air traffic controllers in the foreseeable future. Long before that happens, they need to show up on the stage in assistive mode and prove themselves.

If AI agents are in the assistive mode, it should lessen our concerns. As we speak, a set of software services are assisting air traffic controllers. AI agents are similar to those services with very important distinctions between them. AI agents can collect unstructured data from the environment and make dynamic decisions. Whereas, software services are hardwire logic that may cover several paths but not a new one. It has to be updated to incorporate new rules.

Imagine if there would have been a Voice AI agent in the control tower, and it heard somebody say, “OMG that chopper is too high”? “Beep, beep, beep” goes another AI agent with a flashing aircraft trajectory and helicopter trajectory. In nutshell, an AI agent can draw inference based on available information that was missed in a hardwired software system. It does not need yet another integration to feed the data in structured format, rather it can consume unstructured data in natural language or CSV format.

For the illustration, this blog uses a high level architecture of a traffic control system. This is much simplified architecture. It may not have any resemblance to the real architecture. The idea is to compare conventional architecture with AI agents.

Traditional Aviation Traffic Control Architecture

The above architecture is based on microservices. Once again, this is a simplified architecture for illustration purposes.

A set of microservices receive input data from multiple sources and continuously evaluate them against a set of rules. These rules are hard-coded by developers in the microservices’ implementation. Let us assume, if developers forgot to implement the rule “Helicopter’s max height should be 200ft, else issue a high alert”. This could be a problem. Although it may not have happened in case of the Washington tragedy, it illustrates how hardwired logic works.

Let us examine how an AI agent can dynamically respond to such rules and environmental contexts.

AI Agents for Traffic Control

In the above diagram, a parallel architecture uses AI agents. We have added two additional agents, 1) a Vision Agent and 2) a Voice Agent to capture additional environmental context. Since agents can consume unstructured input, it is relatively easy to add them.

AI agents are collaborating processes. They collaborate through an orchestrator. The Rules Agent is fed with data prepared by other agents. It draws inferences by interpreting natural language rules in the context of data. These inferences may lead to actuating an alert. Let us assume that the helicopter height data was not sent by the helicopter. Instead, someone uttered words in the control room, “OMG, that chopper is too high”. These words and the rule that height has a limit, AI can conclude there is something wrong.

Using this simple illustration, we demonstrated that AI agents can receive unstructured input from the environment. AI agents using the rules described in a natural language can act as an intelligent traffic controller assistant.

AI Agent Risks and Mitigation

The biggest risk of an AI agent is what if, it infers something that is converse of the desired outcome. For example, it may push the helicopter to 300ft. Such inferences will be counter productive and even disastrous. The following are some of the mitigations.

Grounding of AI Result

This could be a cost burden, but a validation system using microservices or another set of AI agents can validate outcomes. Such a system will not be as expensive as the real traffic controller system but incorporate the same rules (implemented differently).

Testing and Validation for Several Months

It is safe to assume that these AI agents will undergo testing with simulated data and real data. Simulated data can be generated by another set of agents.

Use as an Assistant

Agents should be used in assistive mode as helpers to a real human.

Interesting Use Case of AI Agents

Let us say, you are not ready to deploy AI agents in a control tower. Still, there is a very interesting use of them. They can work offline on the same set of data and produce signals. These signals can be compared with the signals generated by the actual system. It will bubble up the signals that are missed by the actual network traffic control system. This way, you can find the gap in the micro services implementation and use this information to improve them.

This is the case where instead of providing training data to train a model, you are using AI output to make a conventional system more intelligent.

Conclusion

AI is the future. This article describes how you can incorporate AI in a stepwise manner in a scenario. Let us hope that one day, AI will avoid another Washington.

New AI capabilities can easily interpret text based content but struggles with more complex structures such as tables, schematics, jump pages, multi-column layouts, page content containing noisy blocks and even bullet points.. In this article, we would later see that the Trillo Doc AI solves this problem and makes their content available online for browsing, search, Q&A and reasoning using AI.

Your organization likely has tons of these documents: manuals, designs, old magazines, contracts, financial statements, and more. Right now, this information is only accessible to a few who already know where to find it. They waste time searching, sometimes hours, for the right document and page. AI can change this. It turns hidden content into knowledge accessible to any authorized user, allowing them to find what they need in seconds. Users can then interact with the information using AI: summarizing it, extracting key concepts, downloading data, and asking questions. The possibilities are endless.

This means users can quickly become experts on any topic. Even seasoned experts can discover new insights, like patterns between successful and problematic contracts. Imagine having all the content ever created, by humanity or just your organization, extracted and analyzed by AI. Imagine interacting with it through an AI agent, learning for pleasure or profit.

Think about that 900-page bill from Congress that is impossible to understand. With AI, you could interactively query it and understand its implications. Or imagine learning from aircraft maintenance manuals, taking notes, and discussing them with an expert, all aided by AI.

Combining AI with human intelligence allows for powerful learning grounded in reality. It lightens the cognitive load and makes the process more enjoyable. The best part? Platforms are available right now to help you get started in minutes.

Example Use Cases

Before you jump into a platform, let’s explore a few use cases to spark your imagination.

Old Magazines and Periodicals: Imagine you’re a publisher with a treasure trove of content, some of it going back to the 1800s. Much of this valuable material might be gathering dust in physical archives or buried in scanned files. Bringing these issues online would be a huge benefit to your readers. Think of a literary magazine like Kadambini (published in India until 2020) — making its entire archive available online would delight readers, and they would be happy to pay for access.

Industrial Manuals: Think about aircraft maintenance manuals, oil rig operation guides, tool specifications, engineering designs, and similar documents. AI can analyze, format, and catalog all of this, making it easily searchable. Users could query these manuals using text, images, charts, or schematics.

Office Documents: AI can be incredibly useful for everyday office documents like contracts, resumes, purchase orders, bank statements, mortgages, and marketing materials. It can power advanced document search, extract key data for workflow automation, and perform semantic matching. For example, AI can match job descriptions to resumes or analyze contracts for specific clauses. And, of course, it can search across all these document types.

Content Processing Platform or Knowledge Agent

So, what platforms can convert your content into usable knowledge?

We’ll focus on Google Cloud AI platforms and our own platform, Trillo Doc AI. We’re including Trillo Doc AI because we believe its features are particularly useful for complex documents that require customization. While other companies and hyperscalers (like Azure and AWS) offer similar agent builder tools and services, we won’t discuss them here due to our limited knowledge of their specific functionalities.

Google Cloud NotebookLM: This platform is incredibly user-friendly. As their website says, “Upload PDFs, websites, YouTube videos, audio files, Google Docs, or Google Slides, and NotebookLM will summarize them and make interesting connections between topics, all powered by Gemini 2.0’s multimodal understanding capabilities.” It’s designed to help you understand complex information by providing a single place for reading, thinking, asking questions, and writing.

Vertex AI Agent Builder (Google Cloud): This platform allows you to build AI agents, from no-code solutions to highly customized ones. Think customer service chatbots, employee agents that automate tasks, knowledge agents for search and Q&A, and even voice agents. For extracting knowledge, the knowledge agent is most relevant. The basic idea is similar to NotebookLM: upload documents, process them, and then enable searching and question answering. Vertex AI Agent Builder also offers connectors to enterprise applications for ingesting content and data. Developers can also create custom or specialized agents for additional needs.

Google AgentSpace: Google AgentSpace is delivered as a SaaS application (like Google Workspace) and built on the top of Vertex AI Agent Builder. It is simpler to use. It offers a more limited set of developer tools and is fully managed by Google Cloud. Vertex AI Agent Builder, on the other hand, is a more comprehensive platform for building and managing AI agents.

Trillo Doc AI: We’ll dedicate the next chapter to Trillo Doc AI, explaining why and how it provides an extended knowledge agent using Gemini and vector databases available on Google Cloud.

Trillo Doc AI

When you look closely, all knowledge agent platforms essentially do the same thing: they ingest content from various sources, process it using a large language model (like Gemini), and then make it available for searching, Q&A, and reasoning. For many common office documents, the processing pipeline is standard. However, documents with specialized domains, layouts, and content require a custom approach. Think about these examples:

1. Old, multi-column magazines with photos, tables, paragraphs, and other elements.
2. Engineering documents like operation manuals, designs, and customer service content, which often include tables, schematics, drawings, and charts.
3. Academic papers with formulas, tables, charts, and equations.

Essentially, any document can benefit from a custom processing approach.

Beyond just processing documents for search, Q&A, and reasoning, Trillo Doc AI also extracts content in a structured format. This means it identifies and extracts headings, paragraphs, bullet points, tables, images, charts, and more. This structured data can then be used to display the document content clearly on any device, ensuring high readability. It’s also crucial for building semantic indexes.

Trillo Doc AI is a platform for building custom pipelines with no-code or low-code (using prompt engineering and serverless functions). In addition, Trillo Doc AI extracts structured parts of documents for intelligent processing and user friendly rendering.

Trillo Doc AI — Example Pipeline for Processing Magazine

Here’s an example of a processing pipeline implemented by Trillo Doc AI for magazines (or any bound document, like a book). It follows these steps:

1. Extract Content: Trillo Doc AI extracts the content using OCR or open-source libraries. This pulls out the text and images embedded in the document. It also generates preview images and thumbnails at this stage.
2. Clean Content: Trillo Doc AI then converts the page content into structured content (JSON format)
3. Summarize: Using the cleaned content, Trillo Doc AI creates a summary of the document.
4. Image Properties: Trillo Doc AI extracts image properties using AI Vision and/or Generative AI.
5. Extract Tags: Trillo Doc AI extracts tags to enable filtering and searching by categories.
6. Extract TOC: The table of contents is extracted by Trillo Doc AI.
7. Make Chapters: For easier browsing, Trillo Doc AI constructs chapters.
8. Clean Chapters: When pages are combined into chapters, Trillo Doc AI cleans them up. This might involve things like combining paragraphs that were split across pages.
9. Post Processing: Finally, Trillo Doc AI uses serverless functions to perform any other rule-based processing on the content.

Trillo Doc AI — Example Pipeline for Extracting Data from Variety of Documents

Trillo Doc AI supports creating custom workflows for different needs. For example, if there’s a need to extract common fields (say, for asset management) from a variety of documents, the following workflow can be used:

1. Extract Content: Trillo Doc AI extracts content from the document using OCR or open-source libraries. It extracts text and images embedded in the document and prepares preview images and thumbnails.
2. Classify Document: Trillo Doc AI classifies the document using Generative AI.
3. Extract Data: Trillo Doc AI extracts data using Generative AI, other NLP processors, and proprietary algorithms.
4. Post Process: Trillo Doc AI post-processes the content for validation, storage, and other needs.

Conclusion — A Summary of Trillo Doc AI Features

In short, Trillo Doc AI offers a powerful set of features crucial for effective content processing.

1. Trillo Doc AI leverages the power of Gemini and other Large Language Models. Its API-driven design also allows it to seamlessly integrate with other platforms, like NotebookLM Enterprise, extending their capabilities.
2. Trillo Doc AI excels at custom document processing, a critical need for complex, domain-specific documents.
3. It extracts content elements — headers, paragraphs, tables, bullet points, charts, images, schematics, graphs, and more — as structured JSON data. This allows for the creation of highly readable content across all devices, dramatically improving user experience.
4. Recognizing that AI isn’t always perfect, Trillo Doc AI incorporates tools for manual review and content curation, putting humans in the loop for essential quality control.
5. Trillo Doc AI provides access control that is easy to define, audit and performs well at search scale.
6. Finally, Trillo Doc AI is built on Trillo Workbench, a robust low-code platform. Trillo Workbench provides a model-driven architecture for data, serverless functions for logic, and a high-level API that abstracts away the complexities of the cloud. This foundation allows us to rapidly add new steps to the processing pipeline without the headaches of security, scalability, compliance, or CI/CD.
7. You can find more about Trillo Doc AI here.

Recently, someone asked me for an elevator pitch. I started — “we have invented a theory, I call it the General Theory of Relativity. According to it, when light passes by a massive body, it will bend……”. Obviously, the conversation was not going anywhere. I am no Einstein. But having spent months in the basement and discovered a thing, it will need some time and attention to communicate the thought. Or, could I have said, “if we travel with the speed of light, we will not age much when we reach 147th floor.” It may not have changed the outcome.

There are things that can’t be explained in an elevator. Therefore, I have some disdain for the elevator pitch. Let us give some ideas a few minutes and a diagram or two. Let us listen.

If you have come this far then may I request to bear with me and read on.

Let us say, a programmer writes a file to a cloud bucket. A cloud bucket could be on AWS, Azure, GCP, Oracle, etc. Should it not be as simple as — “write this content to a file in this directory?” But it is not, it is far more complex — acquire token, include cloud specific client toolkit or construct restful request, etc. After all, we write files to a Linux disk without knowing it will be on Ubuntu or RedHat, therefore we have come to expect simpler API on the cloud as well.

Since a common abstraction of multiple Linux is possible, a common abstraction of all clouds is possible too. But in this case, we can’t rely on cloud providers to build a common abstraction. In fact we should not expect it. Instead we should assume that, it is not the responsibility of cloud providers to standardize on common APIs. Asking them to converge will be counterproductive and will never happen.

It is the responsibility of a platform provider to abstract all clouds in a common set of APIs and provide its cloud specific implementation transparently. It is shown in the following diagram.

Complexity in providing a common API platform?

Our example of writing a file to a bucket is one of the APIs. There are several more such APIs for database, storage, security, AI, etc. They follow the same principles on every cloud therefore they can be abstracted as common APIs. The problem is that implementing them and providing a serverless programming model requires several years of effort. It is a risky proposition. Due to a long time span, it gets boring and teams disintegrate resulting in failure.

We have built such a platform called Trillo Workbench. It is deployed with hundreds of customers. This blog is to create mindshare for the platform. If you are interested in discussing, please send me an email at “anil at trillo dot io”.

Thank you.

RAG meaning in nutshell — RAG application means a ChatGPT or Gemini like application that uses some of your organization’s content in the context of the prompt.

AWS Intelligent Document Processing (IDP) and Trillo Doc AI are platforms to make fully baked content readily available for generative AI (Gen AI) applications and research.

AWS IDP is a reference architecture of document processing with sample open source code. Trillo Doc AI is a production grade platform available through the Google Cloud marketplace.

There is a wide gap between what AWS IDP provides and what a production grade document processing platform would require. The gap is due to the lack of a scalable pipeline to process 100s of millions of documents, security, role / rule / policy based access control, audit logs, business process integration etc. Trillo Doc AI is a production grade platform that bridges this gap.

Whereas AWS IDP consists of sample code via a public git repo, Trillo Doc AI provides an open metamodel and serverless functions framework. It provides source code of serverless functions and metadata. It uses a core platform called Trillo Workbench for other services — scalability, security, access control, audit logs, and admin UI (source code available on request). Trillo Workbench source code is not available except for the admin UI. But you will never need its source code. It is a zero cost, robust, proven and enterprise grade work horse.

The rest of this article covers the following topics:

1. Why is document processing important for Gen AI?
2. A conceptual architecture of document processing.
3. Trillo Doc AI architecture on GCP.
4. AWS IDP reference architecture.
5. A checklist of comparison of AWS IDP and Trillo Doc AI.

There are other equivalent solutions such as AODoc, FileNet, Box, etc. Some of them stop at the content management (ingestion and providing UI to manage folder hierarchy and file sharing). They do not provide further processing of documents that is needed for RAG applications. Some of them do not provide open architecture like AWS IDP and Trillo Doc AI do. The open architecture or open source is important for the fast pace innovation of Gen AI. Therefore, in this document we will limit ourselves to AWS IDP and Trillo Doc AI.

Disclaimer: Other tools are worthy. It is our opinion that they do not provide an open and convenient programming model for customization. It is a fast changing area therefore this information may not be the latest.

Why Document Processing is important

Every organization is looking to apply Gen AI in business use-cases. There are two primary use-cases’ categories.

1. Generate new content using prompts — such as code generation, image/ video generation, text content such as marketing material generation.
2. Generative AI RAG — there are numerous use-cases of RAG in an enterprise. RAG solutions provide contexts and prompts to generate responses. Each context is generated from the enterprise content.

RAG based use-cases, innovations and research will require that the enterprise content is made readily available in the right format to the teams of innovators and implementers. Without it, it is like asking teams to build an analytics application without having a data warehouse. The teams will have to acquire content first. It may take months and be a very inefficient process. By the time you finish acquiring content, your competitions would already have won the battle.

To conclude, in order to spur innovation using Gen AI, the first step is to make the enterprise content readily available. The most efficient way to get the content ready is having an enterprise strategy and using proper toolings.

Generic Architecture of Document Processing

The following diagram shows the architecture of a generic document processing system.

A generic document AI processing pipeline consists of the steps below. Trillo Doc AI and AWS-IDP pipelines are specific implementations of it using GCP and AWS services respectively.

1. Ingestion: Content from multiple sources is brought into a cloud storage bucket for processing.
2. Processing: Document processing pipeline processes it using a set of NLP, LLM services, and open source libraries. It runs multiple substeps as follows: a. Extraction: Text and images are extracted from each document. The raw text and images are stored in new locations on buckets. b. Redaction: Optionally, PHI and PII data is redacted from text and images before storing. c. Structured Data Extraction: Text is parsed to extract structured data from documents. The structured data extraction requires a multitude of steps — entity extraction, form fields extraction, and extraction of data in tables.
3. Chunking: Text is chunked into smaller yet complete chunks (i.e. avoiding splitting a chunk such that the central idea of the chunk is not lost, and thus leading to less accurate semantic meaning).
4. The above two steps require the ingestion of domain knowledge into the pipeline.
5. Storage: Processed data is stored in a variety of databases — relational DBs, vector DBs, search indexing platforms.
6. RBAC: Sometimes, the source document repositories provide access control information (such as enterprise content management systems). This access control information is stored and applied in the API gateway. The access control information can also be enriched or customized.
7. API Gateway: An API gateway delivers content using HTTPs, or bulk transfer from bucket to bucket. RAG pipelines and domain specific applications are consumer of API gateway.
8. Admin UI: Document processing platform provides an Admin UI for the management, monitoring,

Trillo Doc AI on GCP for Document Processing

Trillo Doc AI processing pipeline is a specific implementation of the generic document processing pipeline using GCP services such as Google Vertex AI (which includes Gen AI, LLMs APIs), Google Document AI (NLP based parsers for forms, tables, purchase orders, invoices, etc.), AlloyDB (vector database), Cloud SQL (managed relational database), Google Cloud Storage for buckets.

AWS IDP for Document Processing

AWS-IDP processing pipeline is a specific implementation of the generic document processing pipeline using AWS services except for 3 pieces that are missing and noted below. The above diagram looks different because it opens up steps that are hidden within Trillo Doc AI diagram. Else, they are equivalent.

The missing pieces from AWS-IDP are:

1. RBAC: AWS-IDP does not provide a component for role based access control. It is very critical for enterprise applications.
2. Folder Hierarchy: It does not have a mention of folder hierarchy. Upon visual inspection of the code repository, I did not find anything. A logical folder hierarchy is very important for the use of documents in multiple applications.
3. Search Engine: For small phrase embedding, it does not integrate with a search engine (like Solr, Elastic search that uses tf-idf and BM25 algorithms for embedding).

Out of these 3, 3rd is easy to add to the pipeline. 1 & 2 would require several months of work, if not years.

Checklist — a comparative analysis of Trillo Doc AI and AWS IDP

When to build your own platform?

Document processing as described above is the glue logic. Normally you will not spend millions of dollars building it if you can subscribe to a 3rd party product for thousands.

You may consider building your own document processing pipeline if it meets all of the following criteria.

1. The content processing needs are simple.
2. There are few document formats (mostly PDF and docx).
3. There is no data extraction needed. Especially since there are no complex tables.
4. Your use case does not require access control, governance, or compliance.
5. You will build once and do not see much need to change, maintain or enhace it.
6. The number of documents to be processed is in thousands.
7. You will keep using it for more than 5 years.

A majority of startups, in technology, need an application server for their products. Even if they are building a product based on algorithms, new AI models, or backend data pipelines, they need application server functionality in some form. An application server is needed for a UI that becomes the face of the product to customers and investors.

Building a robust application server is a tedious task requiring valuable time and resources.

Even when the application server is a core part of the startup offering, there is a strong business case for using a platform like Trillo Workbench for MVP. It will let the founders focus on product launch and save money otherwise spent on engineering. Even if they want to build their platform in the long run, launching the product first and learning the exact requirements through the first release is a better strategy. After that, building an application server platform using an open-source stack will be easier and more cost-effective (and only if you have business reasons to build it).

This article describes how a startup can use a pre-built application server and customize it easily using citizen developers.

The rest of the article is relevant only if you anticipate that your offering will have an application server.

What is an Application Server

Let us first define an application server more precisely for the current discussion.

An application server provides functionality such as data service, file, document management service, user management, authentication, authorization, security, scalability, audit logs, and client APIs. A modern, cloud-based application server also includes APIs to interact with cloud services such as bucket service, AI service, secret and key management service, data redaction service, etc.

A modern application server incorporates Model-Driven Architecture, i.e., the data model can be specified using metadata. It should let you add application logic using functions ( serverless functions or lambda) without having to go to the cloud console.

It should not require CI/CD and DevOps. It should not require application developers to learn cloud configuration.Such an application server should run on a cluster such as Kubernetes. It should auto-configure the cloud at the time of deployment.

Trillo Workbench is such a platform.

Process of Building a Platform from Scratch

If you are building a platform from scratch, you will select a stack, such as NodeJS, Python, Golang, Java Spring Boot, etc. Next, you will set up a git repository and CI/CD for development. To deploy the server, you will select a public cloud, configure it, and write Terraform templates to create a cluster (likely Kubernetes).

After that, you will start writing a data model, a service to connect to a database, and code manage DB connection pools efficiently. You will write SQL to create tables. After that, you will write business logic as controllers and services. Every small change will trigger CI/CD.

You would eventually work on authentication, open ID connect / SAML, authorization, and role-based access control: audit, security, scalability, and high performance. If your application uses cloud services such as bucket, memory store, columnar databases, or AI/ML services, you will integrate them using the client toolkit or restful API. You will need a good design for large workflows processing large amounts of data or content. It will require fail-safe, multi-threaded design, and management of the state of each job.

Each piece may seem simple. Integrating pieces into a cohesive platform is time-consuming and challenging. Your goal was to build the best analytics platform for healthcare, instead, you will be building an application platform. Soon you will be so invested, there will be no way out of the mess. You will have to keep raising money and investing. Or, you will call it a day.

How does Trillo Workbench work?

Getting started with Trillo Workbench and building your application’s first set of APIs is a three-step process.

1. Deploy Trillo Workbench from the marketplace.
2. Create a few tables using Trillo Workbench consoles.
3. Write a few functions in Python.

Challenges

Getting started with a new concept or platform (especially when it is a developer platform) is always a challenge. You normally lean towards building one from scratch using some boilerplate open-source code. It looks easy since you can run it quickly on your machine.

Signing up for a cloud console, creating a project, deploying Trillo Workbench, and familiarizing with its console may take a few hours. It creates friction.

Learning APIs is the next step. Although it is simpler than the cloud API, it requires some time. That is another point of friction.

Overcoming Challenges

Bootstrapping the process is a big challenge. If Trillo or its partner can assist you in a cost- effective way, it may be very smooth. In a day, you will be running on the cloud and in weeks you will be showcasing your UI to users. See the following section for more details.

Collaboration with Trillo and its Partners

The Trillo team can help you with the following steps. We recommend that you follow these steps — design and then construct. They are the optimal path to an MVP. Trillo Workbench leaves enough scope for iteration therefore you don’t have to have a perfect design.

Design Application

The key is that an MVP should be simple yet complete. You may not develop rarely used features, but you must complete features that are needed to operate the application. Trillo and its partner team can play more than just the role of a development team; they can also assist you in the right product design (the role of your assistant product manager).

• Design Figma: You can do it yourself, or the Trillo and its partners team can help you. Our engineers and user experience team have experience with 100s of applications. They understand the minimal UI that end-users will love. For a startup, it is so important that this can decide whether you succeed or not.
• Write Business Process: Our teams (Trillo and partners), under the guidance of experienced application designers, can write the optimal business process. They can also help you identify overly complicated processes and how to simplify them.

Bootstrap Development

Once the previous step is completed, Trillo and its partners can help you bootstrap the environment in hours and supercharge development. They can execute the following steps.

• Install Trillo Workbench.
• Identify the UI/UX technology you want to use, such as Angular, React, Flutter, etc.
• Trillo will provide a boilerplate UI for user management, sign-up, and sign-in.
• The Trillo team will set you up with the environment in a few hours, including UI/UX.
• Your team will work with the Trillo team for the rest of the week and another week to co-develop a part of the application.

Business Benefits

If you follow the above process and make an initial investment of $5-$10K, we promise the following business benefits.

1. You will build your application (excluding innovative backend services, algorithms, model training, etc) for 10–20% cost. It may translate into savings of several hundred thousand dollars to millions of dollars.
2. You will build the right application.
3. You will have more time for selling, marketing, interviewing users, talking to investors, and meeting people.

If you are convinced, let’s talk, and we can help you figure out how best to build your solution. Otherwise, you will be caught up in JavaScript, Python, GoLang, Java code, CI/CD, and DevOps. It will spiral out of control.

This article sits somewhere between a whitepaper and a cheat sheet. It introduces the concept of using an LLM for your documents along with the associated problems and solutions.

Doc AI has hype around it and creates a wrong notion that it is simple and easily attainable. But, using Doc AI with high accuracy requires solutions to several problems. Below, we outline some of these challenges and their solutions, and a pathway to solve others.

Note: The term “document” means text content that can come from any source, a file, an API, email, etc.

What is Doc AI

For this article, Doc AI is defined as a Large Language Model (LLM) such as ChatGPT or Gemini applied to a set of your documents or content.

In general, document repositories contain a trove of information that is not easily accessible to users unless they first know which documents to open and then read through the content of each document. With the massive size of most repositories and applicable content, this is not realistic.

Doc AI, using LLMs along with Natural Language Processing (NLP) revolutionizes finding information within document and content repositories with accuracy in the high 90s percentile.

Doc AI Usage Patterns

The following Doc AI patterns illustrate how it works.

1. Extract structured data from unstructured data with very high accuracy: This works using traditional NLP parsers (for example, Google’s Document AI processors) and schema-driven parsers based on LLMs. The goal of high accuracy makes it a tough problem. Some of these problems are discussed below.

2. Semantic search: It means searching a document using an English phrase. Imagine it as, Google Search on your content. It is more complex than just indexing. LLMs are very sensitive to chunking and prompts. Chunking or partitioning paragraphs or blocks while maintaining the right context is very important.

3. Question and Answers: This is more than just a keyword or phrase search. For example, Doc AI using an LLM enables you to ask about context sensitive questions. For example, you could ask “What is the total value of contracts signed in the last year?”.

4. Summarization: This provides a concise overview of the document.

5. Inference: This is a more advanced use case of LLMs. It is about inferring something from the document — for example, an assessment of malpractice from a medical transcription or a problem that may be causing a symptom such as, “I can’t connect to the internet”. The LLM may return a contextually-relevant answer such as “Can you check if your router has a blinking red light?”.

6. Hybrid of all: In general, each use case will be a combination of some or all of the above.

Use Cases

The use cases for Doc AI are numerous. Below, we divide the use cases into simple and complex. The complex use cases require a Retrieval-Augmented Generation (RAG) approach or model fine-tuning.

Simple Use Cases

1. Parse paper forms: This use case refers to parsing digitized versions of paper forms such as POs, invoices, medical reports, bank statements, etc. It is a simple use case but with one complexity where table parsing is still inaccurate. We discuss the problem and solution in more detail below.

2, Search and Q&A on Customer Service Portal — Searching the customer service portal of a bank, cable service provider, or manufacturer can be incredibly frustrating. These search results often return thousands of documents in the results. Finding the needed answer is this sea of results that is often exasperating and fruitless. Using Doc AI with semantic search and Q&A returns immediate succinct answers to the user

3, Multiple use cases for data extraction and search — Several use cases require data extraction and semantic search indexing (vectorization). These support user interactions based on a query. These queries require either data look up, search, or Q&A using LLM prompts. Example queries might include:

• “What are risk factors in the contract?”
• “What is the value of the lease?”
• “How many leases will expire in December 2024?”
• “Which customers are likely to cancel the service?”

Additional use cases can be extrapolated from the above examples.

Complex Use Cases

More complex use cases of Doc AI deal with reasoning or matching (the cross-product of two documents’ vectors).

1. Document Matching: Matching a document against another set of documents to find similarities and differences. A simple example could be, evaluating a newly written document to judge whether it complies with existing corporate policy documents. Another example might be determining the risk factors of a new contract document compared to other contracts written in the past.

2. Inference: There are cases when information is inferred based on historical documents. For example, given a set of contracts, infer what risks are missed. Or, given a set of medical record guidelines for ICD/CPT code assignment, infer the ICD/CPT code from a medical transcription.

Potential difficulties when using Doc AI and LLMs

Several major issues must be solved with Doc AI using LLMs to avoid skepticism and disappointment. Let’s examine

Potential Problems

1. OCR or native PDF parsing inaccuracies: Many documents are stored as PDF files (including form-filled and scanned images of existing documents). Existing PDF extraction libraries often generate inaccurate results, including improperly ordering of text content as well as incorrectly interpreting characters and what text line the character is associated with. Starting with bad data leads to faulty and inaccurate results.

2. Table parsing: Table parsing is still a hard problem. It is difficult to identify boundaries of columns and rows in a PDF. Even when there are lines, embedded tables, cell merging, etc. make it hard to correlate column labels with cell values. Overflows of content across columns also introduce another dimension.

3. Chunking changes context: One chunking strategy does not fit all documents (even if they are the same type).

4. Prompt sensitivity: Doc AI uses prompts behind the scenes to extract data. The quality of data extraction changes based on the prompt designs.

5. LLM model upgrade may break a working system: While upgrading an LLM model may provide benefits, it may also break features and accuracy that existed with the old model.

6. Processing millions of documents and yet track failures: When running millions of documents through a pipeline, there will be several failures that need to be tracked and fixed. It requires a solid foundation platform that can manage a large scale compute farm.

7. Access control in search, QA, and summarization: Without a proper authorization system (such as RBAC) in place, millions of documents may be available to unauthorized users. This could result in confidential and private data being viewed by users who do not have the need or authority to view such information

8. Integration: Although more of a software engineering problem. Doc AI will likely need to be integrated into the application. The following integration points are needed:

• Data Ingestion: Not all content may come from a file system. Data may come via APIs, SFTP, enterprise applications, and content management systems.
• Export Results: Not all resultant data may be consumed by a human actor. Exported data may be collected and pushed to other automated system.
• Enterprise Workflow: Doc AI may be a part of an enterprise application. For example, an insurance underwriter chooses a policy document and the rest of the Doc AI work happens in the context of the policy.

Problem Solutions

1. OCR or native PDF parsing inaccuracies: Use character matrix and configurable line spacing hints to generate content rather than relying on the PDF extraction library’s output.

2. Table parsing: The table parsing can be improved by applying a few techniques:

• Extract only required columns (specify minimal set).
• Stretch columns horizontally using character matrices returned by the library and create space between columns. Using the vicinity of characters, assign overlapping content to the nearest character (on left or right).
• Use LLM to associate columns labels and cell values.

3. Chunking changes context: Chunk documents in 2–3 ways and use each set for data extraction. Pick the one with the best score.

4. Prompt sensitivity: Use multiple prompts and few-shots. Let the system pick the best result.

5. LLM model upgrade may break a working system: Track prior documents processing, select a few of these same documents (as many as economically possible) to QA upgrade by generating a report that highlights differences.

6. Processing millions of documents while tracking failures: Report failures via a database table and UI (with drill-down of logs of each step).

7. Access control in natural language queries: This requires two steps:

• Storing Access Control Matrix: First off, Doc AI needs to know the permissions available for each user. If the number of users is more than a few 10s, this may require a complex configuration (if manually done prone to error). A good solution is to capture information from the source (where you are getting the content) or use a set of policy documents.
• Applying Access Control Matrix Efficiently: This gets complex and slow if the access control rules are complex and needs to be applied to millions of documents.

8. Integration: Use a platform that makes it very easy to write RESTful connectors.

9. Use of metadata and admin console: During POC, domain experts need to try various prompts, chunk sizes, and schemas (data to be extracted). An interface should be provided to enable users to experiment with a limited set of data to find the best metadata. Results should be tracked for each run so users can compare them.

10. Continuous Quality Assurance (QA): Collect audit logs and generate reports so a team of experts can QA the system periodically.

Trillo Doc AI — An Out of the Box Solution to the Above Problems

Trillo Doc AI is a solution built on top of Trillo Workbench that solves the above problems out-of-the-box. It can easily be deployed in your GCP and Azure environments (AWS is coming soon). You can create a POC with a handful of documents in a week or two. If you find the results are great, simply call it a production deployment and start using it.

Trillo provides implementation services to assist you in the adoption of Doc AI technology (directly and partner-assisted).

The foundation platform, Trillo Workbench provides extreme agility to customize Trillo Doc AI to your needs.

Doc AI — Problems and Solutions for Unmatched Accuracy was originally published in trillo-platform on Medium, where people are continuing the conversation by highlighting and responding to this story.