AI System Design: From Requirements to Architecture

A demo proves nothing.

This is the most important thing to internalize before designing an AI system, because demos are the artifact that enterprise AI projects most commonly optimize for. The demo shows the best case: a curated prompt, expected input types, a well-behaved model, a clean data source, no adversarial users, no scale, no operational pressure, and a team standing by to fix things when they go wrong.

The question nobody asked during the demo: what happens when the input is hostile, enormous, or in the wrong language? What happens when the right document is not found, or the wrong user’s document is retrieved? What happens when the provider is slow, down, or retires the model? What happens when the model returns something your code cannot parse? What happens when the model does something real and wrong? What happens when usage grows ten times? What happens when it breaks and nobody knows why?

AI system design is the discipline of answering these questions before they become production incidents. It is the translation process from a business problem — something a stakeholder described in a meeting — to an architecture that handles not just the expected case but the full distribution of real-world inputs, failures, scale, and operational demands.

This article covers that translation process end to end: how to gather requirements that surface the right constraints, how to map business requirements to architectural decisions, how to design the interface between the AI system and its users, how to anticipate and design against the failure modes specific to AI systems, and how to make the architecture decisions that determine whether the system reaches production and stays there.

Why AI System Design Is Different

Traditional software system design deals with deterministic components. Given the same input, a well-implemented function returns the same output. Failure modes are binary: the function works or it does not. Testing confirms correctness. Monitoring confirms uptime.

AI system design must address two distinct but deeply integrated fault-tolerance problems. The first is technical fault tolerance — the classical high-availability engineering of distributed systems. The second is semantic fault tolerance — the novel requirement to contain and correct failures originating from the LLM itself, using guardrails, continuous evaluation, and human oversight to handle bad model outputs.

The semantic fault tolerance layer has no equivalent in traditional software engineering. The LLM is a probabilistic component that behaves correctly most of the time but incorrectly some of the time, in ways that depend on the input, the model version, the temperature, the context, and factors that are not fully observable. The system must be designed to handle this — not by pretending the LLM is deterministic, but by treating it explicitly as an untrusted actor whose outputs require validation before they propagate to downstream effects.

Roughly one in twenty requests already fail in production AI systems, but the systems continue to run and return outputs that appear correct. Production AI requires continuous monitoring because silent failures often appear correct at the application layer.

This 5% silent failure rate is the design constraint that separates AI system design from conventional software system design. A system that fails loudly can be monitored, alerted, and fixed. A system that fails silently — returning plausible-sounding but incorrect outputs — requires semantic monitoring, evaluation pipelines, and human review mechanisms that most software teams have no prior experience building.

Step 1: Requirements Gathering

The Four Categories of AI System Requirements

AI system requirements fall into four categories, each requiring different stakeholders and different elicitation techniques.

Functional requirements — what the system must do. These are the capabilities the business is asking for: “extract key obligations from contracts,” “answer questions about our policy documents,” “route customer queries to the right support team.” Functional requirements are necessary but not sufficient — most teams elicit only these.

Quality requirements — how well the system must perform. Quality requirements for AI systems include accuracy thresholds, automation rate targets, latency budgets, and confidence calibration requirements. These cannot be specified without knowing what “correct” looks like — which requires working with domain experts to define the evaluation criteria before engineering begins.

Constraint requirements — what the system must not do or must always do. For AI systems, constraints include: data sovereignty (where data can and cannot be processed), regulatory compliance (what documentation is required, what actions require human approval), safety requirements (what outputs are never acceptable regardless of model behavior), and cost ceilings.

Operational requirements — how the system must behave under real-world conditions. These include availability targets, graceful degradation behavior when dependencies fail, audit trail requirements, monitoring and alerting requirements, and incident response requirements.

The Requirements Interview

The most valuable requirements are elicited by asking about failures, not features. Rather than “what should the system do?”, ask:

• “What is the worst thing that could happen if this system gives a wrong answer?” — surfaces severity and reversal cost
• “What data does this use case require, and what quality problems does that data have today?” — surfaces data readiness gaps
• “Who needs to see and approve what the system did and why?” — surfaces audit and explainability requirements
• “What happens on a day with 10× normal volume?” — surfaces scale requirements
• “If this were deployed today and broke, how would you know?” — surfaces observability requirements
• “What would make a business stakeholder pull the plug on this?” — surfaces trust and quality thresholds

Conventionally well-formed requirements are not significantly associated with AI design choices that closely match expert judgments. AI development workflows should more explicitly teach, support, and evaluate the stakeholder-to-design translation step so that stakeholder context is preserved as teams move from stakeholder needs to technical design.

The gap this research identifies is real and common: teams produce technically well-specified requirements that fail to capture the constraints that actually determine whether the AI system works in the specific business context. The solution is to involve domain experts in requirements review — not just as approvers of a requirements document, but as active participants in the failure mode analysis that requirements must drive.

The Non-Negotiables: AI-Specific Requirements to Always Ask For

These requirements are routinely missing from AI project briefs and routinely cause production failures:

Baseline measurement. What is the current state of the process this AI system will augment or replace? (From Article #31: establish the baseline before deployment, not after.)

Evaluation criteria. What does a correct output look like? Who is the ground truth authority? What is the evaluation dataset, who curates it, and how large must it be before the system can go to production?

Human escalation paths. For every output that the system produces, who reviews it when the system is uncertain? How does the system communicate uncertainty to the user?

Data access and authorization. Who can see what? What prevents user A from retrieving user B’s documents? Where are authorization checks applied — at query time, at retrieval time, or both?

Model version governance. What happens when the LLM provider releases a new model or retires the current one? Who owns the migration decision and the evaluation that precedes it?

Step 2: Capability Mapping

Capability mapping translates business requirements into AI system component decisions. It answers: which technical capabilities are needed, and in what configuration, to satisfy the requirements?

The Capability Decision Tree

Is the task primarily about retrieving information from a known corpus?
  YES → RAG pipeline required
        Is the corpus highly structured with entity relationships?
          YES → Consider Graph-RAG augmentation (Article #33)
          NO  → Hybrid vector + BM25 retrieval is sufficient
  NO  → Is it a generative task (draft, summarize, classify)?
          YES → Prompt-only or fine-tuned model
          NO  → Is it a multi-step workflow with external actions?
                  YES → Agent pattern required (Article #35)
                  NO  → Rethink whether AI is the right primitive

Does the output require structured format conformance?
  YES → Structured output enforcement required (JSON schema, function calling)
        Schema validation gate before output leaves the AI system

Does the task require domain-specific precision?
  YES → Fine-tuned model required (Article #28)
  MAYBE → Evaluate frontier model with domain-specific few-shot examples first

Is the output used in consequential decisions?
  YES → Evaluation pipeline required · Confidence signal required · HITL checkpoint required
  NO  → Standard quality monitoring sufficient

Is there sensitive data in the context?
  YES → Data residency constraints · PII detection required · Access control at retrieval

Mapping to the Full-Stack Component Set

Every enterprise AI system requires a selection from this component set. The requirements determine which components appear:

Most enterprise AI deployments stall not because the models are wrong, but because the architecture is incomplete. Most first-generation AI implementations failed not because the model was wrong, but because the architecture was incomplete — they treated AI as a feature rather than as a system capability that requires its own structural layer.

The policy layer is where most enterprise AI projects fail at scale. It is added as an afterthought when quality problems appear in production, rather than designed as a first-class architectural layer from the start. The guardrail service (Article #19), schema validation, HITL checkpoints, and audit logging are not optional production additions — they are requirements that must be satisfied for the system to be deployable in a regulated or high-stakes context.

Step 3: Architecture Selection

With requirements elicited and capabilities mapped, the architecture selection process makes four specific decisions.

Decision 1: Retrieval Architecture

The retrieval architecture decision determines how the system finds relevant information for each request.

The common mistake: choosing dense vector retrieval for all query types because it is the simplest path to a working prototype. The retrieval architecture should match the query type distribution of the actual use case — which requires knowing what kinds of questions users will actually ask, not just what kinds of questions the team can think of.

Decision 2: Agent vs. Pipeline

Many enterprise AI tasks can be implemented as either a fixed pipeline (a defined sequence of retrieval, prompt, and generation steps) or an agent (a reasoning loop that selects its own tools and steps). The choice has significant implications for reliability, cost, and debuggability.

Use a fixed pipeline when:

• The task has a well-defined, stable structure
• The sequence of steps does not depend on intermediate results
• Reliability and predictability are more important than flexibility
• The input-output contract is well-defined and testable

Use an agent when:

• The task requires dynamic tool selection based on intermediate results
• The sequence of steps cannot be fully specified upfront
• The task involves exploration or multi-step reasoning where the path depends on findings along the way

The default should be a fixed pipeline. Production failures in LLM systems stem mostly from orchestration issues, not model quality. Fixing these requires deliberate design at the orchestration layer, not a better model. Agents have more orchestration complexity and more failure modes than pipelines. Prefer pipelines until the task genuinely requires agent flexibility.

Decision 3: Synchronous vs. Asynchronous

AI system latency is higher than traditional software service latency. A frontier model call typically takes 2–8 seconds for a standard request; an agentic workflow with multiple tool calls can take 20–60 seconds or more. This latency profile forces an architectural decision that traditional software systems rarely need to make explicitly.

Synchronous (request-response):

• Appropriate when users expect immediate results and the task can complete within acceptable latency (< 10 seconds)
• Requires streaming responses for multi-second generation tasks — never block the frontend waiting for a complete response
• Use for: chat interfaces, single-turn Q&A, classification, extraction

Asynchronous (job queue + polling or webhooks):

• Appropriate for complex agent workflows, batch processing, document analysis, long-running research tasks
• The user submits a task and receives a job ID; results are delivered when ready
• Use for: document batch processing, complex multi-step research, report generation, overnight analysis

The hybrid pattern (streaming + async):

@app.post("/v1/tasks")
async def submit_task(request: TaskRequest):
    """For complex tasks: submit async, return job ID."""
    if request.estimated_duration_s > 10:
        task_id = await task_queue.enqueue(request)
        return TaskResponse(task_id=task_id, mode="async")
    else:
        # Short tasks: stream response
        return StreamingResponse(
            stream_task_result(request),
            media_type="text/event-stream"
        )

Decision 4: State Management

AI systems that handle multi-turn conversations or multi-step agentic workflows require explicit state management decisions. Three state types require architectural attention:

Conversation state — the history of a user’s interaction within a session. Options: in-memory (fast, lost on restart), Redis (fast, distributed, configurable TTL), database (durable, auditable). The choice depends on durability requirements and session length distribution.

Agent execution state — the in-progress state of an agentic workflow (completed steps, pending steps, intermediate results). For long-running agents, this state must survive service restarts. PostgresSaver (LangGraph) or equivalent persistent state storage is required for production agentic workflows.

User profile state — preferences, permissions, history that persist across sessions. Always in a database; always subject to the same access controls as the data the system retrieves.

Step 4: Interface Design

The AI Interface Contract

Every AI system exposes an interface to its users. The interface contract defines what the system accepts, what it returns, and what signals it provides about its own confidence and limitations. Getting the interface contract right is as important as getting the model right — a system with high model quality and a poor interface contract will fail in production because users cannot use it correctly.

Get Navaneethan L’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Input contract:

• Maximum input length (with graceful truncation, not silent failure)
• Supported input types (text only, or multimodal)
• Language support (single language, or multi-language with detection)
• Rate limits and quota behavior (what happens when the user exceeds the limit)

Output contract:

• Output format (free text, structured JSON, streaming tokens)
• Confidence signals (does the system express uncertainty? How?)
• Citation format (what sources does the system cite, in what format)
• Fallback behavior (what the system returns when it cannot answer)

Failure contract:

• How the system communicates that it cannot answer a question
• How the system communicates that it has low confidence
• How the system routes to human review when confidence is below threshold

The Confidence Signal Design Problem

Most AI systems do not communicate uncertainty well. They produce confident-sounding outputs regardless of their actual reliability, because the language model is trained to be fluent and fluency reads as confidence. Designing a meaningful confidence signal into the system interface is a non-trivial engineering problem.

Practical approaches:

• Retrieval-grounded confidence: if the system found relevant source material, it answers with citation; if it did not find relevant material, it explicitly says so rather than generating without grounding
• Abstention design: train or prompt the system to produce a specific abstention response when the question is outside its knowledge scope, and detect that response in the output parser
• LLM-as-judge confidence scoring: a second model call evaluates the output and assigns a confidence score; responses below a threshold route to human review

class ConfidenceAwareResponse:
    answer: str
    confidence: float        # 0.0 - 1.0
    source_quality: str      # "grounded" | "partial" | "ungrounded"
    citations: list[Citation]
    requires_human_review: bool

def should_route_to_human(response: ConfidenceAwareResponse,
                           use_case_threshold: float) -> bool:
    return (
        response.confidence < use_case_threshold
        or response.source_quality == "ungrounded"
        or response.requires_human_review
    )

Authorization at the Right Layer

One serious class of RAG failure is applying authorization only after retrieval. Authorize before you retrieve — not after.

This is a common and serious design error. A system that retrieves all relevant documents and then filters by authorization may expose the existence of restricted documents (through metadata in error messages or response latency differences) even when it correctly withholds their content. Authorization must be applied as a filter on the retrieval query — before any document is fetched — so that restricted documents are never retrieved into the context at all.

async def retrieve_authorized(
    query: str,
    user: User,
    vector_store: VectorStore
) -> list[Document]:
    """Authorization applied at retrieval time, not after."""
    # Build authorization filter from user permissions
    auth_filter = build_permission_filter(user.permissions)

    # Pass filter to vector store — only authorized docs are retrieved
    return await vector_store.search(
        query=query,
        filter=auth_filter,   # Applied BEFORE retrieval, not after
        top_k=10
    )

Step 5: Failure Mode Analysis

This is the step most teams skip. Every AI system has a specific set of failure modes that follow from its architecture. Enumerating them before deployment — and designing detection and mitigation for each — is what separates production-grade systems from demos that break under real-world conditions.

The AI System Failure Mode Taxonomy

A system-level taxonomy identifies fifteen hidden failure modes in production LLM systems including multi-step reasoning drift, latent inconsistency, context-boundary degradation, incorrect tool invocation, version drift, and cost-driven performance collapse.

The practical subset for most enterprise AI systems:

Input failures:

• Prompt injection: malicious instructions embedded in user input or retrieved documents
• Input too long: user sends input exceeding context limits
• Off-distribution input: input type the system was not designed for (wrong language, unexpected format)
• Authorization bypass attempt: user attempts to retrieve documents they should not access

Retrieval failures:

• Relevant document not found: the knowledge base does not contain the answer
• Wrong document retrieved: high retrieval similarity but low semantic relevance
• Stale document: the retrieved document is outdated and the answer has changed
• Cross-tenant data leak: authorization failure allows retrieval of another user’s documents

Model failures:

• Hallucination: model generates plausible but incorrect information
• Instruction non-compliance: model ignores formatting or content instructions
• Schema non-conformance: structured output does not match expected format
• Version drift: model behavior changes after provider update

Output failures:

• Downstream parse failure: output format cannot be parsed by consuming service
• Policy violation: output contains content that violates content policy
• PII leak: output contains personal information from retrieved documents

Operational failures:

• Provider outage: LLM API unavailable
• Latency spike: model inference takes 10× normal time
• Cost runaway: unexpected usage pattern drives cost beyond budget
• Cascading failure: upstream system failure propagates through the AI pipeline

For each failure mode, the design document should specify:

1. Detection signal: how does the system know this is happening?
2. Mitigation: what does the system do when it detects this?
3. User experience: what does the user see when this occurs?
4. Alert: who is notified, and how urgently?

FAILURE_MODE_REGISTRY = {
    "schema_non_conformance": FailureMode(
        detection="JSON schema validation on every output",
        mitigation="retry once; if still non-conforming, return structured error",
        user_experience="'I was unable to produce a correctly structured response. Please rephrase.'",
        alert=AlertConfig(severity="warning", channel="slack-ai-ops")
    ),
    "retrieval_empty": FailureMode(
        detection="retrieved_docs count == 0",
        mitigation="answer from model knowledge; set source_quality='ungrounded'",
        user_experience="answer with explicit caveat: 'I did not find relevant documents...'",
        alert=AlertConfig(severity="info", channel=None)
    ),
    "provider_outage": FailureMode(
        detection="circuit breaker open on primary provider",
        mitigation="failover to secondary provider from fallback chain",
        user_experience="transparent to user if failover succeeds",
        alert=AlertConfig(severity="critical", channel="pagerduty")
    ),
}

Step 6: The Architecture Decision Record

Every significant architectural decision in an AI system should be documented in an Architecture Decision Record (ADR) — a short structured document that captures what was decided, why, what alternatives were considered, and what the consequences are.

AI system ADRs are particularly important because:

• AI architectural decisions have non-obvious performance, cost, and compliance implications
• The team that inherits the system in 12 months will not know why certain decisions were made
• Regulatory compliance (EU AI Act, SOC 2) increasingly requires documentation of design decisions for AI systems

ADR template for AI systems:

# ADR-[number]: [decision title]

## Status
[Proposed | Accepted | Deprecated | Superseded by ADR-X]

## Context
[What is the business requirement or technical constraint that drove this decision?]

## Decision
[What was decided?]

## Rationale
[Why this option over the alternatives? Include the specific requirements this satisfies and the trade-offs accepted.]

## Alternatives considered
[What else was evaluated? Why was each alternative not chosen?]

## Consequences
[What does this decision make easier? What does it make harder? What new requirements does it impose?]

## AI-specific considerations
[Model dependency: which model does this assume? Failure mode: what fails if this decision is wrong? Evaluation: how will we know if this decision is not serving us well? Governance: does this decision have compliance implications?]

The System Design Document

The output of the design process is a system design document that communicates the architecture clearly enough that a new engineer can understand it and implement it correctly. For AI systems, the design document has five sections beyond the standard software design document:

1. AI capability scope — what the system can and cannot do; what query types are in scope and out of scope
2. Model dependencies — which models the system uses, what behavior it depends on, and what the migration plan is if those models change
3. Failure mode catalog — the complete list of failure modes with their detection, mitigation, and alert configuration
4. Evaluation framework — the evaluation dataset, the metrics, the evaluation cadence, and the production quality gate
5. Human escalation design — every point where a human can or must review AI outputs, and the UX that makes that review possible

Architect’s Takeaways

• A demo proves nothing. Design for the full distribution of inputs, failures, scale, and operational demands — not for the expected case.
• AI system design requires two fault-tolerance layers: technical (infrastructure reliability) and semantic (correct model behavior). Most teams build the first and neglect the second.
• Roughly one in twenty production AI requests fail silently — returning outputs that appear correct but are not. Silent failure requires semantic monitoring, not just infrastructure monitoring.
• Requirements for AI systems must include quality thresholds, evaluation criteria, human escalation paths, data authorization requirements, and model version governance — not just functional capabilities.
• Conventionally well-formed requirements are not significantly associated with better AI design choices. The stakeholder-to-design translation step must be explicit and supported.
• Apply authorization at retrieval time, not after retrieval. Post-retrieval authorization is a security design error.
• The policy layer — guardrails, schema validation, HITL checkpoints, audit logging — is where most enterprise AI projects fail at scale. Design it first, not last.
• Every AI system failure mode requires a detection signal, a mitigation, a user experience, and an alert. Enumerate failure modes before deployment, not after.
• Document every significant AI architectural decision in an ADR with AI-specific considerations: model dependency, failure mode, evaluation, governance.
• Prefer fixed pipelines over agents until the task genuinely requires agent flexibility. Orchestration complexity is the most common source of production failures.