Intelligent RAG-Based Chatbot Development

S — Situation

Our team relied heavily on scattered documentation — Confluence pages, configuration files, curl commands, system logs, and pod restart histories.
Engineers had to manually search across these sources, which was slow and error-prone.
During the Internship 2025 program, I proposed the idea of building an intelligent GenAI chatbot to make this information easily searchable through natural language.

T — Task

I was responsible for:

• Designing the end-to-end RAG architecture
• Developing the chatbot integrated into our UI
• Setting up Azure AI Foundry for model hosting and inference
• Building the backend pipeline for embeddings, retrieval, and model routing
• Mentoring an intern who contributed to the project
• Balancing all this with my regular scrum responsibilities

The goal was to deliver a reliable, accurate chatbot that could query internal knowledge, analyze logs, and provide system insights.

A — Action

I executed the project in several steps:

1. Designed the RAG architecture

• Built a consolidated knowledge base from Confluence pages, config docs, logs, and curl commands.
• Chunked and embedded documents using Azure AI Foundry’s embedding model.
• Stored vectors in Azure Cognitive Search for semantic retrieval.

2. Built the GenAI chatbot logic

• Integrated three LLMs (o3, o4-mini, grok-3-mini) for model routing based on query type.
• Implemented retrieval ranking, guardrail prompts, and confidence scoring.
• Developed backend APIs for querying, context assembly, and response generation.

3. NLP integration

• Added intent recognition and entity extraction.
• Enabled semantic search, document grounding, and structured responses.

4. UI Integration

• Exposed REST APIs and ensured real-time, seamless chatbot interactions in the application.

5. Mentorship

• Guided an intern through RAG concepts, embeddings, prompt engineering, and Azure AI Foundry workflows.
• Conducted weekly knowledge-sharing sessions and helped them implement parts of the pipeline.

R — Result

• Delivered a fully functional intelligent GenAI chatbot that provide accurate, context-aware responses.
• Reduced manual effort spent searching documentation and debugging issues by 60–70%.
• Improved access to pod restart history, error logs, and component details directly through conversational queries.
• Enabled high reliability with reduced hallucination through RAG and multi-model routing.
• Successfully mentored an intern, who completed their project goals and contributed meaningfully to the pipeline.
• Managed to deliver this initiative without impacting my regular sprint commitments.

We built a system that helps users query internal documentation, error logs, and pod restart information using natural language. The architecture has four major layers: data ingestion, vector store, LLM orchestration, and UI integration.

1. Data Ingestion & Knowledge Base Creation

We started by collecting all internal documentation — Confluence pages, configuration documents, curl command references, and relevant log/error data.

We cleaned this data, removed noise/duplicates, and then chunked it into manageable pieces (about 200–400 tokens). Each chunk was converted into an embedding using an embedding model hosted in Azure AI Foundry. These embeddings were stored in Azure Cognitive Search, along with metadata like file source, component ID, and timestamps.

2. Vector Store & Retrieval Layer

When a user asks a question, we convert the query into an embedding through Azure AI Foundry’s embedding API.
We perform a vector similarity search against Azure Cognitive Search to find the most relevant chunks. We used hybrid search (vector + keyword) because logs and config docs often contain unique tokens.

Chunks are ranked by similarity scores, and we select the top few to act as context for the LLM.

3. LLM Layer via Azure AI Foundry

This is where Azure AI Foundry played a central role.

We deployed three LLMs:

• o3 for fast/simple lookups
• o4-mini for complex reasoning
• grok-3-mini for logs, curl parsing, and technical analysis

Our backend routes queries to one of these models based on retrieval confidence and query type. We construct a prompt that includes:

• the retrieved document chunks
• the user’s question
• guardrails like “Use only the provided sources. If not found, say ‘Not found.’”

The selected LLM generates the final answer. We apply confidence scoring, validate the output, and attach source references before sending it back to the user.

4. UI Integration

The chatbot is integrated directly into our application UI. When a user asks a question:

• The UI sends the query to our backend
• Backend orchestrates embeddings → retrieval → LLM answer
• The final structured response is returned to the UI

The chatbot also supports features like:

• Fetching pod restart history
• Displaying new entries in the error viewer
• Searching functional component docs
• Summaries and troubleshooting steps

5. Monitoring & Optimization

We used Azure AI Foundry’s observability metrics for:

• token usage
• latency
• error rates
• cost tracking

We also implemented caching, prompt versioning, fallback models, and continuous evaluation to maintain high reliability and low hallucination.

Low-Level Technical Design of the RAG-Based Intelligent Chatbot System

Built an Intelligent Chatbot for Our Application :

We developed a chatbot that can be used inside the existing UI. The chatbot helps users by giving them:

• Information about functional components
• New entries from the error viewer
• Historical details about pod restarts
• Other system-related insights

So instead of searching manually, users can simply ask the chatbot.

Created a Knowledge Base From Multiple Sources

We collected all important information from:

• Confluence pages
• Configuration documents
• CURL commands
• Internal technical notes

From these, we built a comprehensive knowledge base that the chatbot can refer to

Cleaned and Prepared the Data

The raw data was messy. So we did:

• Data cleaning
• Formatting
• Splitting large documents into smaller chunks
• Converting text into vector embeddings This helped the model understand the content and retrieve accurate results.

What are Vector Embeddings ?

Vector embeddings are numerical representations of text that capture semantic meaning. Instead of storing text as raw strings, we convert each document chunk into a vector so that similar meanings are mathematically close to each other.

In RAG, embeddings are essential because they allow semantic search. When a user asks a question, we embed the query and compare it to all stored document embeddings using vector similarity. This retrieves the most relevant chunks even if the user uses different wording.

These retrieved chunks are then provided to the LLM, grounding the model’s answer in real data and significantly reducing hallucination.

1. Capturing Meaning :

Words that mean similar things have similar embeddings.

• “error” and “failure” → vectors close to each other
• “pod” and “container” → similar vectors
• “restart history” vs “pod reboot logs” → similar meaning → similar vectors

2. Searching With Meaning (Semantic Search)

When the chatbot receives a question: “Show me the pod restart history”

We convert this question into a vector and search for similar vectors in our knowledge base. This finds the most relevant document chunks.

How We Converted Text Data Into Vector Embeddings (In Our Project)

In our project, we did not manually write embedding algorithms.
We used Azure AI Foundry’s embedding model API — which internally uses a high-performance embedding model (like text-embedding-3-small).

So the flow was:

⭐ 1. Chunk text → send chunk to Azure embedding API

For each chunk of text (from Confluence, configs, logs):

We sent an API request:

POST /openai/deployments/<embedding_model>/embeddings

With the payload:

{
  "input": "<chunk of text>"
}

Azure returned a vector embedding, such as a 1536-dimensional vector.

⭐ 2. No need to write ML code — we used Azure’s embedding service

We did NOT:

✘ train a model
✘ implement deep learning layers
✘ write embedding algorithms
✘ handle tokenization manually

Azure’s model automatically:

✔ tokenizes the text
✔ encodes context
✔ produces semantic embeddings
✔ maintains consistency across chunks

⭐ 3. Libraries / SDKs Used (Depending on Implementation)

We use one of these:

✔ Azure OpenAI SDK (Python, Java, or JS)

Example (Python-like):

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=chunk_text
)
vector = response.data[0].embedding

Azure returns:

{
  "data": [
    {
      "embedding": [0.12, -0.45, 0.98, ...]
    }
  ]
}

✔ Stored in Azure Cognitive Search (Vector DB)

We inserted:

• chunk text
• metadata
• the embedding vector

⭐ 4. Why Azure AI Foundry Embedding Model?

Because it:

• is optimized for semantic search
• returns high-quality embeddings
• integrates naturally with Azure Cognitive Search
• simplifies embedding generation
• avoids maintaining ML infrastructure

What is RAG? (Retrieval-Augmented Generation)

RAG, or Retrieval-Augmented Generation, is a technique where the system first retrieves relevant information from an external knowledge base and then gives that information to an LLM to generate an accurate answer.

We start by chunking documents and converting each chunk into embeddings (vectors) and storing them in a vector database like Azure Cognitive Search.

When a user asks a question, we embed the query, search for the closest matching vectors in the database using semantic similarity, and retrieve the top relevant chunks. These retrieved chunks are then ‘augmented’ into the prompt sent to the LLM.

This ensures the model’s response is grounded in real company data, reduces hallucinations, and allows the LLM to answer questions it otherwise wouldn’t know

It is an AI technique that improves accuracy by combining:

1. Retrieval

• Search and fetch relevant information from a knowledge base
  (using embeddings + vector search)

2. Generation

• Use a Large Language Model (LLM) to generate an answer
  based on both:
• the retrieved information
• the user’s question

3. Augmented

• Supplementing the LLM with external retrieved knowledge before generation.

How RAG Works (Step-by-Step)

Below is the workflow you can explain in interviews:

🔹 Step 1 — Build a Knowledge Base

You collect all internal documents:

• Confluence pages
• Config files
• Curl commands
• System logs
• Pod restart history
• Documents are cleaned, chunked, and indexed.

🔹 Step 2 — Convert Documents to Embeddings

Each chunk is converted into a vector embedding using an embedding model (Azure Foundry embedding model).

These embeddings are stored in a vector database like Azure Cognitive Search.

🔹 Step 3 — User Asks a Question

Example:
“Show me the pod restart history for component X”

The query is also converted into an embedding.

🔹 Step 4 — Vector Search

The query embedding is compared to document embeddings using similarity search.

The most relevant chunks are retrieved:

• Pod restart logs
• Config references
• Error viewer entries
• Related Confluence notes

🔹 Step 5 — Build the Prompt

The system creates a structured prompt:

Use ONLY the following sources to answer the question:
[Chunk 1...]
[Chunk 2...]

Question: Show me the pod restart history for component X.

🔹 Step 6 — LLM Generates Final Answer

The retrieved chunks + user question are sent to an LLM (o3, o4-mini, or grok-3-mini).

The LLM creates:

• Accurate
• Context-aware
• Source-grounded
• Non-hallucinated
• answers.

🔹 Step 7 — Return to UI

The response is validated and displayed to the user in the chatbot UI.

Why RAG?

LLMs alone can hallucinate (make up answers).
RAG reduces hallucination by ensuring the model ONLY uses information from:

• Confluence pages
• System documents
• Logs
• Configurations
• Internal datasets

This is why RAG is widely used in enterprise chatbots.

RAG reduces hallucination by forcing the model to use real data.

Ref link : https://pureinsights.com/blog/2023/what-is-retrieval-augmented-generation-rag/

Simple Explanation : RAG is like giving the model the correct pages from a manual before it answers the question. Instead of relying only on the model’s memory, we show it the exact relevant documents, and then it uses that information to generate a precise answer.

Why Are Embeddings Used in RAG (Retrieval-Augmented Generation)?

RAG depends on finding the right information to send to the LLM. RAG works like this:

• Convert all documents into embeddings
• Convert the user’s query into an embedding
• Compare them
• Retrieve the closest-matching chunks
• Send them to the model for final answer

Without embeddings, the chatbot cannot perform semantic search or retrieve context accurately.

How Embeddings Are Created

Embeddings are generated by a special model called an embedding model.

Get Avinash Dogiparthi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Steps to create an embedding:

1. Input text : Example: “Pod restart history”

2. Tokenization : The text is broken into units called tokens (words/subwords). Example: [“Pod”, “restart”, “history”]

3. Model processes tokens

4. The embedding model (like text-embedding-3-small) uses neural network layers to understand:

• word meaning
• syntax
• relationships
• context

5. Outputs a vector

• The final layer outputs a long list of numbers (vector).
• Example: [0.12, -0.54, 0.88, ...]
• Usually 384, 768, or 1536 dimensions depending on the model.

These numbers encode semantic meaningSimilar texts produce vectors close to each other.

How Embedding Models Like text-embedding-3-small Work

text-embedding-3-small is a compact embedding model used widely for RAG.

Key characteristics:

• Outputs a vector of fixed size (e.g., 1536 dimensions)
• Designed specifically to capture semantic meaning
• Fast and cost-efficient

Works extremely well for:

• semantic search
• document retrieval
• classification
• clustering

How it works internally (simplified):

1. Input text → tokenization
2. Transformer architecture processes tokens
3. Self-attention layers understand relationships between words
4. Encoding layer generates dense vector
5. Captures meaning, synonyms, intent, context
6. Output vector returned

How Vector Search Works

Once documents are converted into embeddings, you store them in a vector database like:

• FAISS
• Pinecone
• Milvus
• Weaviate
• Azure Cognitive Search (Vector Store) : This is the vector database used in our project.

Vector search works like this:

1. User asks a question →
2. Convert question into an embedding →
3. Compare this vector with all stored document vectors →
4. Find the vectors closest to it
5. Return those document chunks as search results

How “closeness” is measured?

Using mathematical similarity:

• Cosine similarity (most common)
• Dot product
• Euclidean distance

Example:

Query: “How many times was the pod restarted?”

Closest embedding results may be:

• “pod restart history logs”
• “health check failed — restart count”
• “kubernetes container restarts”

These are passed to the LLM for the final answer.

How Chunking + Embeddings Help RAG

RAG = Retrieval-Augmented Generation
Chunking + embeddings form the backbone of RAG.

Why chunking is needed?

Chunking is the process of breaking large documents into smaller, meaningful pieces before storing them in a vector database. It’s necessary because embedding models and LLMs have token limits, and embedding an entire large document as one vector would make retrieval inaccurate.

By chunking documents (200–400 tokens each), we ensure that the vector database can return only the most relevant sections when the user asks a question.

Chunking improves retrieval accuracy, reduces hallucination, and ensures that the LLM receives the exact context needed to generate a reliable answer.

Each chunk gets its own embedding.

How it helps:

✔ Prevents the model from missing important info
✔ Makes vector search more accurate
✔ Reduces hallucination
✔ Improves retrieval speed
✔ Fits model context window limits

RAG flow with chunking + embeddings:

1. Chunk documents
2. Embed each chunk
3. Store vectors
4. Query → embed query
5. Vector search finds matching chunks
6. LLM uses these chunks as context
7. Model generates accurate answer

This ensures that the LLM bases answers on real organizational data, not guesses.

Implemented a RAG (Retrieval-Augmented Generation) Pipeline

RAG is a method where the model:

1. Searches the knowledge base
2. Retrieves relevant chunks
3. Uses the LLM to generate a final answer

We used small but powerful models like:

• o4-mini
• o3
• grok-3-mini

This ensures accuracy + reduced hallucination.

We used a three-tier model strategy in production: o3 for high-throughput classification and short factual lookups, o4-mini for higher-quality answer generation when retrieval confidence was lower, and grok-3-mini for developer-focused tasks such as parsing logs, curl requests, and pod diagnostics. Each model was routed based on retrieval confidence and query type, with prompt-engineered guardrails and provenance attribution to reduce hallucinations and ensure reliable responses.

Backend Setup Using Azure AI Foundry

We used Azure AI Foundry for:

• Hosting the model
• Sending queries via API
• Getting responses
• Running AI workflows

The pipeline works like this:

User → Chatbot UI → API Request → Azure Model → Knowledge Search → AI Response → UI Display

Built the AI Processing Logic

We implemented:

• Natural Language Processing (NLP)
• Intent understanding
• Entity extraction
• Semantic matching

This helps the system understand what exactly the user is asking.

Response Scoring and Selection

The backend generates multiple possible answers and assigns a confidence score to each.

We return the response with the highest confidence, ensuring accuracy and reliability.

How We Used Azure AI Foundry in Our GenAI Project

Azure AI Foundry was the backbone of our GenAI system. We used it to host and serve our LLMs (o3, o4-mini, grok-3-mini) through API endpoints. Our backend sent user queries to Foundry for embedding, retrieved relevant chunks from Azure Cognitive Search, and then used the LLM API to generate the final answer. Foundry handled model hosting, scaling, routing, observability, and token usage tracking — so we didn’t have to manage GPUs or infrastructure ourselves.

Let me break it down step by step:

1. Hosting and Serving the LLMs

We deployed multiple models on Azure AI Foundry:

• o4-mini → for complex reasoning queries
• o3 → for fast, simple queries
• grok-3-mini → for log/technical parsing

Azure AI Foundry exposed these models through REST APIs.

So our backend could call:

POST /openai/deployments/o4-mini/chat/completions

This allowed us to use the models in production without managing infrastructure.

2. Embedding Generation

We used embedding models from Azure AI Foundry to generate vector embeddings.

• Every document chunk
• Every section of a Confluence page
• Every curl command
• Every error snippet

…was converted into an embedding vector via an Azure Foundry embedding API.

Example:

POST /openai/deployments/embeddingmodel/embeddings

These embeddings were then stored in Azure Cognitive Search for retrieval.

3. Query Processing Through APIs

When a user typed a question in the UI, our backend sent the query to Azure AI Foundry’s endpoint:

1. The query text was embedded by the embedding model
2. The LLM was called with retrieved context
3. Azure returned a structured, high-confidence answer

This eliminated the need to self-host the model or manage GPUs.

4. RAG Pipeline Orchestration

Azure AI Foundry helped orchestrate the entire RAG workflow:

Step 1 — Embed the user query : We sent the user query to Foundry’s embedding model.

Step 2 — Retrieve relevant chunks. We used Azure Cognitive Search to get similar chunks.

Step 3 — Send context to the LLM

The selected chunks + the user question were sent to Azure Foundry for final generation.

Step 4 — Get model response

Azure AI Foundry returned the final answer that we displayed in the chatbot UI.

5. Model Routing with Azure Foundry

Based on query type and retrieval quality:

• Simple factual queries → o3
• Technical/log queries → grok-3-mini
• Complex reasoning → o4-mini

Azure AI Foundry allowed us to switch models dynamically using their deployment endpoints.

This helped us reduce cost and improve response time.

How did we utilize Azure AI Foundry in our Project

Azure AI Foundry acted as the backbone of our GenAI architecture.
We used it for model hosting, embeddings, inference, orchestration, routing, scaling, and monitoring.

⭐ 1. Hosted All Our LLM Deployments

We deployed multiple models on Azure AI Foundry:

• o3 → simple factual queries, high-speed lookup
• o4-mini → reasoning-heavy, complex answers
• grok-3-mini → log parsing, technical/debugging queries

Azure AI Foundry exposed these through REST API endpoints, like:

POST /openai/deployments/o4-mini/chat/completions

This allowed our backend to call the right model on demand, without managing GPUs or infrastructure ourselves.

⭐ 2. Generated Embeddings for RAG

We used Azure AI Foundry’s embedding model to convert:

• Document chunks
• Confluence pages
• curl command references
• Logs/error messages
• User queries

…into vector embeddings.

These embeddings were then stored in Azure Cognitive Search (Vector DB) for semantic retrieval.

⭐ 3. Ran the Full RAG Inference Pipeline

Azure AI Foundry was used to:

✔ Embed the query
✔ Receive retrieved document chunks
✔ Build the prompt
✔ Send prompt to the chosen LLM
✔ Generate the final answer

The workflow:

UI → Backend → Azure Foundry Embedding → Azure Cognitive Search → Azure Foundry LLM → UI

Azure handled the heavy compute.

⭐ 4. Model Routing Based on Query Type

We implemented dynamic model selection:

• If query = simple → use o3
• If logs/debugging → use grok-3-mini
• If complex reasoning → use o4-mini

Azure AI Foundry made this easy with separate deployment endpoints for each model.

⭐ 5. Monitoring, Logging & Cost Optimization

Azure AI Foundry gave us:

• Token usage metrics
• Latency monitoring
• Error tracking
• Deployment health
• Cost breakdown

This helped us optimize the system and reduce unnecessary LLM calls.

Challenges & How I Solved Them

1️⃣ Challenge: Unstructured and Scattered Documentation

Our data came from Confluence pages, config files, curl commands, and logs — all in different formats.

Solution:
I built a data ingestion pipeline that cleaned, normalized, and chunked documents into 200–400 token sections. Then I generated embeddings and stored them with metadata in Azure Cognitive Search.
This improved retrieval accuracy significantly.

2️⃣ Challenge: Retrieval Returning Irrelevant Chunks

Initially, vector search returned text that was semantically close but contextually incorrect.

Solution:

• Tuned chunk sizes and added overlap
• Implemented hybrid search (vector + keyword)
• Applied metadata filters (component, doc type, timestamps)
• Added reranking logic

This improved the relevance of retrieved chunks and reduced noise.

3️⃣ Challenge: LLM Hallucinations

Without grounding, the model sometimes generated confident but incorrect answers.

Solution:

• Added guardrail prompts: “Use ONLY the provided sources”
• Included citations and provenance
• Added a confidence score threshold
• Used model routing (o3 → simple, o4-mini → reasoning)

This reduced hallucinations dramatically.