Building a Retrieval-Augmented Generation (RAG) Solution with Azure Storage Account and Cognitive Search
Retrieval-Augmented Generation (RAG) is quickly becoming the go-to architecture for building intelligent applications that combine Large Language Models (LLMs) with enterprise data. Instead of relying only on the LLM’s training data (which may be outdated or incomplete), RAG allows your model to ground its responses in your organization’s private data sources.
In this article, we’ll explore how to set up a RAG pipeline in Azure using:
• Azure Storage Account — to store your documents (PDFs, HTML, text, etc.)
• Azure AI Search (Cognitive Search) — to index and retrieve relevant content
• Azure OpenAI Service — to generate natural language responses
• Azure App Service (or Function Apps) — to expose the solution to users
Why RAG in Azure?
A plain LLM might generate hallucinations or fail to provide company-specific knowledge. With RAG, you enhance LLM responses by:
1. Retrieving relevant content from your own data sources.
2. Augmenting the prompt with this content.
3. Generating an informed, accurate response using Azure OpenAI.
Step 1: Ingesting Data with Azure Storage Account
The Azure Storage Account acts as the foundation of your data lake. You upload raw documents here — PDFs, Word files, text, or even web archives.
Best practice: organize data into containers (e.g., /policies/, /manuals/, /faqs/).
Step 2: Indexing with Azure Cognitive Search
Azure AI Search connects to your Storage Account through a data source and runs skillsets (such as OCR, text extraction, entity recognition). It creates a searchable index that can be queried by keywords or semantic search.
Get Charlesg’s stories in your inbox
Join Medium for free to get updates from this writer.
For RAG, you’ll usually enable:
• Vector embeddings (using Azure OpenAI’s embedding model, e.g., text-embedding-ada-002).
• Chunking of large documents to improve retrieval accuracy.
Step 3: Retrieval-Augmented Generation Flow
When a user asks a question:
1. The app sends the query to Azure Cognitive Search.
2. Cognitive Search returns the top-k most relevant chunks of data.
3. The application augments the user’s query with these chunks.
4. Azure OpenAI generates a grounded response.
Step 4: Building the Application Layer
You can integrate this RAG pipeline into:
• A web app (via Azure App Service or Static Web Apps)
• A bot (via Azure Bot Service)
• An API layer (via Azure Functions)
This makes your RAG-powered assistant available to end users.
Azure Architecture Diagram
Here’s an architecture diagram that illustrates the flow:
