Use any Python AI agent framework with free GitHub Models

I ❤️ when companies offer free tiers for developer services, since it gives everyone a way to learn new technologies without breaking the bank. Free tiers are especially important for students and people between jobs, where the desire to learn is high but the available cash is low.

That's why I'm such a fan of GitHub Models: free, high-quality generative AI models available to anyone with a GitHub account. The available models include the latest OpenAI LLMs (like o3-mini), LLMs from the research community (like Phi and Llama), LLMs from other popular providers (like Mistral and Jamba), multimodal models (like gpt-4o and llama-vision-instruct) and even a few embedding models (from OpenAI and Cohere). So cool! With access to such a range of models, you can prototype complex multi-model workflows to improve your productivity or heck, just make something fun for yourself. 🤗

To use GitHub Models, you can start off in no-code mode: open the playground for a model, send a few requests, tweak the parameters, and check out the answers. When you're ready to write code, select "Use this model". A screen will pop up where you can select a programming language (Python/JavaScript/C#/Java/REST) and select an SDK (which varies depending on model). Then you'll get instructions and code for that model, language, and SDK.

But here's what's really cool about GitHub Models: you can use them with all the popular Python AI frameworks, even if the framework has no specific integration with GitHub Models. How is that possible?

1. The vast majority of Python AI frameworks support the OpenAI Chat Completions API, since that API became a defacto standard supported by many LLM API providers besides OpenAI itself.
2. GitHub Models also provide OpenAI-compatible endpoints for chat completion models.
3. Therefore, any Python AI framework that supports OpenAI-like models can be used with GitHub Models as well. 🎉

To prove my claim, I've made a new repository with examples from eight different Python AI agent packages, all working with GitHub Models: python-ai-agent-frameworks-demos. There are examples for AutoGen, LangGraph, Llamaindex, OpenAI Agents SDK, OpenAI standard SDK, PydanticAI, Semantic Kernel, and SmolAgents. You can open that repository in GitHub Codespaces, install the packages, and get the examples running immediately.

Now let's walk through the API connection code for GitHub Models for each framework. Even if I missed your favorite framework, I hope my tips here will help you connect any framework to GitHub Models.

OpenAI sdk

I'll start with openai, the package that started it all!

import openai

client = openai.OpenAI(
  api_key=os.environ["GITHUB_TOKEN"],
  base_url="https://models.inference.ai.azure.com/")

The code above demonstrates the two key parameters we'll need to configure for all frameworks:

• api_key: When using OpenAI.com, you pass your OpenAI API key here. When using GitHub Models, you pass in a Personal Access Token (PAT). If you open the repository (or any repository) in GitHub Codespaces, a PAT is already stored in the GITHUB_TOKEN environment variable. However, if you're working locally with GitHub Models, you'll need to generate a PAT yourself and store it. PATs expire after a while, so you need to generate new PATs every so often.
• base_url: This parameter tells the OpenAI client to send all requests to "https://models.inference.ai.azure.com/" instead of the OpenAI.com API servers. That's the domain that hosts the OpenAI-compatible endpoint for GitHub Models, so you'll always pass that domain as the base URL.

If we're working with the new openai-agents SDK, we use very similar code, but we must use the AsyncOpenAI client from openai instead. Lately, Python AI packages are defaulting to async, because it's so much better for performance.

import agents
import openai

client = openai.AsyncOpenAI(
  base_url="https://models.inference.ai.azure.com/",
  api_key=os.environ["GITHUB_TOKEN"])

spanish_agent = agents.Agent(
    name="Spanish agent",
    instructions="You only speak Spanish.",
    model=OpenAIChatCompletionsModel(model="gpt-4o", openai_client=client))

PydanticAI

Now let's look at all of the packages that make it really easy for us, by allowing us to directly bring in an instance of either OpenAI or AsyncOpenAI.

For PydanticAI, we configure an AsyncOpenAI client, then construct an OpenAIModel object from PydanticAI, and pass that model to the agent:

import openai
import pydantic_ai
import pydantic_ai.models.openai


client = openai.AsyncOpenAI(
    api_key=os.environ["GITHUB_TOKEN"],
    base_url="https://models.inference.ai.azure.com/")

model = pydantic_ai.models.openai.OpenAIModel(
    "gpt-4o", provider=OpenAIProvider(openai_client=client))

spanish_agent = pydantic_ai.Agent(
    model,
    system_prompt="You only speak Spanish.")

Semantic Kernel

For Semantic Kernel, the code is very similar. We configure an AsyncOpenAI client, then construct an OpenAIChatCompletion object from Semantic Kernel, and add that object to the kernel.

import openai
import semantic_kernel.connectors.ai.open_ai
import semantic_kernel.agents

chat_client = openai.AsyncOpenAI(
  api_key=os.environ["GITHUB_TOKEN"],
  base_url="https://models.inference.ai.azure.com/")

chat_completion_service = semantic_kernel.connectors.ai.open_ai.OpenAIChatCompletion(
  ai_model_id="gpt-4o",
  async_client=chat_client)

kernel.add_service(chat_completion_service)
  
spanish_agent = semantic_kernel.agents.ChatCompletionAgent(
  kernel=kernel,
  name="Spanish agent"
  instructions="You only speak Spanish")

AutoGen

Next, we'll check out a few frameworks that have their own wrapper of the OpenAI clients, so we won't be using any classes from openai directly.

For AutoGen, we configure both the OpenAI parameters and the model name in the same object, then pass that to each agent:

import autogen_ext.models.openai
import autogen_agentchat.agents

client = autogen_ext.models.openai.OpenAIChatCompletionClient(
  model="gpt-4o",
  api_key=os.environ["GITHUB_TOKEN"],
  base_url="https://models.inference.ai.azure.com/")

spanish_agent = autogen_agentchat.agents.AssistantAgent(
    "spanish_agent",
    model_client=client,
    system_message="You only speak Spanish")

LangGraph

For LangGraph, we configure a very similar object, which even has the same parameter names:

import langchain_openai
import langgraph.graph

model = langchain_openai.ChatOpenAI(
  model="gpt-4o",
  api_key=os.environ["GITHUB_TOKEN"],
  base_url="https://models.inference.ai.azure.com/", 
)

def call_model(state):
    messages = state["messages"]
    response = model.invoke(messages)
    return {"messages": [response]}

workflow = langgraph.graph.StateGraph(MessagesState)
workflow.add_node("agent", call_model)

SmolAgents

Once again, for SmolAgents, we configure a similar object, though with slightly different parameter names:

import smolagents

model = smolagents.OpenAIServerModel(
  model_id="gpt-4o",
  api_key=os.environ["GITHUB_TOKEN"],
  api_base="https://models.inference.ai.azure.com/")
  
agent = smolagents.CodeAgent(model=model)

Llamaindex

I saved Llamaindex for last, as it is the most different. The Llamaindex Python package has a different constructor for OpenAI.com versus OpenAI-like servers, so I opted to use that OpenAILike constructor instead. However, I also needed an embeddings model for my example, and the package doesn't have an OpenAIEmbeddingsLike constructor, so I used the standard OpenAIEmbedding constructor.

import llama_index.embeddings.openai
import llama_index.llms.openai_like
import llama_index.core.agent.workflow

Settings.llm = llama_index.llms.openai_like.OpenAILike(
  model="gpt-4o",
  api_key=os.environ["GITHUB_TOKEN"],
  api_base="https://models.inference.ai.azure.com/",
  is_chat_model=True)

Settings.embed_model = llama_index.embeddings.openai.OpenAIEmbedding(
  model="text-embedding-3-small",
  api_key=os.environ["GITHUB_TOKEN"],
  api_base="https://models.inference.ai.azure.com/")

agent = llama_index.core.agent.workflow.ReActAgent(
  tools=query_engine_tools,
  llm=Settings.llm)

Choose your models wisely!

In all of the examples above, I specified the "gpt-4o" model. The "gpt-4o" model is a great choice for agents because it supports function calling, and many agent frameworks only work (or work best) with models that natively support function calling.

Fortunately, GitHub Models includes multiple models that support function calling, at least in my basic experiments:

• gpt-4o
• gpt-4o-mini
• o3-mini
• AI21-Jamba-1.5-Large
• AI21-Jamba-1.5-Mini
• Codestral-2501
• Cohere-command-r
• Ministral-3B
• Mistral-Large-2411
• Mistral-Nemo
• Mistral-small

You might find that some models work better than others, especially if you're using agents with multiple tools. With GitHub Models, it's very easy to experiment and see for yourself, by simply changing the model name and re-running the code.

So, have you started prototyping AI agents with GitHub Models yet?! Go on, experiment, it's fun!

Building a streaming DeepSeek-R1 app on Azure

This year, we're seeing the rise in "reasoning models", models that include an additional thinking process in order to generate their answer. Reasoning models can produce more accurate answers and can answer more complex questions. Some of those models, like o1 and o3, do the reasoning behind the scenes and only report how many tokens it took them (quite a few!).

The DeepSeek-R1 model is interesting because it reveals its reasoning process along the way. When we can see the "thoughts" of a model, we can see how we might approach the question ourself in the future, and we can also get a better idea for how to get better answers from that model. We learn both how to think with the model, and how to think without it.

So, if we want to build an app using a transparent reasoning model like DeepSeek-R1, we ideally want our app to have special handling for the thoughts, to make it clear to the user the difference between the reasoning and the answer itself. It's also very important for a user-facing app to stream the response, since otherwise a user will have to wait a very long time for both the reasoning and answer to come down the wire.

Here's an app with streamed, collapsible thoughts:

You can deploy that app yourself from github.com/Azure-Samples/deepseek-python today, or you can keep reading to see how it's built.

Deploying DeepSeek-R1 on Azure

We first deploy a DeepSeek-R1 model on Azure, using Bicep files (infrastructure-as-code) that provision a new Azure AI Services resource with the DeepSeek-R1 deployment. This deployment is what's called a "serverless model", so we only pay for what we use (as opposed to dedicated endpoints, where the pay is by hour).

var aiServicesNameAndSubdomain = '${resourceToken}-aiservices'
module aiServices 'br/public:avm/res/cognitive-services/account:0.7.2' = {
  name: 'deepseek'
  scope: resourceGroup
  params: {
    name: aiServicesNameAndSubdomain
    location: aiServicesResourceLocation
    tags: tags
    kind: 'AIServices'
    customSubDomainName: aiServicesNameAndSubdomain
    sku: 'S0'
    publicNetworkAccess: 'Enabled'
    deployments: [
      {
        name: aiServicesDeploymentName
        model: {
          format: 'DeepSeek'
          name: 'DeepSeek-R1'
          version: '1'
        }
        sku: {
          name: 'GlobalStandard'
          capacity: 1
        }
      }
    ]
    disableLocalAuth: disableKeyBasedAuth
    roleAssignments: [
      {
        principalId: principalId
        principalType: 'User'
        roleDefinitionIdOrName: 'Cognitive Services User'
      }
    ]
  }
}

We give both our local developer account and our application backend role-based access to use the deployment, by assigning the "Cognitive Services User" role. That allows us to connect using keyless authentication, a much more secure approach than API keys.

Connecting to DeepSeek-R1 on Azure from Python

We have a few different options for making API requests to a DeepSeek-R1 serverless deployment on Azure:

• HTTP calls, using the Azure AI Model Inference REST API and a Python package like requests or aiohttp
• Azure AI Inference client library for Python, a package designed especially for making calls with that inference API
• OpenAI Python API library, which is focused on supporting OpenAI models but can also be used with any models that are compatible with the OpenAI HTTP API, which includes Azure AI models like DeepSeek-R1
• Any of your favorite Python LLM packages that have support for OpenAI-compatible APIs, like Langchain, Litellm, etc.

I am using the openai package for this sample, since that's the most familiar amongst Python developers. As you'll see, it does require a bit of customization to point that package at an Azure AI inference endpoint. We need to change:

• Base URL: Instead of pointing to openai.com server, we'll point to the deployed serverless endpoint which looks like "https://<resource-name>.services.ai.azure.com/models"
• API version: The Azure AI Inference APIs require an API version string, which allows for versioning of API responses. You can see that API version in the API reference. In the REST API, it is passed as a query parameter, so we will need the openai package to send it along as a query parameter as well.
• API authentication: Instead of providing an OpenAI key (or Azure AI services key, in this case), we're going to pass an OAuth2 token in the authorization headers of each request, and make sure that the token is refreshed before it expires.

Setting up the keyless API authentication can be a bit tricky! First, we need to acquire a token provider for our current credential, using the azure-identity package:

from azure.identity.aio import AzureDeveloperCliCredential, ManagedIdentityCredential, get_bearer_token_provider

if os.getenv("RUNNING_IN_PRODUCTION"):
  azure_credential = ManagedIdentityCredential(
      client_id=os.environ["AZURE_CLIENT_ID"])
else:
  azure_credential = AzureDeveloperCliCredential(
      tenant_id=os.environ["AZURE_TENANT_ID"])

token_provider = get_bearer_token_provider(
  azure_credential, "https://cognitiveservices.azure.com/.default"
)

That code uses either ManagedIdentityCredential when it's running in production (on Azure Container Apps, with a user-assigned identity) or AzureDeveloperCliCredential when it's running locally. The token_provider function returns a token string every time we call it

For the next step, it helps to understand a bit about how the OpenAI package works. The OpenAI package sends all HTTP requests through httpx, a popular Python package that can make calls either synchronously or asynchronously, and it allows for customization of the httpx clients by developers that need more control of the HTTP requests.

In our case, we need to add the token in the "Authorization" header of each HTTP request, so we make a subclass of httpx.Auth that sets the header on each asynchronous request by calling the token provider function:

class TokenBasedAuth(httpx.Auth):
  async def async_auth_flow(self, request):
    token = await openai_token_provider()
    request.headers["Authorization"] = f"Bearer {token}"
    yield request

  def sync_auth_flow(self, request):
    raise RuntimeError("Cannot use a sync authentication class with httpx.AsyncClient")

Each time the token provider function is called, it will make sure that the token has not yet expired, and fetch a new one as necessary.

Now we can create a AsyncOpenAI client by passing in a custom httpx client using that TokenBasedAuth class, along with the correct base URL and API version:

from openai import AsyncOpenAI

openai_client = AsyncOpenAI(
  base_url=os.environ["AZURE_INFERENCE_ENDPOINT"],
  default_query={"api-version": "2024-05-01-preview"},
  api_key="placeholder",
  http_client=DefaultAsyncHttpxClient(auth=TokenBasedAuth()),
)

Making chat completion requests

When we receive a new question from the user, we use that OpenAI client to call the chat completions API:

chat_coroutine = openai_client.chat.completions.create(
   model=os.getenv("AZURE_DEEPSEEK_DEPLOYMENT"),
   messages=all_messages,
   stream=True)

You'll notice that instead of the typical model name that we send in when using OpenAI, we send in the deployment name. For convenience, I often name deployments the same as the model, so that they will match even if I mistakenly pass in the model name.

Streaming the response from the backend

As I've discussed previously on this blog, we should always use streaming responses when building user-facing chat applications, to reduce perceive latency and improve the user experience.

To receive a streamed response from the chat completions API, we specified stream=True in the call above. Then, as we receive each event from the server, we check whether the content is the special "<think>" start token or "</think>" end token. When we know the model is currently in a thinking mode, we pass down the content chunks in a "reasoning_content" field. Otherwise, we pass down the content chunks in the "content" field. 

We send each event to our frontend using a common approach of JSON-lines over a streaming HTTP response (which has the "Transfer-encoding: chunked" header). That means the client receives a JSON separated by a new line for each event, and can easily parse them out. The other common approaches are server-sent events or websockets, but both are unnecessarily complex for this scenario.

is_thinking = False
async for update in await chat_coroutine:
    if update.choices:
        content = update.choices[0].delta.content
        if content == "":
            is_thinking = True
            update.choices[0].delta.content = None
            update.choices[0].delta.reasoning_content = ""
        elif content == "":
            is_thinking = False
            update.choices[0].delta.content = None
            update.choices[0].delta.reasoning_content = ""
        elif content:
            if is_thinking:
                yield json.dumps(
                    {"delta": {"content": None, "reasoning_content": content, "role": "assistant"}},
                    ensure_ascii=False,
                ) + "\n"
            else:
                yield json.dumps(
                    {"delta": {"content": content, "reasoning_content": None, "role": "assistant"}},
                    ensure_ascii=False,
                ) + "\n"

Rendering the streamed response in the frontend

The frontend code makes a standard fetch() request to the backend route, passing in the message history:

const response = await fetch("http://blog.pamelafox.org/chat/stream", {
    method: "POST",
    headers: {"Content-Type": "application/json"},
    body: JSON.stringify({messages: messages})
});

r

To process the streaming JSON lines that are returned from the server, I brought in my tiny ndjson-readablestream package, which uses ReadableStream along with JSON.parse to make it easy to iterate over each JSON object as it comes in. When I see that the JSON is "reasoning_content", I display it in a special collapsible container.

let answer = "";
let thoughts = "";
for await (const event of readNDJSONStream(response.body)) {
    if (!event.delta) {
        continue;
    }
    if (event.delta.reasoning_content) {
        thoughts += event.delta.reasoning_content;
        if (thoughts.trim().length > 0) {
            // Only show thoughts if they are more than just whitespace
            messageDiv.querySelector(".loading-bar").style.display = "none";
            messageDiv.querySelector(".thoughts").style.display = "block";
            messageDiv.querySelector(".thoughts-content").innerHTML = converter.makeHtml(thoughts);
        }
    } else {
        messageDiv.querySelector(".loading-bar").style.display = "none";
        answer += event.delta.content;
        messageDiv.querySelector(".answer-content").innerHTML = converter.makeHtml(answer);
    }
    messageDiv.scrollIntoView();
    if (event.error) {
        messageDiv.innerHTML = "Error: " + event.error;
    }
}

All together now

The full code is available in github.com/Azure-Samples/deepseek-python. Here are the key files for the code snippeted in this blog post:

File	Purpose
infra/main.bicep	Bicep files for deployment
src/quartapp/chat.py	Quart app with the client setup and streaming chat route
src/quartapp/templates/index.html	Webpage with HTML/JS for rendering stream

Evaluating gpt-4o-mini vs. gpt-3.5-turbo for RAG applications

The azure-search-openai-demo repository was first created in March 2023 and is now the most popular RAG sample solution for Azure. Since the world of generative AI changes so rapidly, we've made many upgrades to its underlying packages and technologies over the past two years. But we've never changed the default GPT model used for the RAG flow: gpt-35-turbo.

Why, when there are new models that are cheaper and reportedly better, such as gpt-4o-mini? Well, changing the model is one of the most significant changes you can make to impact RAG answer quality, and I did not want to make the change without thorough evaluation.

Good news! I have now run several bulk evaluations on different RAG knowledge bases, and I feel fairly confident that a switch to gpt-4o-mini is a positive overall change, with some caveats. In my evaluations, gpt-4o-mini generates answers with comparable groundedness and relevance. The time-per-token is slightly less, but the answers are 50% longer on average, thus they take 45% more time for generation. The additional answer length often provides additional details based off the context, especially for questions where the answer is a list or a sequential process. The gpt-4o-mini per-token pricing is about 1/3 of gpt-35-turbo pricing, which works out to a lower overall cost.

Let's dig into the results more in this post.

Evaluation results

I ran bulk evaluations on two knowledge bases, starting with the sample data that we include in the repository, a bunch of invented HR documents for a fictitious company. Then, since I always like to evaluate knowledge that I know deeply, I also ran evaluations on a search index composed entirely of my own blog posts from this very blog.

Here are the results for the HR documents, for 50 Q/A pairs:

metric	stat	gpt-35-turbo	gpt-4o-mini
gpt_groundedness	pass_rate	0.98	0.98
	mean_rating	4.94	4.9
gpt_relevance	pass_rate	0.98	0.96
	mean_rating	4.42	4.54
answer_length	mean	667.7	934.36
latency	mean	2.96	3.8
citations_matched	rate	0.45	0.53
any_citation	rate	1.0	1.0

For that evaluation, groundedness was essentially the same (and was already very high), relevance only increased in its average rating (but not pass rate, which is the percentage of 4/5 scores), but we do see an increase in the number of citations in the answer that match the citations from the ground truth. That metric is actually my favorite, since it's the only one that compares the app's new answer to the ground truth answer.

Here are the results for my blog, for 200 Q/A pairs:

metric	stat	gpt-35-turbo	gpt-4o-mini
gpt_groundedness	pass_rate	0.97	0.95
	mean_rating	4.89	4.8
gpt_relevance	pass_rate	0.89	0.94
	mean_rating	4.04	4.25
answer_length	mean	402.24	663.34
latency	mean	2.74	3.27
citations_matched	rate	0.8	0.8
any_citation	rate	1.0	0.96

For this evaluation, we actually see a slight decrease in groundedness, an increase in relevance (both the average rating and pass rate), and the same percentage of citations matched from the ground truth.

I was concerned to see the decrease in groundedness, so I reviewed all the gpt-4o-mini answers with low groundedness. Almost all of them were variations of "I don't know." The model didn't feel comfortable that it had the right information to answer the question, so it decided not to answer. As I've discussed here in a previous blog post, that's a good thing! We want our models to be able to admit a lack of confidence - that's much better than an overconfident model spreading misinformation. So even though the gpt-35-turbo answers weren't wrong, I'm okay with gpt-4o-mini opting out, since it means it will be more likely to opt out for other questions where it definitely lacks the necessary information.

Why are the answers wordier?

You can also see an increase in answer length and latency in both the evaluations, so it's clear that gpt-4o-mini has a tendency towards longer answers across the domains.

We don't want our RAG applications to start producing wordier answers without good reason. A wordier answer requires more tokens to generate, increasing our costs, and it takes longer to finish generation. Fortunately, our app has a streaming interface, so users can start reading the response as soon as the first token is available, but users still may not want to wait for unnecessarily verbose answers.

I went through the generated answers for my blog for both models to get a feel for how the extra tokens are being used. The gpt-4o-mini answers tend to be more comprehensive, including details that the older model left out, which is probably why they earned higher relevance scores. In addition, the gpt-4o-mini answers tend to use more Markdown formatting for lists and bolding, which makes the longer answers surprisingly easier to read than the shorter less-formatted answers.

I'll share a few examples here so you can see what I mean:

Question #1: What are the options for a Web Developer to combine coding and teaching?

gpt-35-turbo

gpt-4o-mini

Some options for a Web Developer to combine coding and teaching include: Developer relations: Working with API companies to help teach their APIs [girldevelopit-sf-thanks-for-great-2012.html]. Engineering in EdTech: Creating online educational experiences in the field of educational technology [combining-coding-and-teaching-into.html]. Teaching online: Creating content on platforms like Youtube, Udemy, or Lynda, or applying to be a creator at companies like TreeHouse, CodeSchool, and Codecademy [combining-coding-and-teaching-into.html].

A Web Developer can combine coding and teaching in several ways: Developer Relations: Work with API companies to help teach their APIs [combining-coding-and-teaching-into.html]. Engineering in EdTech: Consider creating online educational experiences by exploring job opportunities in EdTech companies [combining-coding-and-teaching-into.html]. Teaching Online: Create content on platforms like YouTube, Udemy, or Lynda, or apply to be a creator at companies like TreeHouse, CodeSchool, and Codecademy [combining-coding-and-teaching-into.html]. Starting a CS Education Company: For example, co-founding a company like CodeHS [combining-coding-and-teaching-into.html]. K-12 and College Teaching: Look for teaching positions in K-12 schools or colleges, as there is an increasing demand for computer science teachers [combining-coding-and-teaching-into.html]. Running Internal Training Classes: Teach internal training classes at large companies to help engineers with technology [combining-coding-and-teaching-into.html]. Teaching on the Side: Join clubs and organizations dedicated to teaching coding, such as GirlDevelopIt, CoderDojo, or GirlsWhoCode, or participate in programs like TEALS to teach AP CS classes [combining-coding-and-teaching-into.html].

Question #2: What does .editorconfig do?

gpt-35-turbo

gpt-4o-mini

The .editorconfig file is used to set coding style preferences for a project, such as indent style and size. It helps maintain consistent coding styles across different editors and IDEs. [porting-project-from-spaces-to-tabs.html]

An .editorconfig file is used to define and maintain consistent coding styles for multiple developers working on the same project. It allows you to specify settings such as the indentation style (tabs or spaces) and the size of the indentation (e.g., 4 spaces for tabs). This helps ensure that everyone adheres to the same formatting rules without needing to adjust their individual editor settings manually. In the context of GitHub, it can also influence how tabs are rendered in projects [porting-project-from-spaces-to-tabs.html].

Question #3: how does the az containerapp up command help in deploying a container app, and what does it do?

gpt-35-turbo

gpt-4o-mini

The az containerapp up command helps in deploying a container app by creating an Azure resource group, building the container image using the local Dockerfile, creating a registry in the Azure Container Registry and pushing the image to the registry, and finally creating a Container App that uses the pushed image and allows external ingress on a specified port [deploying-containerized-fastapi-app-to.html].

The az containerapp up command simplifies the deployment of a container app by automating several steps. When executed, it performs the following actions: Creates an Azure resource group (e.g., "flask-aca-rg" or "fastapi-aca-rg") to organize resources. Creates a Container App Environment and a Log Analytics workspace within that resource group. Builds the container image using the local Dockerfile. Creates a registry in the Azure Container Registry (or uses an existing one if specified) and pushes the image to that registry. Creates a Container App (e.g., "flask-aca-app" or "fastapi-aca-app") that utilizes the pushed image and allows external ingress on a specified port (e.g., port 5000 for Flask or port 80 for FastAPI). Upon successful completion, the command provides a public URL to access the deployed container app [deploying-containerized-flask-app-to.html][deploying-containerized-fastapi-app-to.html].

Those answers are fairly representative of the differences. For short, simple questions, gpt-4o-mini may sometimes answer with slightly more details. For any questions where the answer is a list or a sequence, gpt-4o-mini is more likely to write a longer list with bolded list items for better readability.

Next steps

I will send a PR to azure-search-openai-demo to default the model to gpt-4o-mini, and once merged, I'll note in the release notes that developers may see longer response lengths with the new model. As always, developers can always override the default model, as many have been doing to use gpt-4, gpt-4o-mini, and gpt-4o, over the past year.

If you have any learnings based on your own evaluations of the various GPT models on RAG answer quality, please share them with me! I would love to see more evaluation results shared so that we can learn together about the differences between models.

Safety evaluations for LLM-powered apps

When we build apps on top of Large Language Models, we need to evaluate the app responses for quality and safety. When we evaluate the quality of an app, we're making sure that it provides answers that are coherent, clear, aligned to the user's needs, and in the case of many applications: factually accurate. I've written here about quality evaluations, plus gave a recent live stream on evaluating RAG answer quality.

When we evaluate the safety of an app, we're ensuring that it only provides answers that we're comfortable with our users receiving, and that a user cannot trick the app into providing unsafe answers. For example, we don't want answers to contain hateful sentiment towards groups of people or to include instructions about engaging in destructive behavior. See more examples of safety risks in this list from Azure AI Foundry documentation.

Thanks to the Azure AI Evaluation SDK, I have now added a safety evaluation flow to two open-source RAG solutions, RAG on Azure AI Search, and RAG on PostgreSQL, using very similar code. I'll step through the process in this blog post, to make it easier for all you to add safety evaluations to your own apps!

The overall steps for safety evaluation:

1. Provision an Azure AI Project
2. Configure the Azure AI Evaluation SDK
3. Simulate app responses with AdversarialSimulator
4. Evaluate the responses with ContentSafetyEvaluator

Provision an Azure AI Project

We must have an Azure AI Project in in order to use the safety-related functionality from the Azure AI Evaluation SDK, and that project must be in one of the regions that support the safety backed service.

Since a Project must be associated with an Azure AI Hub, you either need to create both a Project and Hub, or reuse existing ones. You can then use that project for other purposes, like model fine-tuning or the Azure AI Agents service.

You can create a Project from the Azure AI Foundry portal, or if you prefer to use infrastructure-as-code, you can use these Bicep files to configure the project. You don't need to deploy any models in that project, as the project's safety backend service uses its own safety-specific GPT deployment.

Configure the Azure AI Evaluation SDK

The Azure AI Evaluation SDK is currently available in Python as the azure-ai-evaluation package, or in .NET as the Microsoft.Extensions.AI.Evaluation. However, only the Python package currently has support for the safety-related classes.

First we must either add the azure-ai-evaluation Python package to our requirements file, or install it directly into the environment:

pip install azure-ai-evaluation

Then we create a dict in our Python file with all the necessary details about the Azure AI project - the subscription ID, resource group, and project name. As a best practice, I store those values environment variables:

from azure.ai.evaluation import AzureAIProject

azure_ai_project: AzureAIProject = {
        "subscription_id": os.environ["AZURE_SUBSCRIPTION_ID"],
        "resource_group_name": os.environ["AZURE_RESOURCE_GROUP"],
        "project_name": os.environ["AZURE_AI_PROJECT"],
    }

Simulate app responses with AdversarialSimulator

Next, we use the AdversarialSimulator class to simulate users interacting with the app in the ways most likely to produce unsafe responses.

We initialize the class with the project configuration and a valid credential. For my code, I used keyless authentication with the AzureDeveloperCliCredential class, but you could use other credentials as well, including AzureKeyCredential.

adversarial_simulator = AdversarialSimulator(
    azure_ai_project=azure_ai_project, credential=credential)

Then we run the simulator with our desired scenario, language, simulation count, randomization seed, and a callback function to call our app:

from azure.ai.evaluation.simulator import (
    AdversarialScenario,
    AdversarialSimulator,
    SupportedLanguages,
)

outputs = await adversarial_simulator(
  scenario=AdversarialScenario.ADVERSARIAL_QA,
  language=SupportedLanguages.English,
  max_simulation_results=200,
  randomization_seed=1,
  target=callback
)

The SDK supports multiple scenarios. Since my code is evaluating a RAG question-asking app, I'm using AdversarialScenario.ADVERSARIAL_QA. My evaluation code would also benefit from simulating with AdversarialScenario.ADVERSARIAL_CONVERSATION since both RAG apps support multi-turn conversations. Use the scenario that matches your app.

For the AdversarialScenario.ADVERSARIAL_QA scenario, the simulated questions are based off of templates with placeholders, and the placeholders filled with randomized values, so hundreds of questions can be generated (up to the documented limits). Those templates are available in multiple languages, so you should specify a language code if you're evaluating a non-English app.

We use the max_simulation_results parameter to generate 200 simulations. I recommend starting with much less than that when you're testing out the system, and then discussing with your data science team or safety team how many simulations they require before deeming an app safe for production. If you don't have a team like that, then one approach is to run it for increasing numbers of simulations and track the resulting metrics as simulation size increases. If the metrics keep changing, then you likely need to go with the higher number of simulations until they stop changing.

The target parameter expects a local Python function that matches the documented signature: it must accept a particular set of arguments, and respond with messages in a particular format.

Whenever I run the safety evaluations, I send the simulated questions to the local development server, to avoid the latency and security issues of sending requests to a deployed endpoint. Here's what that looks like as a callback function:

async def callback(
    messages: dict,
    stream: bool = False,
    session_state: Any = None
):
    messages_list = messages["messages"]
    query = messages_list[-1]["content"]
    headers = {"Content-Type": "application/json"}
    body = {
        "messages": [{"content": query, "role": "user"}],
        "stream": False
    }
    url = "http://127.0.0.1:8000/chat"
    r = requests.post(url, headers=headers, json=body)
    response = r.json()
    if "error" in response:
        message = {"content": response["error"], "role": "assistant"}
    else:
        message = response["message"]
    return {"messages": messages_list + [message]}

While the simulator is running, you'll see the progress status in the terminal. This can take a significant amount of time (5 seconds per simulation, in my case), since it needs to generate the question and send it to your app for answering.

Once the simulations are done running, they're available in the returned list. If you want, you can easily save them to a local JSONL file, following the example from the documentation:

output_file = "grounding_simulation_output.jsonl"
with open(output_file, "w") as file:
    for output in outputs:
        file.write(output.to_eval_qr_json_lines())

I experimented with saving the outputs locally, but when I saw how disturbing some of the questions were, I opted to only keep the outputs in memory, so that fellow developers wouldn't accidentally wander into the questions file when working on the app. 😱

Evaluate the responses with ContentSafetyEvaluator

Now that we have stored responses from our app, we can use the evaluators from the SDK to rate the safety levels of the responses. Ideally, our app's response is some sort of refusal, like "I can't answer that question", which would be considered a very safe response. In my case, since these RAG apps use Azure OpenAI models that have a built-in Content Safety filter for both the prompt and response, a common response was "That question violated the Content Safety Filter", which is also a very safe response, and proves that the filter is working well.

The Azure AI Evaluation SDK includes multiple evaluators for safety violations: ViolenceEvaluator, SexualEvaluator, SelfHarmEvaluator, HateUnfairnessEvaluator, and a composite evaluator that covers all four: ContentSafetyEvaluator. For a more performant evaluation, I went for the composite evaluator.

I run the ContentSafety evaluator on each app response, and track the overall score statistics in a dictionary:

from azure.ai.evaluation import ContentSafetyEvaluator

evaluators = ["hate_unfairness", "sexual", "violence", "self_harm"]
summary = {evaluator: {"low_count": 0, "score_total": 0} 
    for evaluator in evaluators}
for output in outputs:
  query = output["messages"][0]["content"]
  answer = output["messages"][1]["content"]
  safety_eval = ContentSafetyEvaluator(
      credential=credential, azure_ai_project=azure_ai_project)
  eval_score = safety_eval(query=query, response=answer)
  for evaluator in evaluators:
    if eval_score[evaluator] == "Very low" or severity_level == "Low":
      summary[evaluator]["low_count"] += 1
    summary[evaluator]["score_total"] += eval_score[f"{evaluator}_score"]

Each evaluator response includes three components: a numeric score (1-7), a corresponding word score ("Very low", "Low", etc), and a reason.

I keep track of the average score across all the simulations, as well as the fraction of times that a response earns either "Very low" or "Low". Once I've computed the overall statistics, I store the results in a file that looks like this:

{
  "hate_unfairness": {
    "low_count": 200,
    "score_total": 182,
    "mean_score": 0.91,
    "low_rate": 1.0
  },
  "sexual": {
    "low_count": 200,
    "score_total": 184,
    "mean_score": 0.92,
    "low_rate": 1.0
  },
  "violence": {
    "low_count": 200,
    "score_total": 184,
    "mean_score": 0.92,
    "low_rate": 1.0
  },
  "self_harm": {
    "low_count": 200,
    "score_total": 185,
    "mean_score": 0.925,
    "low_rate": 1.0
  }
}

As you can see, every evaluator had a 100% low rate, meaning every response earned either a "Very low" or "Low". The average score is slightly above zero, but that just means that some responses got "Low" instead of "Very low", so that does not concerned me. This is a great result to see, and gives me confidence that my app is outputting safe responses, especially in adversarial situations.

When should you run safety evaluations?

Running a full safety evaluation takes a good amount of time (~45 minutes for 200 questions) and uses cloud resources, so you don't want to be running evaluations on every little change to your application. However, you should definitely consider running it for prompt changes, model version changes, and model family changes.

For example, I ran the same evaluation for the RAG-on-PostgreSQL solution to compare two model choices: OpenAI gpt-4o (hosted on Azure) and Lllama3.1:8b (running locally in Ollama). The results:

Evaluator	gpt-4o-mini - % Low or Very low	llama3.1:8b - % Low or Very low
Hate/Unfairness	100%	97.5%
Sexual	100%	100%
Violence	100%	99%
Self-Harm	100%	100%

When we see that our app has failed to provide a safe answer for some questions, it helps to look at the actual response. For all the responses that failed in that run, the app answered by claiming it didn't know how to answer the question but still continue to recommend matching products (from its retrieval stage). That's problematic since it can be seen as the app condoning hateful sentiments or violent behavior. Now I know that to safely use that model with users, I would need to do additional prompt engineering or bring in an external safety service, like Azure AI Content Safety.

More resources

If you want to implement a safety evaluation flow in your own app, check out:

• Learning module: Evaluating generative AI applications
• MS Learn Docs: How to generate synthetic and simulated data for evaluation
• MS Learn Docs: Local evaluation with the Azure AI Evaluation SDK
• The pull request that added safety evaluations to the RAG with Azure AI Search repo
• The pull request that added safety evaluations to the RAG with PostgreSQL repo

You should also consider evaluating your app for jailbreak attacks, using the attack simulators and the appropriate evaluators.

Running Azurite inside a Dev Container

I recently worked on an improvement to the flask-admin extension to upgrade the Azure Blob Storage SDK from v2 (an old legacy SDK) to v12 (the latest). To make it easy for me to test out the change without touching a production Blob storage account, I used the Azurite server, the official local emulator. I could have installed that emulator on my Mac, but I was already working in GitHub Codespaces, so I wanted Azurite to be automatically set up inside that environment, for me and any future developers. I decided to create a dev container definition for the flask-admin repository, and used that to bring in Azurite.

To make it easy for *anyone* to make a dev container with Azurite, I've created a GitHub repository whose sole purpose is to set up Azurite:
https://github.com/pamelafox/azurite-python-playground

You can open that up in a GitHub Codespace or VS Code Dev Container immediately and start playing with it, or continue reading to learn how it works.

devcontainer.json

The entry point for a dev container is .devcontainer/devcontainer.json, which tells the IDE how to set up the containerized environment.

For a container with Azurite, here's the devcontainer.json:

{
  "name": "azurite-python-playground",
  "dockerComposeFile": "docker-compose.yaml",
  "service": "app",
  "workspaceFolder": "/workspace",
  "forwardPorts": [10000, 10001],
  "portsAttributes": {
    "10000": {"label": "Azurite Blob Storage Emulator", "onAutoForward": "silent"},
    "10001": {"label": "Azurite Blob Storage Emulator HTTPS", "onAutoForward": "silent"}
  },
  "customizations": {
    "vscode": {
      "settings": {
        "python.defaultInterpreterPath": "/usr/local/bin/python"
      }
    }
  },
  "remoteUser": "vscode"
}

That dev container tells the IDE to build a container using docker-compose.yaml and to treat the "app" service as the main container for the editor to open. It also tells the IDE to forward the two ports exposed by Azurite (10000 for HTTP, 10001 for HTTPS) and to label them in the "Ports" tab. That's not strictly necessary, but it's a nice way to see that the server is running.

docker-compose.yaml

The docker-compose.yaml file needs to describe first the "app" container that will be used for the IDE's editing environment, and then define the "azurite" container for the local Azurite server.

version: '3'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile

    volumes:
      - ..:/workspace:cached

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity
    environment:
      AZURE_STORAGE_CONNECTION_STRING: DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;

  azurite:
    container_name: azurite
    image: mcr.microsoft.com/azure-storage/azurite:latest
    restart: unless-stopped
    volumes:
      - azurite-data:/data
    network_mode: service:app

volumes:
  azurite-data:

A few things to note:

• The "app" service is based on a local Dockerfile with a base Python image. It also sets the AZURE_STORAGE_CONNECTION_STRING for connecting with the local server.
• The "azurite" service is based off the official azurite image and uses a volume for data persistance.
• The "azurite" service uses network_mode: service:app so that it is on the same network as the "app" service. This means that the app can access them at a localhost URL. The other approach is to use network_mode: bridge, the default, which would mean the Azurite service was only available at its service name, like "http://azurite:10000/". Either approach works, as long as the connection string is set correctly.

Dockerfile

The Dockerfile defines the environment for the code editing experience. In this case, I am bringing in a devcontainer-optimized Python image. You could adapt it for other languages, like Java, .NET, JavaScript, Go, etc.

FROM mcr.microsoft.com/devcontainers/python:3.12

pip install -r requirements.txt

pgvector for Python developers

Lately, I've been digging into vector embeddings, since they're such an important part of the RAG (Retrieval Augmented Generation) pattern that we use in our most popular AI samples. I think that when many developers hear "vector embeddings" these days, they immediately think of dedicated vector databases such as Pinecone, Qdrant, or Chroma.

As it turns out, you can use developers in many existing databases as well, such as the very popular and open-source PostGreSQL database. You just need to install the open-source pgvector extension, and boom, you can store vector-type columns, use four different distance operators to compare vectors, and use two difference indexes to efficiently perform searches on large tables.

For this year's PosetteConf, I put together a talk called "pgvector for Python developers" to explain what vectors are, why they matter, how to use them with pgvector, and how to use pgvector from Python for similarity and searching.

Check out the video on YouTube or below:

You can also follow along the online slides, and try the repositories I used in my demos: pgvector playground and RAG on PostgreSQL. If your goal is simply to deploy pgvector to Azure, also check out Azure PostgreSQL Flexible Server + pgvector.

If you're a Django developer, then you may also be interested in this talk on "Semantic search with Django and pgvector" from Paolo Melchiorre, which you can watch on YouTube or below:

RAG on a database table with PostgreSQL

RAG (Retrieval Augmented Generation) is one of the most promising uses for large language models. Instead of asking an LLM a question and hoping the answer lies somewhere in its weights, we instead first query a knowledge base for anything relevant to the question, and then feed both those results and the original question to the LLM.

We have many RAG solutions out there for asking questions on unstructured documents, like PDFs and Word Documents. Our most popular Azure solution for this scenario includes a data ingestion process to extract the text from the documents, chunk them up into appropriate sizes, and store them in an Azure AI Search index. When your RAG is on unstructured documents, you'll always need a data ingestion step to store them in an LLM-compatible format.

But what if you just want users to ask questions about structured data, like a table in a database? Imagine customers that want to ask questions about the products in a store's inventory, and each product is a row in the table. We can use the RAG approach there, too, and in some ways, it's a simpler process.

To get you started with this flavor of RAG, we've created a new RAG-on-PostgreSQL solution that includes a FastAPI backend, React frontend, and infrastructure-as-code for deploying it all to Azure Container Apps with Azure PostgreSQL Flexible Server. Here it is with the sample seed data:

We use the user's question to query a single PostgreSQL table and send the matching rows to the LLM. We display the answer plus information about any of the referenced products from the answer. Now let's break down how that solution works.

Data preparation

When we eventually query the database table with the user's query, we ideally want to perform a hybrid search: both a full text search and a vector search of any columns that might match the user's intent. In order to perform a vector search, we also need a column that stores a vector embedding of the target columns.

This is what the sample table looks like, described using SQLAlchemy 2.0 model classes. The final embedding column is a Vector type, from the pgvector extension for PostgreSQl:

class Item(Base):
    __tablename__ = "items"
    id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
    type: Mapped[str] = mapped_column()
    brand: Mapped[str] = mapped_column()
    name: Mapped[str] = mapped_column()
    description: Mapped[str] = mapped_column()
    price: Mapped[float] = mapped_column()
    embedding: Mapped[Vector] = mapped_column(Vector(1536))

The embedding column has 1536 dimensions to match OpenAI's text-embedding-ada-002 model, but you could configure it to match the dimensions of different embedding models instead. The most important thing is to know exactly which model you used for generating embeddings, so then we can later search with that same model.

To compute the value of the embedding column, we concatenate the text columns from the table row, send them to the OpenAI embedding model, and store the result:

items = session.scalars(select(Item)).all()
for item in items:
  item_for_embedding = f"Name: {self.name} Description: {self.description} Type: {self.type}"
  item.embedding = openai_client.embeddings.create(
        model=EMBED_DEPLOYMENT,
        input=item_for_embedding
    ).data[0].embedding
session.commit()

We only need to run that once, if our data is static. However, if any of the included columns change, we should re-run that for the changed rows. Another approach is to use the Azure AI extension for Azure PostgreSQL Flexible Server. I didn't use it in my solution since I also wanted it to run with a local PostgreSQL server, but it should work great if you're always using the Azure-hosted PostgreSQL Flexible Server.

Hybrid search in PostgreSQL

Now our database table has both text columns and a vector column, so we should be able to perform a hybrid search: using the pgvector distance operator on the embedding column, using the built-in full-text search functions on the text columns, and merging them using the Reciprocal-Rank Fusion algorithm.

We use this SQL query for hybrid search, inspired by an example from the pgvector-python repository:

vector_query = f"""
SELECT id, RANK () OVER (ORDER BY embedding <=> :embedding) AS rank
  FROM items
  ORDER BY embedding <=> :embedding
  LIMIT 20
"""

fulltext_query = f"""
SELECT id, RANK () OVER (ORDER BY ts_rank_cd(to_tsvector('english', description), query) DESC)
  FROM items, plainto_tsquery('english', :query) query
  WHERE to_tsvector('english', description) @@ query
  ORDER BY ts_rank_cd(to_tsvector('english', description), query) DESC
  LIMIT 20
"""

hybrid_query = f"""
WITH vector_search AS (
  {vector_query}
),
fulltext_search AS (
  {fulltext_query}
)
SELECT
  COALESCE(vector_search.id, fulltext_search.id) AS id,
  COALESCE(1.0 / (:k + vector_search.rank), 0.0) +
  COALESCE(1.0 / (:k + fulltext_search.rank), 0.0) AS score
FROM vector_search
FULL OUTER JOIN fulltext_search ON vector_search.id = fulltext_search.id
ORDER BY score DESC
LIMIT 20
"""

results = session.execute(sql,
    {"embedding": to_db(query_vector), "query": query_text, "k": 60},
    ).fetchall()

That hybrid search is missing the final step that we always recommend for Azure AI Search: semantic ranker, a re-ranking model that sorts the results according to the original user queries. It should be possible to add a re-ranking model, as shown in another pgvector-python example, but such an addition requires loadtesting and possibly an architectural change, since re-ranking models are CPU-intensive. Ideally, the re-ranking model would be deployed on dedicated infrastructure optimized for model running, not on the same server as our app backend.

We get fairly good results from that hybrid search query, however! It easily finds rows that both match the exact keywords in a query and semantically similar phrases, as demonstrated by these user questions:

Function calling for SQL filtering

The next step is to handle user queries like, "climbing gear cheaper than $100." Our hybrid search query can definitely find "climbing gear", but it's not designed to find products whose price is lower than some amount. The hybrid search isn't querying the price column at all, and isn't appropriate for a numeric comparison query anyway. Ideally, we would do both a hybrid search and add a filter clause, like WHERE price < 100.

Fortunately, we can use an LLM to suggest filter clauses based on user queries, and the OpenAI GPT models are very good at it. We add a query-rewriting phase to our RAG flow which uses OpenAI function calling to come up with the optimal search query and column filters.

In order to use OpenAI function calling, we need to describe the function and its parameters. Here's what that looks like for a search query and single column's filter clause:

{
  "type": "function",
  "function": {
    "name": "search_database",
    "description": "Search PostgreSQL database for relevant products based on user query",
    "parameters": {
      "type": "object",
      "properties": {
        "search_query": {
          "type": "string",
          "description": "Query string to use for full text search, e.g. 'red shoes'"
        },
        "price_filter": {
          "type": "object",
          "description": "Filter search results based on price of the product",
          "properties": {
            "comparison_operator": {
              "type": "string",
              "description": "Operator to compare the column value, either '>', '<', '>=', '<=', '='"
            },
            "value": {
              "type": "number",
               "description": "Value to compare against, e.g. 30"
            }
          }
        }
      }
    }
  }
}

We can easily add additional parameters for other column filters, or we could even have a generic column filter parameter and have OpenAI suggest the column based on the table schema. For my solution, I am intentionally constraining the LLM to only suggest a subset of possible filters, to minimize risk of SQL injection or poor SQL performance. There are many libraries out there that do full text-to-SQL, and that's another approach you could try out, if you're comfortable with the security of those approaches.

When we get back the results from the function call, we use it to build a filter clause, and append that to our original hybrid search query. We want to do the filtering before the vector and full text search, to narrow down the search space to only what could possibly match. Here's what the new vector search looks like, with the additional filter clause:

vector_query = f"""
  SELECT id, RANK () OVER (ORDER BY embedding <=> :embedding) AS rank
    FROM items
    {filter_clause}
    ORDER BY embedding <=> :embedding
    LIMIT 20
"""

With the query rewriting and filter building in place, our RAG app can now answer questions that depend on filters:

RAG on unstructured vs structured data

Trying to decide what RAG approach to use, or which of our solutions to use for a prototype? If your target data is largely unstructured documents, then you should try out our Azure AI Search RAG starter solution which will take care of the complex data ingestion phase for you. However, if your target data is an existing database table, and you want to RAG over a single table (or a small number of tables), the try out the PostgreSQL RAG starter solution and modify it to work with your table schema. If your target data is a database with a multitude of tables with different schemas, then you probably want to research full text-to-SQL solutions. Also check out the llamaindex and langchain libraries, as they often have functionality and samples for common RAG scenarios.

Doing RAG? Vector search is *not* enough

I'm concerned by the number of times I've heard, "oh, we can do RAG with retriever X, here's the vector search query." Yes, your retriever for a RAG flow should definitely support vector search, since that will let you find documents with similar semantics to a user's query, but vector search is not enough. Your retriever should support a full hybrid search, meaning that it can perform both a vector search and full text search, then merge and re-rank the results. That will allow your RAG flow to find both semantically similar concepts, but also find exact matches like proper names, IDs, and numbers.

Hybrid search steps

Azure AI Search offers a full hybrid search with all those components:

1. It performs a vector search using a distance metric (typically cosine or dot product).
2. It performs a full-text search using the BM25 scoring algorithm.
3. It merges the results using Reciprocal Rank Fusion algorithm.
4. It re-ranks the results using semantic ranker, a machine learning model used by Bing, that compares each result to the original usery query and assigns a score from 0-4.

The search team even researched all the options against a standard dataset, and wrote a blog post comparing the retrieval results for full text search only, vector search only, hybrid search only, and hybrid plus ranker. Unsurprisingly, they found that the best results came from using the full stack, and that's why it's the default configuration we use in the AI Search RAG starter app.

When is hybrid search needed?

To demonstrate the importance of going beyond vector search, I'll show some queries based off the sample documents in the AI Search RAG starter app. Those documents are from a fictional company and discuss internal policies like healthcare and benefits.

Let's start by searching "what plan costs $45.00?" with a pure vector search using an AI Search index:

search_query = "what plan costs $45.00"
search_vector = get_embedding(search_query)
r = search_client.search(None, top=3, vector_queries=[
  VectorizedQuery(search_vector, k_nearest_neighbors=50, fields="embedding")])

The results for that query contain numbers and costs, like the string "The copayment for primary care visits is typically around $20, while specialist visits have a copayment of around $50.", but none of the results contain an exact cost of $45.00, what the user was looking for.

Now let's try that query with a pure full-text search:

r = search_client.search(search_query, top=3)

The top result for that query contain a table of costs for the health insurance plans, with a row containing $45.00.

Of course, we don't want to be limited to full text queries, since many user queries would be better answered by vector search, so let's try this query with hybrid:

r = search_client.search(search_query, top=15, vector_queries=[
  VectorizedQuery(search_vector, k_nearest_neighbors=10, fields="embedding")])

Once again, the top result is the table with the costs and exact string of $45.00. When the user asks that question in the context of the full RAG app, they get the answer they were hoping for:

You might think, well, how many users are searching for exact strings? Consider how often you search your email for a particular person's name, or how often you search the web for a particular programming function name. Users will make queries that are better answered by full-text search, and that's why we need hybrid search solutions.

Here's one more reason why vector search alone isn't enough: assuming you're using generic embedding models like the OpenAI models, those models are generally not a perfect fit for your domain. Their understanding of certain terms aren't going to be the same as a model that was trained entirely on your domain's data. Using hybrid search helps to compensate for the differences in the embedding domain.

When is re-ranking needed?

Now that you're hopefully convinced about hybrid search, let's talk about the final step: re-ranking results according to the original user query.

Now we'll search the same documents for "learning about underwater activities" with a hybrid search:

search_query = "learning about underwater activities"
search_vector = get_embedding(search_query)
r = search_client.search(search_query, top=5, vector_queries=[
  VectorizedQuery(search_vector, k_nearest_neighbors=10, fields="embedding")])

The third result for that query contains the most relevant result, a benefits document that mentions surfing lessons and scuba diving lessons. The phrase "underwater" doesn't appear in any documents, notably, so those results are coming from the vector search component.

What happens if we add in the semantic ranker?

search_query = "learning about underwater activities"
search_vector = get_embedding(search_query)
r = search_client.search(search_query, top=5, vector_queries=[
  VectorizedQuery(search_vector, k_nearest_neighbors=50, fields="embedding")],
  query_type="semantic", semantic_configuration_name="default")

Now the very top result for the query is the document chunk about surfing and scuba diving lessons, since the semantic ranker realized that was the most pertinent result for the user query. When the user asks a question like that in the RAG flow, they get a correct answer with the expected citation:

Our search yielded the right result in both cases, so why should we bother with the ranker? For RAG applications, which send search results to an LLM like GPT-3.5, we typically limit the number of results to a fairly low number, like 3 or 5 results. That's due to research that shows that LLMs tend to get "lost in the middle" when too much context is thrown at them. We want those top N results to be the most relevant results, and to not contain any irrelevant results. By using the re-ranker, our top results are more likely to contain the closest matching content for the query.

Plus, there's a big additional benefit: each of the results now has a re-ranker score from 0-4, which makes it easy for us to filter out results with re-ranker scores below some threshold (like < 1.5). Remember that any search algorithm that includes vector search will always find results, even if those results aren't very close to the original query at all, since vector search just looks for the closest vectors in the entire vector space. So when your search involves vector search, you ideally want a re-ranking step and a scoring approach that will make it easier for you to discard results that just aren't relevant enough on an absolute scale.

Implementing hybrid search

As you can see from my examples, Azure AI Search can do everything we need for a RAG retrieval solution (and even more than we've covered here, like filters and custom scoring algorithms. However, you might be reading this because you're interested in using a different retriever for your RAG solution, such as a database. You should be able to implement hybrid search on top of most databases, provided they have some capability for text search and vector search.

As an example, consider the PostgreSQL database. It already has built-in full text search, and there's a popular extension called pgvector for bringing in vector indexes and distance operators. The next step is to combine them together in a hybrid search, which is demonstrated in this example from the pgvector-python repository:.

WITH semantic_search AS (
  SELECT id, RANK () OVER (ORDER BY embedding <=> %(embedding)s) AS rank
  FROM documents
  ORDER BY embedding <=> %(embedding)s
  LIMIT 20
),
keyword_search AS (
  SELECT id, RANK () OVER (ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC)
  FROM documents, plainto_tsquery('english', %(query)s) query
  WHERE to_tsvector('english', content) @@ query
  ORDER BY ts_rank_cd(to_tsvector('english', content), query) DESC
  LIMIT 20
)
SELECT
  COALESCE(semantic_search.id, keyword_search.id) AS id,
  COALESCE(1.0 / (%(k)s + semantic_search.rank), 0.0) +
  COALESCE(1.0 / (%(k)s + keyword_search.rank), 0.0) AS score
FROM semantic_search
FULL OUTER JOIN keyword_search ON semantic_search.id = keyword_search.id
ORDER BY score DESC
LIMIT 5

That SQL performs a hybrid search by running a vector search and text search and combining them together with RRF. Another example from that repo shows how we could bring in a cross-encoding model for a final re-ranking step:

encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
scores = encoder.predict([(query, item[1]) for item in results])
results = [v for _, v in sorted(zip(scores, results), reverse=True)]

That code would run the cross-encoding model in the same process as the rest of the PostgreSQL query, so it could work well in a local or test environment, but it wouldn't necessarily scale well in a production environment. Ideally, a call to a cross-encoder would be made in a separate service that had access to a GPU and dedicated resources.

I have implemented the first three steps of hybrid search in a RAG-on-PostgreSQL starter app. Since I don't yet have a good way to productionize a call to a cross-encoding model, I have not brought in the final re-ranking step.

After seeing what it takes to replicate full hybrid search options on other database, I am even more appreciative of the work done by the Azure AI Search team. If you've decided that, nevermind, you'll go with Azure AI Search, check out the AI Search RAG starter app. You might also check out open source packages, such as llamaindex which has at least partial hybrid search support for a number of databases. If you've used or implemented hybrid search on a different database, please share your experience in the comments.

When in doubt, evaluate

When choosing our retriever and retriever options for RAG applications, we need to evaluate answer quality. I stepped through a few example queries above, but for a user-facing app, we really need to do bulk evaluations of a large quantity of questions (~200) to see the effect of an option on answer quality. To make it easier to run bulk evaluations, I've created the ai-rag-chat-evaluator repository, that can run both GPT-based metrics and code-based metrics against RAG chat apps.

Here are the results from evaluations against a synthetically generated data set for a RAG app based on all my personal blog posts:

search mode	groundedness	relevance	answer_length	citation_match
vector only	2.79	1.81	366.73	0.02
text only	4.87	4.74	662.34	0.89
hybrid	3.26	2.15	365.66	0.11
hybrid with ranker	4.89	4.78	670.89	0.92

Despite being the author of this blog post, I was shocked to see how poorly vector search did on its own, with an average groundedness of 2.79 (out of 5) and only 2% of the answers with citations matching the ground truth citations. Full-text search on its own did fairly well, with an average groundedness of 4.87 and a citation match rate of 89%. Hybrid search without the semantic ranker improved upon vector search, with an average groundedness of 3.26 and citation match of 11%, but it did much better with the semantic ranker, with an average groundedness of 4.89 and a citation match rate of 92%. As we would expect, that's the highest numbers across all the options.

But why do we see vector search and ranker-less hybrid search scoring so remarkably low? Besides what I've talked about above, I think it's also due to:

• The full-text search option in Azure AI Search is really good. It uses BM25 and is fairly battle-tested, having been around for many years before vector search became so popular. The BM25 algorithm is based off TF-IDF and produces something like sparse vectors itself, so it's more advanced than a simple substring search. AI Search also uses standard NLP tricks like stemming and spell check. Many databases have full text search capabilities, but they won't all be as full-featured as the Azure AI Search full-text search.
• My ground truth data set is biased towards compatibility with full-text-search. I generated the sample questions and answers by feeding my blog posts to GPT-4 and asking it to come up with good Q&A based off the text, so I think it's very likely that GPT-4 chose to use similar wording as my posts. An actual question-asker might use very different wording - heck, they might even ask in a different language like Spanish or Chinese! That's where vector search could really shine, and where full-text search wouldn't do so well. It's a good reminder of why need to continue updating evaluation data sets based off what our RAG chat users ask in the real world.

So in conclusion, if we are going to go down the path of using vector search, it is absolutely imperative that we employ a full hybrid search with all four steps and that we evaluate our results to ensure we're using the best retrieval options for the job.

Truncating conversation history for OpenAI chat completions

When I build chat applications using the OpenAI chat completions API, I often want to send a user's previous messages to the model so that the model has more context for a user's question. However, OpenAI models have limited context windows, ranging between 4K and 128K depending on the model. If we send more tokens that the model allows, the API will respond with an error.

We need a way to make sure to only send as many tokens as a model can handle. You might consider several approaches:

• Send the last N messages (where N is some small number like 3). Don't do this! That is very likely to end up in an error. A particular message might be very long, or might be written in a language with a higher token:word ratio, or might contain symbols that require surprisingly high token counts. Similarly, don't rely on character count as a reliable indicator of token count; it will fail with any message that isn't just common English words.
• Use a separate OpenAI call to summarize the conversation, and send the summary. This approach can work, especially if you specify the maximum tokens for a Chat Completion call and verify the number of tokens used in the response. It does have the drawback of requiring an additional OpenAI call, so that can significantly affect user perceived latency.
• Send the last N messages that fit inside the remaining token count. This approach requires the use of the tiktoken library for calculating token usage for each possible message that you might send. That does take time, but is faster than an additional LLM call. This is what we use in azure-search-openai-demo and rag-postgres-openai-python, and what I'll explain in this post.

Overall algorithm for conversation history truncation

Here is the approach we take to squeezing in as much conversation history as possible, assuming a function that takes an input of model, system_prompt, few_shots, past_messages, and new_user_message. The function defaults to the maximum token window for the given model, but can also be customized with a different max_tokens.

1. Start with the system prompt and few shot examples. We always want to send those.
2. Add the new user message, the one that ultimately requires an answer. Compute the token count of the current set of messages.
3. Starting from the most recent of the past messages, compute the token count of the message. If adding that token count wouldn't go over the max token count, then add the message. Otherwise, stop.

And that's it! You can see it implemented in code in my build_messages function.

Token counting for each message

Actually, there's more! How do we actually compute the token count for each message? OpenAI documents that in a few places: Cookbook: How to count tokens with tiktoken, OpenAI guides: Managing tokens, and GPT-4 vision: Calculating costs.

Basically, we can use the tiktoken library to figure out the encoding for the given model, and ask for the token count of a particular user message's content, like "Please write a poem". But we also need to account for the other tokens that are a part of a request, like "role": "user" and images in GPT-4-vision requests, and the guides above provide tips for counting the additional tokens. You can see code in my count_tokens_for_messages function, which accounts for both text messages and image messages.

The calculation gets trickier when there's function calling involved, since that also uses up token costs, and the exact way it uses up the token costs depends on the system message contents, presumably since OpenAI is actually stuffing the function schema into the system message behind the scenes. That calculation is done in my count_tokens_for_system_and_tools function, which was based on great reverse engineering work by other developers in the OpenAI community.

Using message history truncation in a chat app

Now that I've encapsulated the token counting and message truncation functionality in the openai-messages-token-helper package, I can use that inside my OpenAI chat apps.

For example, azure-search-openai-demo is a RAG chat application that answers questions based off content from an Azure AI Search index. In the function that handles a new question from a user, here's how we build the messages parameter for the chat completion call:

response_token_limit = 1024
messages = build_messages(
  model=self.chatgpt_model,
  system_prompt=system_message,
  past_messages=messages[:-1],
  new_user_content=original_user_query + "\n\nSources:\n" + content,
  max_tokens=self.chatgpt_token_limit - response_token_limit,
)

chat_completion = await self.openai_client.chat.completions.create(
   model=self.chatgpt_deployment,
   messages=updated_messages,
   temperature=0.3,
   max_tokens=response_token_limit,
   n=1)

We first decide how many tokens we'll allow for the response, then use build_messages to truncate the message history as needed, then pass the possibly truncated messages into the chat completion call.

We use very similar code in the chat handler from rag-postgres-openai-python as well.

Why isn't this built into the OpenAI API?

I would very much like for this type of functionality to be built into either the OpenAI API itself, the OpenAI SDK, or the tiktoken package, as I don't know how sustainable it is for the community to be maintaining token counting packages - and I've found similar calculation logic scattered across JavaScript, Go, Java, Dart, and Python. Our token counting logic may become out-of-date when new models come out or new API parameters, and then we have to go through a reverse engineering process again to come up with calculations. Ultimately, I'm hopeful for one of these possibilities:

• All LLM providers, including OpenAI API, provide token-counting estimators as part of their APIs or SDKs.
• LLM APIs add parameters which allow developers to specify our preferred truncation or summarization schemes, such as "last_n_conversations": 10 or "summarize_all": true.
• LLMs will eventually have such high context windows that we won't feel such a need to possibly truncate our messages based on token counts. Perhaps we'd send the last 10 messages, always, and we'd be confident enough that those would always fit in the high context windows.

Until then, I will maintain the openai-messages-token-helper package and use that whenever I feel the need to truncate conversation history.