AGENTIC AI

Creating Custom (Code-Based) Agent Evaluators in Microsoft Foundry with Python

Build, register, and debug an evaluator using the Foundry SDK. Plus some common runtime errors discussed.

Introduction

Microsoft Foundry has several built-in evaluators for measuring things such as intent resolution, task adherence, groundedness, and tool-call accuracy. These are a very useful starting point to evaluate an agent because we can add them to an evaluation run without writing the scoring logic ourselves. The setup is also super simple, with the option to create them on Foundry Playground or via SDK.

But what happens when the quality check is specific to our application or agent? Or the agent caters to judgment tasks, like how well did it respond to a customer complaint?

For instance, I have a fact-checker agent that searches the web and returns the sources used to verify a claim. In addition to checking whether the agent understood the task (i.e. intent_resolution), I want to ask a more application-specific question:

Did the agent return enough sources for the input claim and were these sources recent enough?

As of date, there is no built-in evaluator called source_recency, so I created a custom code-based evaluator for it.

The scoring logic (between 0 and 1) is straightforward:

• 0 if returned sources were not dated (i.e., no published or updated date available)
• 0.5 if at least one of the returned sources was dated and fulfilled recency requirements
• 1 if all the returned sources were dated and fulfilled recency requirements

In the rest of the article, we will learn how to (a) write this custom evaluator, (b) register it in Foundry so anyone on your team can call it as easily as the built-in evaluators, and (c) run an eval dataset against it.

Let’s begin!

What are custom code-based evaluators?

A code-based evaluator is a deterministic Python function that Foundry runs for every item in the evaluation dataset. It is useful when the scoring rule can be expressed without asking another LLM to judge the response.

Some examples include:

• checking whether a response is valid JSON;
• verifying that mandatory fields are present;
• measuring response length;
• detecting whether citations were returned;
• confirming that enough sources meet a date threshold; or
• validating an application-specific output format.

Apart from code-based evaluators, we can also create prompt-based evaluators.

Every code-based evaluator must expose a grade function with this shape:

def grade(sample: dict, item: dict) -> float:
    ...

It takes only two inputs — sample which is the output from the agent and item which is each row in the eval dataset.

The function returns a single float between 0.0 and 1.0, where a higher value is better.

Note: Code-based evaluators are best suited to deterministic rules. If the requirement is subjective — for example, “Does this response sound empathetic?” — a prompt-based evaluator is usually a better fit.

Easiest mental mapping is, if you need to use LLM-as-a-judge i.e. an LLM will score an agent output for qualities like politeness, rudeness, sentiment, etc. then use prompt-based evaluator.

Prerequisites

For this example, I am using:

• Python 3.10 or above;
• azure-ai-projects==2.8.0;
• azure-identity;
• python-dotenv; and
• an existing Microsoft Foundry project containing an agent and a model deployment.

Install the packages:

pip install azure-ai-projects==2.8.0
pip install azure-identity
pip install python-dotenv

Create a .env file containing the project endpoint and the deployment used by the evaluation service:

AZURE_AI_PROJECT_ENDPOINT="https://<account>.services.ai.azure.com/api/projects/<project>"
JUDGE_MODEL_DEPLOYMENT="gpt-4o-mini"

Note: The evaluator catalog and some evaluation APIs are still preview features. The exact API surface can change between SDK versions, so pinning the version is helpful when you want the same code to behave consistently across environments.

Our example: a source-recency evaluator

Let’s look at the base agent first

Assume that the fact-checker agent returns JSON containing its final verdict and a list of sources:

{
  "verdict": "True",
  "reasoning": "The claim is supported by recent sources.",
  "sources": [
    {
      "url": "https://example.com/source-a",
      "published_or_updated_date": "2026-10-01"
    },
    {
      "url": "https://example.com/source-b",
      "published_or_updated_date": "2026-09-15"
    }
  ]
}

If you are wondering — do i need to have an agent which returns a JSON object instead of free text?

The answer is — not really! However, it makes life a lot easier if you have a JSON object initialized at the time of agent creation, since we can easily extract entities of interest from it — in this case published_or_updated_date. If it was free text i.e. a string response from the agent, I would need to write complex regex patterns to extract these entities.

Let’s build the eval dataset

Each dataset row (henceforth called an item) carries the policy against which the sources should be checked:

{
  "item": {
    "query": "Claim: Trump administration has launched sustainability campaigns.",
    "as_of_date": "2026-10-09", # today's date
    "max_source_age_days": 180,
    "requires_dated_sources": true,
    "min_sources": 2 # minimum sources needed to verify this claim
  }
}

This is useful because recency is not necessarily a single global rule. One test case might allow sources from the previous 180 days, while another could require information from the last 30 days.

Here’s what the final eval dataset looks like:

{"id": "fc_01", "min_sources": 2, "requires_dated_sources": true, "max_source_age_days": 60, "as_of_date": "2026-10-07", "query": "Claim: Trump administration has launched sustainability campaigns.", "expected_tool_calls": "1 web search (1 claim)", "tool_definitions": [{"name": "web_search", "description": "Searches the web for information to verify a factual claim.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "The search query used to verify a single claim."}}, "required": ["query"]}}]}
{"id": "fc_02", "min_sources": 1, "requires_dated_sources": true, "max_source_age_days": 180, "as_of_date": "2026-10-07", "query": "Claim: Remote work has reduced employee productivity in large technology companies..", "expected_tool_calls": "1 web search (1 claim)", "tool_definitions": [{"name": "web_search", "description": "Searches the web for information to verify a factual claim.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "The search query used to verify a single claim."}}, "required": ["query"]}}]}
{"id": "fc_03", "min_sources": 3, "requires_dated_sources": true, "max_source_age_days": 30, "as_of_date": "2026-10-07", "query": "Claim: Electric vehicles are becoming more environmentally sustainable than petrol cars over their full lifecycle.", "expected_tool_calls": "1 web search (1 claim)", "tool_definitions": [{"name": "web_search", "description": "Searches the web for information to verify a factual claim.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "The search query used to verify a single claim."}}, "required": ["query"]}}]}
{"id": "fc_04", "min_sources": 0, "requires_dated_sources": false, "max_source_age_days": 180, "as_of_date": "2026-10-07", "query": "Claim: Social media platforms have become more effective at limiting the spread of misinformation since 2020.", "expected_tool_calls": "1 web search (1 claim)", "tool_definitions": [{"name": "web_search", "description": "Searches the web for information to verify a factual claim.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "The search query used to verify a single claim."}}, "required": ["query"]}}]}

Writing the grade() function

Here’s the shortened version with helper functions skipped.

def grade(sample: dict, item: dict) -> float:
    """
    Score whether the agent returned enough recent, dated sources.
    """
    try:
        generated_sample = item.get("sample", {})
        response_text = generated_sample.get("output_text", "")

        payload = json.loads(response_text)
        sources = payload.get("sources", [])

        as_of_date = parse_date(item.get("as_of_date"))
        max_age = int(item.get("max_source_age_days", 14))
        min_sources = int(item.get("min_sources", 1))
        requires_dates = bool(item.get("requires_dated_sources", True))

        if not as_of_date or not isinstance(sources, list) or not sources:
            return 0.0

        dated_sources = 0
        recent_sources = 0

        for source in sources:
            source_date = parse_date(
                source.get("published_or_updated_date")
                or source.get("published_date")
                or source.get("updated_date")
            )

            if not source_date:
                continue

            dated_sources += 1
            age_days = (as_of_date - source_date).days

            if 0 <= age_days <= max_age:
                recent_sources += 1

        if recent_sources >= min_sources:
            return 1.0

        if requires_dates and dated_sources == 0:
            return 0.0

        return 0.5 if dated_sources else 0.0

    except Exception:
        return 0.0

A couple of pointers:

• As explained before, the grade()function expects item dictionary, which in our case looks like this (I will explain shortly how to create this in evals.runs.create()).

item["sample"]["output_text"]
item["as_of_date"]
item["max_source_age_days"]
item["min_sources"]
item["requires_dated_sources"]

• The agent response (henceforth called sample) should include JSON like:

{
        "sources": [
          {
            "url": "...",
            "published_or_updated_date": "2026-10-01"
          }
        ]
}

• If you notice line 1 in grade(), we have generated_sample = item.get(“sample”, {}), instead of just generated_sample = sample. This confused me initially — after all, why aren’t we using the sample parameter passed to the grade() directly, instead of calling from inside item?
  Simple answer — This is just the way Msft wants us to do it for the time being; it may change in future API updates. And yes, the grade function must still accept both sample and item, even though the current Foundry target-evaluation path places the generated response inside item.

Source: https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/custom-evaluators#code-based-evaluators

• Our fact-checker agent returned a JSON object, but output_text contains its serialized JSON string. We therefore call json.loads() in line 3 before reading sources.

Where did item["sample"]["output_text"] come from?

This was the part I initially found confusing.

I did not add a field called sample to every JSONL row. Foundry adds it at runtime for a model or agent target evaluation. The generated response is currently available to code-based evaluators using:

item.get("sample", {}).get("output_text", "")

The key name sample is not an arbitrary variable chosen by us. It is a Foundry-defined runtime key.

In other words, this will not work automatically:

item["blahblah"]["output_text"]

Note: We could certainly create a dataset field called blahblah, but Foundry would not automatically place the generated agent response inside it. The automatic response is exposed under the reserved sample key. Likewise, output_text is also reserved.

Registering the evaluator

At the time of registering an evaluator using evaluators.create_version(), one of the things you need to define is the data_schema. This schema declares fields specifically required by source_recency for the scoring logic, such as today’s date (as_of_date), age limit (max_source_age_days), and mandatory sources (min_sources) etc., along with their types.

"data_schema" = {
                "type": "object",
                "properties": {
                    "item": {              <------ place all JSONL columns under item
                        "type": "object",
                        "properties": {
                            "sample": {
                                "type": "object",
                                "properties": {
                                    "output_text": {"type": "string"},
                                },
                            },
                            "as_of_date": {"type": "string"},
                            "max_source_age_days": {"type": "integer"},
                            "requires_dated_sources": {"type": "boolean"},
                            "min_sources": {"type": "integer"},
                        },
                    },
                },
          }

Note: Declaring these fields in data_schema only documents their shape—it does not name or create them. In short, you cannot rename output_text to blahblah and expect Foundry to populate it.

If you will notice, source_recency does not declare query in data_schema because its scoring logic does not use it. However, if the evaluator actually depended on it, you should explicitly add "query": {"type": "string"}.

"data_schema" = {
                "type": "object",
                "properties": {
                    "item": {
                        "type": "object",
                        "properties": {
                            "sample": {
                                "type": "object",
                                "properties": {
                                    "output_text": {"type": "string"},
                                },
                            },
                            "as_of_date": {"type": "string"},
                            "max_source_age_days": {"type": "integer"},
                            "requires_dated_sources": {"type": "boolean"},
                            "min_sources": {"type": "integer"},
                            "query": {"type": "string"} < -------- newly added
                        },
                    },
                },
          }

Note: The sample argument in grade(sample, item) and the nested item["sample"] can look like the same thing, but they are not interchangeable variables. For the current Foundry target-evaluation contract, Microsoft documents that generated output should be read from the nested value under item.

Creating the evals with the registered evaluator

To make sure the schema for sample is shared with the Foundry evaluators (to allow extracting stuff inside it e.g. sample.output_*), it is important to set include_sample_schema=True when defining the data_source_config

data_source_config = {
    "type": "custom",
    "item_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string"},
            "as_of_date": {"type": "string"},
            "max_source_age_days": {"type": "integer"},
            "requires_dated_sources": {"type": "boolean"},
            "min_sources": {"type": "integer"},
        },
        "required": [
            "query",
            "as_of_date",
            "max_source_age_days",
            "requires_dated_sources",
            "min_sources",
        ],
    },
    "include_sample_schema": True, < --- # exposes {{sample.output_*}} and Foundry then supplies them at runtime to code evaluators under item["sample"]
}

This tells the evaluation configuration to expose the generated sample schema. During a target-based evaluation, Foundry constructs the runtime input passed to the evaluator and supplies the generated output under item["sample"].

P.S> It does not rewrite the original JSONL file or permanently append a sample object to the dataset row.

Registering the evaluator

The evaluator code must be registered in the Foundry evaluator catalog before it can be referenced by an evaluation.

Get Dr. Varshita Sher’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The important part of the registration is in the snippet above, using project_client.evaluators.create_version().

evaluator = project_client.evaluators.create_version(
            name = "source_recency",
            headers = ...,
            evaluator_version = {
                "name": "source_recency",
                "categories": [EvaluatorCategory.QUALITY],
                "display_name": "Source Recency",
                "description": "Checks whether returned sources are recent enough for the test case policy.",
                "definition": { 
                     ...,
                    "data_schema": {
                        "type": "object",
                        "properties": {
                            "item": {
                                "type": "object",
                                "properties": {
                                    "sample": {
                                        "type": "object",
                                        "properties": {
                                            "output_text": {"type": "string"},
                                        },
                                    },
                                    "as_of_date": {"type": "string"},
                                    "max_source_age_days": {"type": "integer"},
                                    "requires_dated_sources": {"type": "boolean"},
                                    "min_sources": {"type": "integer"},
                                },
                            },
                        },
                    },
                  }
            }
)

Run the registration script once:

python register_evaluator.py

You should see the newly created version printed in the terminal.

Note: “Run once” means once per evaluator version — not once for the evaluator’s entire lifetime. Do not register it before every evaluation run. Run the script again only when the evaluator code or definition changes, because create_version() creates another catalog version.

Pin the evaluator version

After creating a new version, pass that version in the testing criterion:

SOURCE_RECENCY_EVALUATOR_VERSION = os.getenv(
    "SOURCE_RECENCY_EVALUATOR_VERSION"
)

source_recency_criterion = {
    "type": "azure_ai_evaluator",
    "name": "source_recency",
    "evaluator_name": "source_recency",
    "evaluator_version": SOURCE_RECENCY_EVALUATOR_VERSION,
    "initialization_parameters": {
        "deployment_name": JUDGE_MODEL,
        "pass_threshold": 0.2,
    },
}

You may ask — Why pin it when the SDK says it can use the latest version?

During development, I had several versions of the same evaluator registered with different schemas. One version expected the flat fields response, as_of_date, and min_sources, while a later version expected only item. An evaluation resolved the older contract and failed with MissingRequiredDataMapping.

P.S. You need one criterion dictionary per evaluator. So if you are using source_recency along with the inbuilt task_adherence evaluator, you will have:

source_recency_criterion = {
    "type": "azure_ai_evaluator",
    "name": "source_recency",
    "evaluator_name": "source_recency",
    "evaluator_version": SOURCE_RECENCY_EVALUATOR_VERSION,
    "initialization_parameters": {
        "deployment_name": JUDGE_MODEL,
        "pass_threshold": 0.2,
    },
}

task_adherence_criterion = {
    "type": "azure_ai_evaluator",
    "name": "task_adherence",
    "evaluator_name": "builtin.task_adherence",
    "evaluator_version": SOURCE_RECENCY_EVALUATOR_VERSION,
    "initialization_parameters": {
        "model": JUDGE_MODEL,
    },
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{sample.output_items}}"
    }
}

criteria = [source_recency_criterion, task_adherence_criterion]

This will be used for creating evals later on:

item_properties = {
            "query": {"type": "string"},
            "as_of_date": {"type": "string"},
            "max_source_age_days": {"type": "integer"},
            "requires_dated_sources": {"type": "boolean"},
            "min_sources": {"type": "integer"},
        }

required = [
            "query",
            "as_of_date",
            "max_source_age_days",
            "requires_dated_sources",
            "min_sources",
        ]

eval_object = openai_client.evals.create(
        name=f"{agent_key} dataset evaluation",
        data_source_config={
            "type": "custom",
            "item_schema": {
                "type": "object",
                "properties": item_properties,
                "required": required,
            },
            "include_sample_schema": True,   # exposes {{sample.output_*}} and Foundry then supplies them at runtime to code evaluators under item["sample"]
        },
        testing_criteria=criteria,
    )

A couple of things:

• Criterion will look different depending on the evaluator you are using. For example, observe how we don’t need to pass on data_mapping parameter for custom evaluators like source_recency. This is because, unlike the built-in evaluators, a Foundry code-based evaluator does not need an explicit data_mapping. Its registered data_schema requires item, and its dataset fields are read from that dictionary.

Built-in evaluators need data mappings; this code evaluator does not

Built-in evaluators describe the inputs they expect. For example, an intent-resolution evaluator needs a query and a response:

{
    "type": "azure_ai_evaluator",
    "name": "intent_resolution",
    "evaluator_name": "builtin.intent_resolution",
    "initialization_parameters": {"model": JUDGE_MODEL},
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{sample.output_text}}",
    },
}

So we use data_mapping dictionary where key is what an evaluator might need for evaluation, and the value is the location where to find it. For instance, the query for intent-resolution comes from the eval dataset, i.e. item and can be found at item.query. Likewise, intent-resolution also needs the response to check whether intent was resolved in the final agent output. To do this, it can use sample.output_text.

Other evaluators like tool_call_accuracy might instead need the structured response items

"data_mapping": {
        "query": "{{item.query}}",
        "response": "{{sample.output_items}}",
    },

So why don’t we need a data_mapping for a custom evaluator?

Unlike the built-in evaluators, a Foundry code-based evaluator does not need an explicit data_mapping.

The code-based source_recency evaluator is different. Its registered schema requires the complete item object, and Foundry supplies the generated target sample inside that object. Therefore, its testing criterion does not need an explicit data_mapping.

P.S. This is not a general rule that all custom evaluator types ignore mappings. It is the contract used by Foundry code-based evaluators. Prompt-based and endpoint-based evaluators have their own input and mapping requirements.

Since I was not aware of this initially, I made the mistake of adding explicit mappings to the code evaluator. My scoring logic, i.e. grade() needed sample and item and this is how I created the data mapping:

"data_mapping": {
    "sample": "{{sample.output_text}}",
    "item": "{{item.recency_input}}",
}

Funnily enough, the evaluation run completed without errors, but the fact-checker agent responded as if the user had literally asked it to score intent-resolution for this input query: {{item.query}}

Figure 1: Incorrect data_mapping: Failed response showing the literal {{item.query}} value

The built-in evaluators also produced strange results, which initially made it look as though all the evaluators were broken. But the evaluators were downstream of the same generated response. The real issue was that the agent target had received an unresolved input template. Also, looking at intent_resolution.reason column confirmed my doubts that the actual user input was never passed.

Figure 2: A (succesful-ish) eval run but with all evaluators failing (due to incorrect data_mapping)

To debug, I tried removing source_recency and that made the built-in evaluations work again. That was the useful clue: the problem was not the dataset query or the built-in evaluators; it was the custom criterion’s contract, i.e. data_mapping.

The fix was to follow the code-based evaluator pattern:

• declare item in the registered data_schema;
• enable the sample schema for the target evaluation;
• read the generated response from item["sample"]["output_text"]; and
• omit data_mapping from the code-evaluator criterion.

Figure 3: Correct data_mapping producing the user input = “Centrum is receiving positive feedback”

Pro Tip: When adding a new evaluator breaks an otherwise working target run, temporarily remove only that evaluator. If the agent starts receiving the correct query again, compare the custom evaluator’s registered schema with its testing criterion before changing the agent prompt or dataset.

criteria = [
    # source_recency_criterion,  <--- temporarily remove custom evaluator
    task_adherence_criterion
]

eval_object = openai_client.evals.create(
        name=f"{agent_key} dataset evaluation",
        data_source_config=data_source_config,
        testing_criteria=criteria
    )

Why is there no source_recency.reason column?

If you look at Figure 2 above, you’ll notice all built-in evaluators often show both a score and a reason in the Foundry report. For instance,

intent_resolution
intent_resolution.reason

A code-based evaluator has a smaller output contract: grade() returns a single float. Consequently, our report contains source_recency, but it does not contain source_recency.reason.

This is expected behaviour — not a missing schema field.

If an explanation is mandatory, there are two alternatives:

1. Use a prompt-based evaluator, which can return result and reason.
2. Use an endpoint-based evaluator, which can return score, reason, statusand optional properties.

I kept source-recency code-based because its scoring rule is deterministic and does not need another LLM call. For this scenario, predictable scoring was more important than generating a natural-language reason for every row.

Complete evaluation workflow

At a high level, the final flow looks like this:

1. Load each JSONL row into item.
2. Foundry renders {{item.query}} and sends it to the agent target.
3. The agent generates its fact-checking response.
4. Because the sample schema is enabled, Foundry makes that generated response available to the code evaluator under item["sample"].
5. grade(sample, item) parses item["sample"]["output_text"].
6. The evaluator checks the source dates against the policy fields from the same item.
7. Foundry records the returned float as the source_recency score.

The key evaluation configuration is:

eval_object = openai_client.evals.create(
    name="fact-checker dataset evaluation",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": item_properties,
            "required": required,
        },
        "include_sample_schema": True,
    },
    testing_criteria=testing_criteria,
)

The target run then connects each dataset query to the agent:

run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="fact-checker dataset run",
    data_source={
        "type": "azure_ai_target_completions",
        "source": {
            "type": "file_content",
            "content": rows,
        },
        "input_messages": {
            "type": "template",
            "template": [
                {
                    "type": "message",
                    "role": "user",
                    "content": {
                        "type": "input_text",
                        "text": "{{item.query}}",
                    },
                }
            ],
        },
        "target": {
            "type": "azure_ai_agent",
            "name": "fact-checker-agent",
        },
    },
)

A couple of pointers:

• Setting “type”: “azure_ai_target_completions” is to let Foundry know we don't have ground truth responses, so each query in the eval dataset must be run against the fact-checker agent, and generated responses will be used for the evaluations.
• Under input_messages the role user is used for passing in the query (if you have worked with the openai or langchain library, this way of sending payload should look familiar).
• When you are creating the content each row must contain everything that will be needed by the evaluator for the scoring. For instance, at a basic level:

item1 = {
          "query": "Ice caps are melting fast"
          "as_of_date": "26 Oct 2026"
          "max_source_age_days": 180
          "requires_dated_sources": True
          "min_sources": 2,
}

rows = [{"item": item1}, {"item": item2}, .....]
content = rows

Interestingly, you can also send additional context in each dataset item for built-in evaluators. For example, if I were using the built-in task-adherence evaluator to evaluate a risk-assessment agent, it makes sense to pass in my company-specific guidelines to tell the evaluator what constitutes a high vs medium vs low risk rather than letting the evaluator use its own inherent judgment.

COMPANY_SYSTEM_PROMPT = "If they experience rashes, make as high risk, ...."

{
    "query": "I used your medicines and experiened rashes."
    "task_adherence_query": [
        {
            "role": "system",
            "content": COMPANY_SYSTEM_PROMPT,
        },
        {
            "role": "user",
            "content": "I used your medicines and experiened rashes.",
        },
    ],
}

And later on, to make sure it is used, I will update the data_mapping

"data_mapping": 
    {
        "query": "{{item.task_adherence_query}}",
        "response": "{{sample.output_items}}",
    }

So there are two distinct inputs:

• input_messages calls the agent. (The agent already receives its system instructions from its registered definition, so no need to pass the system prompt again).
• task_adherence_query gives the evaluator both the system prompt and user query.

Final Thoughts

Creating the scoring function was the easy part. The more valuable lesson was understanding the boundary between the dataset item, the target-generated sample, and the evaluator’s registered schema.

The main points I would remember are:

• code-based evaluators always expose grade(sample, item);
• for a Foundry model or agent target, generated text is currently available under item["sample"]["output_text"];
• sample is a reserved runtime key—it cannot be renamed to blahblah and still receive the generated response automatically;
• built-in evaluators commonly need explicit mappings, while this code-based evaluator does not;
• registration creates versions, so pin the version used by the evaluation; and
• code-based evaluators return a score only, without a .reason column.

Once these contracts are clear, custom code-based evaluators become a powerful way to turn application-specific expectations into repeatable quality checks.

Happy Learning 🚀