I ran the experiment. Let’s look at what changed.

The Architecture

To understand what changes when BM25 moves into the text index, I split the experiment into two flows: ingestion and query evaluation. Both collections share the same source content and dense embeddings, while each takes a different route for lexical retrieval.

Ingestion: one source, two collections

The PDF is parsed into Markdown and divided into sections, preserving section numbers, titles, and parent relationships. Each section becomes a chunk with a stable point ID and a dense embedding generated once and reused across both collections. This shared foundation keeps differences in document preparation from influencing the comparison.

Collection A follows the sparse-vector approach. Each chunk passes through the BM25 sparse encoder, with average document length calculated from the prepared chunks. The collection stores dense vectors, sparse vectors, and the text payload.

Collection B uses index-native BM25. It receives the same dense vectors and payloads, then builds a full-text index on the text field with BM25 scoring enabled. This removes the separate sparse-vector encoding step from this ingestion path.

Both lexical paths use English stemming and stopword handling, with BM25 parameters set to (k = 1.2) and (b = 0.75). These settings help align the comparison, although they do not guarantee identical tokenization or rankings across implementations.

Querying: a shared dense path, a different lexical path

Each question is embedded once, and that dense query vector is reused for both collections. Collection A also generates a BM25 sparse query vector. Collection B sends the question text directly to its full-text index.

Each collection retrieves 20 dense candidates and 20 lexical candidates. Reciprocal Rank Fusion combines the two ranked lists and returns the top five results. The candidate limits and fusion method stay the same, so the comparison focuses on the lexical retrieval path.

Evaluation: from retrieved sections to generated answers

The retrieved sections are compared with the reference sections using Recall@k, Precision@k, MRR, and nDCG@k. This checks whether the expected evidence appears and how highly it ranks.

The flow then continues beyond the retrieval diagram. Each collection’s context is passed to the same Ollama-hosted generation model with the same prompt and settings. DeepEval evaluates the resulting answers for relevancy, faithfulness, contextual precision, contextual recall, and correctness.

Together, these checks connect the architectural change to its practical consequence: whether a different lexical path changes the evidence retrieved and the answers produced.

What Is This Feature All About?

With Qdrant 1.20, a full-text index can do more than identify matching text. It can rank that text using BM25 directly, giving hybrid retrieval a lexical search path without separately generated BM25 sparse vectors.

Previously, the sparse-vector approach required encoding documents during ingestion and encoding questions at query time. With index-native BM25, you enable scoring on a text payload index and query it using plain text. Qdrant maintains the corpus statistics needed for scoring, including average document length.

The distinction matters as a collection grows. In the sparse approach described by Qdrant, average document length is incorporated during encoding, so refreshing it requires re-encoding documents. The native index manages that statistic internally. It also lets you adjust the BM25 parameters k and b per query without rebuilding the index.

For this experiment, the useful part is how naturally the text query fits into hybrid search. It can sit alongside a dense-vector query in the same Query API, with Reciprocal Rank Fusion combining their results. Dense embeddings still handle semantic retrieval; the full-text index supplies the BM25-ranked lexical candidates.

The opportunity is a simpler lexical pipeline. Whether it preserves retrieval quality depends on the documents, questions, text processing, and dense model paired with it. That is the question the implementation puts to the test.

The Implementation

Document Parsing (doc_parser.py)

The experiment starts by turning the source PDF into structured text. This module provides parsing options through Surya OCR, Indra OCR, LiteParse, and Docling’s vision-language pipeline. The ingestion entry point reads the Markdown output produced through the Surya OCR path.

Once the Markdown is available, MarkdownSectionReader identifies numbered headings and extracts each section’s number, title, body, and parent relationship. It also removes matching document footers to keep repeated page information out of the retrieval content. Each non-empty section becomes a chunk, preserving the document’s logical boundaries and making it possible to trace retrieved evidence back to a specific section.

from docling.datamodel import vlm_model_specs
from docling.datamodel.pipeline_options_vlm_model import InlineVlmOptions
import re
from dataclasses import dataclass

def parse_using_indra_ocr():
    import indra_ocr

    conv = indra_ocr.load(threads=4, mode="ocr")          # mode="auto" | "ocr" | "text"
    markdown_string = ""
    for i, page in conv.convert("./data/information-security-and-business-resiliency-supplement_0.pdf"):
        # print(page.source, page.timings["total_ms"])
        print(page.markdown)                 # headings, lists, tables as HTML
        # print(page.blocks)                    # [{"label", "bbox", "text"}] in reading order
        # print(page.lines)
        markdown_string += page.markdown
    return markdown_string

def parse_using_surya_ocr(pdf_path="./data/information-security-and-business-resiliency-supplement_0.pdf", dpi=96, batch_size=8):
    import logging
    import time
    import pypdfium2 as pdfium
    from markdownify import markdownify
    from surya.inference import SuryaInferenceManager
    from surya.recognition import RecognitionPredictor

    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
    log = logging.getLogger("surya_parse")

    log.info("starting inference backend")
    recognition_predictor = RecognitionPredictor(SuryaInferenceManager())

    pdf = pdfium.PdfDocument(pdf_path)
    total = len(pdf)
    log.info("rendering %d pages at %d dpi", total, dpi)
    images = [page.render(scale=dpi / 72).to_pil() for page in pdf]

    markdown_string = ""
    for start in range(0, total, batch_size):
        batch = images[start:start + batch_size]
        t0 = time.perf_counter()
        predictions = recognition_predictor(batch)
        elapsed = time.perf_counter() - t0
        for offset, page in enumerate(predictions):
            page_html = "\n".join(
                block.html for block in page.blocks if block.html and not block.skipped
            )
            page_markdown = markdownify(
                page_html, heading_style="ATX", keep_inline_images_in=[], strip=["div"]
            )
            markdown_string += page_markdown + "\n\n"
        done = min(start + batch_size, total)
        log.info("pages %d-%d/%d done in %.1fs (%.2f s/page)", start + 1, done, total, elapsed, elapsed / len(batch))
    log.info("finished %d pages", total)
    return markdown_string

def parse_using_liteparse():
    from liteparse import LiteParse

    lp = LiteParse(output_format="markdown")
    _result = lp.parse("./data/information-security-and-business-resiliency-supplement_0.pdf")
    return _result.text

def parse_using_docling_vlm(vlm_model: InlineVlmOptions, input_file: str):
    import datetime
    from docling.datamodel.base_models import InputFormat
    from docling.datamodel.pipeline_options import VlmPipelineOptions
    from docling.document_converter import DocumentConverter, PdfFormatOption
    from docling.pipeline.vlm_pipeline import VlmPipeline

    start_time = datetime.datetime.now()
    pipeline_options = VlmPipelineOptions(
        vlm_options=vlm_model,  # <-- change the model here
    )

    converter = DocumentConverter(
        format_options={
            InputFormat.PDF: PdfFormatOption(
                pipeline_cls=VlmPipeline,
                pipeline_options=pipeline_options,
            ),
        }
    )

    doc = converter.convert(source=input_file).document
    end_time = datetime.datetime.now()
    print(f"Elapsed time: {end_time - start_time}")
    md = doc.export_to_markdown()
    print(md)
    return md

def save_output(markdown: str, output_file: str = "output.md"):
    with open(output_file, "w") as f:
        f.write(markdown)


@dataclass
class Section:
    number: str
    title: str
    text: str
    parent: str


class MarkdownSectionReader:
    HEADING = re.compile(r"^#{1,6}\s*\**\s*(\d+(?:\.\d+)*)\.?\s+\**(.+?)\**\s*$", re.M)
    FOOTER = re.compile(r"^©\s+\d{4}\s+Fannie Mae\s+Page\s+\d+\s+of\s+\d+\s*$", re.M)

    def __init__(self, markdown: str, strip_footers: bool = True):
        self.markdown = self.FOOTER.sub("", markdown) if strip_footers else markdown

    def __iter__(self):
        matches = list(self.HEADING.finditer(self.markdown))
        for i, m in enumerate(matches):
            start = m.end()
            end = matches[i + 1].start() if i + 1 < len(matches) else len(self.markdown)
            number = m.group(1)
            yield Section(
                number=number,
                title=m.group(2).strip(),
                text=self.markdown[start:end].strip(),
                parent=number.rsplit(".", 1)[0] if "." in number else "",
            )

    def front_matter(self) -> str:
        first = self.HEADING.search(self.markdown)
        return self.markdown[: first.start()].strip() if first else self.markdown.strip()


if __name__ == "__main__":
    # parse_using_qwen38()
    # markdown = parse_using_liteparse()
    # markdown = parse_using_indra_ocr()
    markdown = parse_using_surya_ocr()
    # markdown = parse_using_docling_vlm(vlm_model_specs.LIGHTONOCR_MLX,
    #                                    input_file='./data/information-security-and-business-resiliency-supplement_0.pdf')
    save_output(markdown)

    with open("output_surya_ocr.md", encoding="utf-8") as f:
        reader = MarkdownSectionReader(f.read())

    for section in reader:
        print(section.number, section.title, len(section.text))

Collection Ingestion (doc_processor.py)

QdrantIngestor creates the shared foundation for the comparison. It combines each section’s title and body, generates dense embeddings through Ollama in batches, and assigns deterministic point IDs using the document ID and section number. Both collections receive these same embeddings, IDs, and metadata, including section titles, parent sections, positions, and source text.

The two ingestion paths then diverge. For Collection A, the module calculates average chunk length using the sparse model’s text-processing pipeline and supplies it to FastEmbed’s BM25 encoder with k = 1.2 and b = 0.75. The resulting sparse vectors are uploaded alongside the dense vectors, with Qdrant’s IDF modifier enabled on the sparse field.

import time
import uuid
from statistics import mean

from fastembed import SparseTextEmbedding
from ollama import Client as OllamaClient
from qdrant_client import QdrantClient, models

from doc_parser import MarkdownSectionReader, Section


class QdrantIngestor:
    NAMESPACE = uuid.UUID("6f1c2a4e-9b3d-4c7a-8e5f-2d1b0a9c8e7f")
    K, B = 1.2, 0.75

    def __init__(
            self,
            client: QdrantClient,
            doc_id: str,
            collection_a: str = "fnma_hybrid_sparse",
            collection_b: str = "fnma_hybrid_text_index",
            dense_model: str = "embeddinggemma-2:latest",
            ollama_host: str = "http://localhost:11434",
            sparse_model: str = "Qdrant/bm25",
            batch_size: int = 64,
    ):
        self.client = client
        self.doc_id = doc_id
        self.collection_a = collection_a
        self.collection_b = collection_b
        self.batch_size = batch_size
        self.dense_model = dense_model
        self.ollama = OllamaClient(host=ollama_host)
        self.sparse_model = sparse_model
        self.dense_dim = len(self._embed_dense(["dim probe"])[0])
        self.report = {}

    def run(self, reader: MarkdownSectionReader) -> dict:
        sections = [s for s in reader if s.text]
        chunks = self._build_chunks(sections)
        avg_len = self._avg_len(chunks)
        sparse = SparseTextEmbedding(self.sparse_model, k=self.K, b=self.B, avg_len=avg_len)
        self.report["chunks"] = len(chunks)
        self.report["avg_len"] = avg_len
        self._ingest_a(chunks, sparse)
        self._ingest_b(chunks)
        return self.report

    def _embed_dense(self, texts: list[str]) -> list[list[float]]:
        out = []
        for i in range(0, len(texts), self.batch_size):
            out.extend(self.ollama.embed(model=self.dense_model, input=texts[i : i + self.batch_size])["embeddings"])
        return out

    def _build_chunks(self, sections: list[Section]) -> list[dict]:
        texts = [f"{s.title}\n{s.text}" for s in sections]
        t0 = time.perf_counter()
        vectors = self._embed_dense(texts)
        self.report["dense_encode_s"] = time.perf_counter() - t0
        return [
            {
                "id": str(uuid.uuid5(self.NAMESPACE, f"{self.doc_id}:{s.number}")),
                "text": t,
                "dense": v,
                "payload": {
                    "doc_id": self.doc_id,
                    "section": s.number,
                    "title": s.title,
                    "parent": s.parent,
                    "position": i,
                    "text": t,
                },
            }
            for i, (s, t, v) in enumerate(zip(sections, texts, vectors))
        ]

    def _avg_len(self, chunks: list[dict]) -> float:
        probe = SparseTextEmbedding(self.sparse_model).model
        return mean(len(probe._stem(probe.tokenizer.tokenize(c["text"]))) for c in chunks)

    # creates the collection if not exist else the existing collection is directly used
    def _recreate(self, name: str, with_sparse: bool):
        if not self.client.collection_exists(name):
            # self.client.delete_collection(name)
            self.client.create_collection(
                collection_name=name,
                vectors_config={"dense": models.VectorParams(size=self.dense_dim, distance=models.Distance.COSINE)},
                sparse_vectors_config={"sparse": models.SparseVectorParams(modifier=models.Modifier.IDF)} if with_sparse else None,
            )

    def _upsert(self, name: str, points: list[models.PointStruct]):
        for i in range(0, len(points), self.batch_size):
            self.client.upsert(collection_name=name, points=points[i : i + self.batch_size], wait=True)

    def _ingest_a(self, chunks: list[dict], sparse: SparseTextEmbedding):
        t0 = time.perf_counter()
        self._recreate(self.collection_a, with_sparse=True)
        sparse_vectors = list(sparse.embed([c["text"] for c in chunks], batch_size=self.batch_size))
        t_encode = time.perf_counter() - t0
        points = [
            models.PointStruct(
                id=c["id"],
                vector={"dense": c["dense"], "sparse": models.SparseVector(indices=sv.indices.tolist(), values=sv.values.tolist())},
                payload=c["payload"],
            )
            for c, sv in zip(chunks, sparse_vectors)
        ]
        self._upsert(self.collection_a, points)
        self.report["a_sparse_encode_s"] = t_encode
        self.report["a_total_s"] = time.perf_counter() - t0

    def _ingest_b(self, chunks: list[dict]):
        t0 = time.perf_counter()
        self._recreate(self.collection_b, with_sparse=False)
        points = [models.PointStruct(id=c["id"], vector={"dense": c["dense"]}, payload=c["payload"]) for c in chunks]
        self._upsert(self.collection_b, points)
        t_upsert = time.perf_counter() - t0
        self.client.create_payload_index(
            collection_name=self.collection_b,
            field_name="text",
            field_schema=models.TextIndexParams(
                type=models.TextIndexType.TEXT,
                tokenizer=models.TokenizerType.WORD,
                lowercase=True,
                stemmer=models.SnowballParams(type=models.Snowball.SNOWBALL, language=models.SnowballLanguage.ENGLISH),
                stopwords=models.Language.ENGLISH,
                scoring=True,
            ),
            wait=True,
        )
        self.report["b_upsert_s"] = t_upsert
        self.report["b_index_build_s"] = time.perf_counter() - t0 - t_upsert
        self.report["b_total_s"] = time.perf_counter() - t0


if __name__ == "__main__":
    with open("output_surya_ocr.md", encoding="utf-8") as f:
        reader = MarkdownSectionReader(f.read())
    ingestor = QdrantIngestor(QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y"),
                              doc_id="fnma-information-security-and-business-resiliency-supplement")
    print(ingestor.run(reader))

Collection B receives only the dense vectors and shared payloads. After uploading the points, the module creates a scored full-text index on the text field, configured with word tokenization, lowercasing, English stemming, and English stopwords. This is where the separate sparse-encoding step is replaced by index-native BM25. The module records dense encoding time, ingestion timings, and the time spent building Collection B’s text index.

Hybrid Retrieval and Answer Generation (doc_ai.py)

QdrantRetriever reads questions from the golden dataset and generates one dense query embedding per question. That embedding is reused across both collections, keeping the semantic input consistent throughout the comparison.

For Collection A, the question also passes through FastEmbed to produce a BM25 sparse query vector. Collection B instead receives the question as a text query against its indexed text field, with BM25 parameters supplied at query time. Both paths retrieve up to 20 dense candidates and 20 lexical candidates, then use Reciprocal Rank Fusion to select the final five results.

The module formats the retrieved passages with their section references and sends each collection’s context to the same Ollama-hosted generation model. A shared prompt instructs the model to answer strictly from that context, acknowledge missing information, and cite section numbers where available. The question, generated answer, reference answer, retrieved sections, and context are saved separately for each collection.

Retrieval timing is measured around each query path. It includes sparse query encoding for Collection A but excludes the shared dense embedding step and subsequent answer generation, which matters when interpreting the latency comparison.

import json
import logging
import time

from fastembed import SparseTextEmbedding
from ollama import Client as OllamaClient
from qdrant_client import QdrantClient, models

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", datefmt="%H:%M:%S")
log = logging.getLogger("retriever")


class QdrantRetriever:
    K, B = 1.2, 0.75
    SYSTEM = (
        "You answer strictly from the provided context. "
        "If the context does not contain the answer, say so. Cite section numbers where present."
    )

    def __init__(
            self,
            client: QdrantClient,
            collection_a: str = "fnma_hybrid_sparse",
            collection_b: str = "fnma_hybrid_text_index",
            dense_model: str = "embeddinggemma-2:latest",
            llm_model: str = "gemma3:latest",
            ollama_host: str = "http://localhost:11434",
            sparse_model: str = "Qdrant/bm25",
            prefetch_limit: int = 20,
            top_k: int = 5,
            num_ctx: int = 16384,
    ):
        self.client = client
        self.collection_a = collection_a
        self.collection_b = collection_b
        self.dense_model = dense_model
        self.llm_model = llm_model
        self.ollama = OllamaClient(host=ollama_host)
        self.sparse = SparseTextEmbedding(sparse_model)
        self.prefetch_limit = prefetch_limit
        self.top_k = top_k
        self.num_ctx = num_ctx

    def run(self, gold_path: str, out_a: str, out_b: str, limit: int | None = None) -> dict:
        rows = [json.loads(line) for line in open(gold_path, encoding="utf-8") if line.strip()][:limit]
        log.info("processing %d question(s) against %s and %s", len(rows), self.collection_a, self.collection_b)
        results_a, results_b, timings = [], [], {"a": [], "b": []}
        for i, row in enumerate(rows, 1):
            q = row["question"]
            log.info("Q%d/%d: %s", i, len(rows), q[:80])
            dense = self._embed_query(q)
            t0 = time.perf_counter()
            hits_a = self._query_a(q, dense)
            timings["a"].append(time.perf_counter() - t0)
            t0 = time.perf_counter()
            hits_b = self._query_b(q, dense)
            timings["b"].append(time.perf_counter() - t0)
            log.info("  A sections=%s  B sections=%s", [h.payload["section"] for h in hits_a], [h.payload["section"] for h in hits_b])
            results_a.append(self._record(q, hits_a, row["answer"], "A"))
            results_b.append(self._record(q, hits_b, row["answer"], "B"))
        self._write(out_a, results_a)
        self._write(out_b, results_b)
        log.info("wrote %s and %s", out_a, out_b)
        return {
            "questions": len(rows),
            "a_query_mean_s": sum(timings["a"]) / len(rows),
            "b_query_mean_s": sum(timings["b"]) / len(rows),
        }

    def _embed_query(self, text: str) -> list[float]:
        return self.ollama.embed(model=self.dense_model, input=text)["embeddings"][0]

    def _query_a(self, text: str, dense: list[float]) -> list[models.ScoredPoint]:
        sv = next(self.sparse.query_embed(text))
        return self.client.query_points(
            collection_name=self.collection_a,
            prefetch=[
                models.Prefetch(query=dense, using="dense", limit=self.prefetch_limit),
                models.Prefetch(
                    query=models.SparseVector(indices=sv.indices.tolist(), values=sv.values.tolist()),
                    using="sparse",
                    limit=self.prefetch_limit,
                ),
            ],
            query=models.FusionQuery(fusion=models.Fusion.RRF),
            limit=self.top_k,
            with_payload=True,
        ).points

    def _query_b(self, text: str, dense: list[float]) -> list[models.ScoredPoint]:
        return self.client.query_points(
            collection_name=self.collection_b,
            prefetch=[
                models.Prefetch(query=dense, using="dense", limit=self.prefetch_limit),
                models.Prefetch(
                    query=models.TextQuery(
                        text=models.TextQueryInput(
                            query=text,
                            scoring=models.Bm25Scoring(bm25=models.Bm25Params(k=self.K, b=self.B)),
                        )
                    ),
                    using="text",
                    limit=self.prefetch_limit,
                ),
            ],
            query=models.FusionQuery(fusion=models.Fusion.RRF),
            limit=self.top_k,
            with_payload=True,
        ).points

    def _generate(self, question: str, context: list[str]) -> str:
        joined = "\n\n".join(context)
        response = self.ollama.chat(
            model=self.llm_model,
            messages=[
                {"role": "system", "content": self.SYSTEM},
                {"role": "user", "content": f"Context:\n{joined}\n\nQuestion: {question}"},
            ],
            options={"temperature": 0, "num_ctx": self.num_ctx},
        )
        return response["message"]["content"].strip()

    def _record(self, question: str, hits: list[models.ScoredPoint], ground_truth: str, tag: str) -> dict:
        context = [f"[Section {h.payload['section']}] {h.payload['text']}" for h in hits]
        t0 = time.perf_counter()
        answer = self._generate(question, context)
        log.info("  %s generated in %.1fs (%d chars)", tag, time.perf_counter() - t0, len(answer))
        return {
            "question": question,
            "llm_response": answer,
            "ground_truth": ground_truth,
            "retrieved_sections": [h.payload["section"] for h in hits],
            "context": context,
        }

    @staticmethod
    def _write(path: str, rows: list[dict]):
        with open(path, "w", encoding="utf-8") as f:
            for r in rows:
                f.write(json.dumps(r, ensure_ascii=False) + "\n")


if __name__ == "__main__":
    retriever = QdrantRetriever(QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y"))
    print(retriever.run("golden_dataset.jsonl", "responses_collection_a.jsonl", "responses_collection_b.jsonl"))

Retrieval and Answer Evaluation (doc_evaluator.py)

RagEvaluator checks both the evidence retrieved and the answers produced from it. It first extracts reference section numbers from the golden dataset’s context and compares them with the retrieved sections. Recall@k and Precision@k measure coverage and relevance, while MRR and nDCG@k capture where relevant sections appear in the ranking.

The module then constructs a DeepEval test case using the question, generated answer, reference answer, and retrieved context. Answer relevancy, faithfulness, contextual precision, and contextual recall assess different aspects of the RAG output. A separate correctness criterion checks whether the answer agrees with the reference and covers its key points, explicitly penalizing incorrect section numbers and missing obligations.

import json
import logging
import math
import os
import re
import time
from statistics import mean
from dotenv import load_dotenv, find_dotenv

from deepeval.metrics import (
    AnswerRelevancyMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    FaithfulnessMetric,
    GEval,
)
from deepeval.models import GPTModel
from deepeval.test_case import LLMTestCase, SingleTurnParams

load_dotenv(find_dotenv())

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", datefmt="%H:%M:%S")
log = logging.getLogger("evaluator")


class RagEvaluator:
    SECTION = re.compile(r"Section\s+(\d+(?:\.\d+)*)")

    def __init__(self, collection: str, judge_model: str | None = None, threshold: float = 0.5):
        self.collection = collection
        self.judge = GPTModel(model=judge_model) if judge_model else None
        self.metrics = [
            AnswerRelevancyMetric(threshold=threshold, model=self.judge, include_reason=False),
            FaithfulnessMetric(threshold=threshold, model=self.judge, include_reason=False),
            ContextualPrecisionMetric(threshold=threshold, model=self.judge, include_reason=False),
            ContextualRecallMetric(threshold=threshold, model=self.judge, include_reason=False),
            GEval(
                name="Correctness",
                criteria="Judge whether the actual output is factually consistent with and covers the key points of the expected output. Penalise wrong section numbers and missing obligations.",
                evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
                model=self.judge,
                threshold=threshold,
            ),
        ]

    def run(self, responses_path: str, gold_path: str, out_path: str, limit: int | None = None) -> dict:
        gold = {r["question"]: self._gold_sections(r["context"]) for r in self._read(gold_path)}
        responses = self._read(responses_path)[:limit]
        done = {r["question"] for r in self._read(out_path)} if os.path.exists(out_path) else set()
        log.info("[%s] evaluating %d question(s), %d already done", self.collection, len(responses), len(done))
        for i, r in enumerate(responses, 1):
            if r["question"] in done:
                continue
            t0 = time.perf_counter()
            log.info("[%s] Q%d/%d: %s", self.collection, i, len(responses), r["question"][:80])
            retrieved = r["retrieved_sections"]
            case = LLMTestCase(
                input=r["question"],
                actual_output=r["llm_response"],
                expected_output=r["ground_truth"],
                retrieval_context=r["context"],
            )
            scores = {m.__name__: self._measure(m, case) for m in self.metrics}
            scores.update(self._retrieval_scores(retrieved, gold.get(r["question"], [])))
            self._append(out_path, {"question": r["question"], "retrieved_sections": retrieved, "gold_sections": gold.get(r["question"], []), **scores})
            log.info("[%s] Q%d done in %.1fs: %s", self.collection, i, time.perf_counter() - t0, {k: round(v, 3) for k, v in scores.items()})
        rows = self._read(out_path)
        summary = {k: mean(row[k] for row in rows) for k in rows[0] if isinstance(rows[0][k], (int, float))}
        summary["collection"] = self.collection
        summary["questions"] = len(rows)
        return summary

    def _measure(self, metric, case: LLMTestCase) -> float:
        t0 = time.perf_counter()
        metric.measure(case)
        log.info("    %-26s %.3f  (%.1fs)", metric.__name__, metric.score, time.perf_counter() - t0)
        return float(metric.score)

    def _gold_sections(self, context: str) -> list[str]:
        seen = []
        for s in self.SECTION.findall(context):
            if s not in seen:
                seen.append(s)
        return seen

    def _retrieval_scores(self, retrieved: list[str], gold: list[str]) -> dict:
        if not gold:
            return {"recall_at_k": 0.0, "precision_at_k": 0.0, "mrr": 0.0, "ndcg_at_k": 0.0}
        hits = [1 if s in gold else 0 for s in retrieved]
        recall = sum(hits) / len(gold)
        precision = sum(hits) / len(retrieved)
        mrr = next((1 / (i + 1) for i, h in enumerate(hits) if h), 0.0)
        dcg = sum(h / math.log2(i + 2) for i, h in enumerate(hits))
        idcg = sum(1 / math.log2(i + 2) for i in range(min(len(gold), len(retrieved))))
        return {"recall_at_k": recall, "precision_at_k": precision, "mrr": mrr, "ndcg_at_k": dcg / idcg}

    @staticmethod
    def _read(path: str) -> list[dict]:
        with open(path, encoding="utf-8") as f:
            return [json.loads(line) for line in f if line.strip()]

    @staticmethod
    def _append(path: str, row: dict):
        with open(path, "a", encoding="utf-8") as f:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")


if __name__ == "__main__":
    gold = "golden_dataset.jsonl"
    summary_a = RagEvaluator("fnma_hybrid_sparse").run("responses_collection_a.jsonl", gold,
                                                       "eval_collection_b.jsonl")
    summary_b = RagEvaluator("fnma_hybrid_text_index").run("responses_collection_b.jsonl", gold,
                                                           "eval_collection_a.jsonl")
    print(json.dumps({"A": summary_a, "B": summary_b}, indent=2))

Each question’s scores are appended to an output file, allowing an interrupted evaluation to resume by skipping questions already completed. Finally, the module averages the recorded scores to produce a collection-level summary. This lets the comparison follow the change from lexical retrieval through to the quality of the final answer.

The Evaluation Dataset (golden_dataset.jsonl)

This dataset contains 20 unique, scenario-based questions about Fannie Mae’s Information Security and Business Resiliency Supplement. It serves as the shared evaluation set for both collections, allowing their retrieval and answer quality to be compared against the same reference material.

Each record contains three fields: question, answer, and context. The question presents a practical situation, the answer describes the expected response, and the context provides supporting policy excerpts with section references. The evaluator uses these references to identify which sections should be retrieved.

The questions cover cybersecurity incidents, notification deadlines, access management, third-party responsibilities, data protection, cloud adoption, and business continuity. For example, one asks what happens when a vendor suffers a ransomware attack, while another examines the separate notification obligations triggered by an incident and a regulatory filing.

Many questions require evidence from multiple sections. Retrieving a passage that mentions ransomware is only part of the task; the system may also need clauses covering access restoration, security assessments, and exceptions to explain the full obligation.

This makes the dataset useful for a focused comparison of evidence retrieval and answer completeness. With 20 questions drawn from one document, however, the findings describe this particular workload and should be validated on broader datasets before being generalized.

{"question": "A Company's third-party vendor is hit by ransomware, and Fannie Mae responds by disabling the Company's integration interfaces. Does this event qualify as a Cybersecurity Incident under the Supplement, what must the Company provide before its access is restored, and how does this affect its obligation to perform an independent security assessment?", "answer": "Yes. The definition of Cybersecurity Incident expressly lists ransomware as an example and states that the incident may occur at the Company or at the Company's third party(ies), so a vendor-side ransomware event qualifies. Under Section 4.1, once access is disabled it is restored only after the Company provides an attestation as to the security and safety of its systems and satisfies any other conditions Fannie Mae imposes; that attestation may come either from an independent third-party investigative service provider the Company retained or as a signed statement by two officers of the Company. Under Section 3.3, the Company would normally have to perform an independent security assessment upon any Cybersecurity Incident, but because access was suspended under Section 4.1, an attestation may be substituted for the Independent Security Assessment if mutually agreed upon by both parties.", "context": "Section 2 (Definitions) - Cybersecurity Incident: examples include ransomware, denial of service, BEC and vulnerabilities, occurring at the Company or at the Company's third party(ies). Section 4.1 (Actions by Fannie Mae): Fannie Mae may disable or terminate Company user or system access to Fannie Mae systems, including suspending integration interfaces and APIs; if disabled, access and integration will be restored only after the Company provides an attestation as to the security and safety of its systems and satisfies any other conditions Fannie Mae may impose; such attestation may be provided by an independent third-party investigative service provider the Company has retained or be a signed statement by two officers of the Company. Section 3.3 (Audit and Accountability): perform an independent security assessment of the control environment upon the occurrence of any Cybersecurity Incident and at least annually; following a Cybersecurity Incident where access to Fannie Mae has been suspended pursuant to Section 4.1, an attestation may be substituted in place of an Independent Security Assessment if mutually agreed upon by both parties."}
{"question": "A publicly traded Company identifies a Cybersecurity Incident on Monday at 9:00 AM and files a Form 8-K about it on Wednesday. Describe the two separate 36-hour notification obligations that apply, when each clock starts, to what address notice must go, and whether satisfying one discharges the other.", "answer": "Two distinct 36-hour obligations apply. First, under Section 4, without undue delay and no later than 36 hours after identification of the Cybersecurity Incident (or the reasonable conclusion one may have occurred), the Company must provide Fannie Mae all known details of the incident via e-mail at privacy_office@fanniemae.com, so this clock starts Monday at 9:00 AM. Second, under Section 4.2, because the Company filed a Form 8-K with the SEC relating to the incident, it must provide a link to or copy of that filing to the same e-mail address within 36 hours of the day on which the form was filed, so this second clock is tied to Wednesday's filing. Section 4.2 states this filing obligation is in addition to the general notification obligation, so satisfying one does not discharge the other.", "context": "Section 4 (Cybersecurity Incident Management): without undue delay and no later than 36 hours after identification of the Cybersecurity Incident, or the reasonable conclusion a Cybersecurity Incident may have occurred, and promptly thereafter as requested, provide Fannie Mae via e-mail at privacy_office@fanniemae.com all known details of the Cybersecurity Incident. Section 4.2 (Filings by the Company): if a Company has filed a Form 8-K, Form 6-K, or other Cybersecurity Incident disclosure with the Securities and Exchange Commission, or any other filings with a regulatory body relating to a Cybersecurity Incident, the Company must provide a link to or copy of such filing to Fannie Mae via e-mail at privacy_office@fanniemae.com within 36 hours of the day on which the form was filed; this obligation is in addition to the general obligation to provide notification under this section."}
{"question": "A Company discovers that a mailing containing borrowers' nonpublic personal information was sent to the wrong recipient. Which sections of the Supplement govern this event, what specific details must be sent to the Fannie Mae Privacy Office, what must be done for the affected borrowers, what permission must be sought before naming Fannie Mae, and who bears the resulting third-party losses?", "answer": "A misdirected mailing is expressly cited in Section 4.3 as an example of a Cybersecurity Incident impacting Fannie Mae Confidential Information, and because it involves physical information incorrectly routed outside the Company, Section 4.4 also applies, requiring investigation, correction and mitigation at the Company's expense and notification to privacy_office@fanniemae.com within 36 hours. Under Section 4.3 the Company must inform the Privacy Office of the number of individuals impacted, the jurisdictions involved, the company(ies) involved if more than one entity, a description of the related NPI, and the root cause. It must provide timely written notice to affected borrowers, other impacted parties and any state or federal agencies in accordance with applicable privacy and notification laws, and keep a copy of the notice in the individual loan file. It must request permission from Fannie Mae's Privacy Office before referring to Fannie Mae in any notices to borrowers or regulators. Because the incident relates to NPI and was caused by the Company, the Company must pay all Losses relating to third-party claims and all Losses Fannie Mae incurs complying with its legal obligations.", "context": "Section 4.3 (Additional Company Requirements): while investigating a Cybersecurity Incident that impacts or could potentially impact Fannie Mae Confidential Information (e.g., lost/stolen file; misdirected mailing; Company or third-party Cybersecurity Incident), the Company must inform the Fannie Mae Privacy Office via e-mail at privacy_office@fanniemae.com of the number of individuals impacted, from what jurisdictions, the company(ies) involved if there is more than one entity, description of the related nonpublic personal information (NPI), and root cause; provide timely written notice to affected borrowers, other impacted parties and any state or federal agencies in accordance with applicable privacy, data security and notification laws and maintain a copy of the notice in the individual loan file; request permission from Fannie Mae's Privacy Office to use Fannie Mae's name in any notices; and in connection with any Cybersecurity Incident relating to Nonpublic Personal Information caused by Company, its subcontractors, technology, networks or personnel, Company shall pay and be responsible for all Losses relating to third party claims and all Losses incurred by Fannie Mae in complying with its legal obligations. Section 4.4 (Lost/Stolen/Incorrectly Routed Physical Information): if Company incorrectly routes outside of Company physical information such as paper files or other media which includes Fannie Mae Confidential Information, it must promptly investigate, correct and mitigate at its expense and notify Fannie Mae within 36 hours via privacy_office@fanniemae.com."}
{"question": "A third-party solution provider builds a tool that connects to a Fannie Mae system through an API and incorporates open-source libraries, and it hires contractors to maintain that tool. Is this provider subject to the Supplement, and what specific obligations across the Human Resource Security, System Development, and Vulnerability Management sections apply to it?", "answer": "Yes. The definition of Company explicitly includes integrators, also referred to as third-party solution providers, so the Supplement applies. Under Section 3.2, because the contractors will have access to Fannie Mae systems through indirect means such as an API, the provider must run a formal on-boarding process with background verifications for them, require them to attest to a code of conduct before access, and give them annual security awareness training. Under Section 3.8, because the tool connects to a Fannie Mae system, the provider must maintain a formal SDLC policy that includes secure coding requirements, pre- and post-deployment code scanning, separation of development/testing from production, requirements for documenting and keeping current a software bill of materials, and requirements for assessing Vulnerabilities in third-party code including open-source code. Under Section 3.4, an independent third-party penetration test must be conducted at least annually on the systems or components that connect to a Fannie Mae system, alongside regular vulnerability scanning.", "context": "Section 2 (Definitions) - Company: any entity expressly identified as being subject to this Supplement, including SF Lenders, MF Lenders, document custodians and integrators (also referred to as third-party solution providers). Section 3.2 (Human Resource Security): formal on-boarding process including background verifications for all employee candidates and contractors that will have access to Fannie Mae Confidential Information or Fannie Mae systems, either directly or through indirect means such as an application programming interface; code of conduct attestation prior to gaining access; at least annual information security awareness training. Section 3.8 (System Development and Change Management): if the Company develops, deploys, updates or maintains applications or tools that will access, store or process Confidential Information or connect to a Fannie Mae system, it must maintain a formal SDLC policy including secure coding, code scanning pre- and post-deployment, separation between development/testing and production, testing prior to production, defect management, documenting and assuring currency of software bill of materials, and assessing Vulnerabilities in third party code including open-source code. Section 3.4 (Vulnerability Management): an independent third-party penetration test conducted at least annually on systems or components used to store, access, process or transmit Confidential Information or connect to a Fannie Mae system, and regular vulnerability scanning/testing."}
{"question": "Compare the two attestations referenced in the Supplement: the annual attestation about the Information Security Program and the attestation required to restore access after a Cybersecurity Incident. Who may execute each, what does each attest to, and what alternative mechanism is available to SF Lenders for one of them?", "answer": "The annual attestation under Section 3 must be executed by a duly authorized corporate officer and attests that the Information Security Program and its associated programs, plans, policies, standards and procedures meet the requirements of the Supplement; for SF Lenders, this attestation may be provided through the annual Form 582 process described in the SF Guides. The post-incident attestation under Section 4.1 is required only after Fannie Mae has disabled the Company's access, attests to the security and safety of the Company's systems, and may be provided either by an independent third-party investigative service provider the Company has retained or as a signed statement by two officers of the Company. The annual attestation needs one officer and is recurring; the restoration attestation needs two officers or an independent investigator and is event-driven, with no Form 582 alternative.", "context": "Section 3 (Information Security Program): annually provide a written attestation executed by a duly authorized corporate officer that the Information Security Program and its associated programs, plans, policies, standards and procedures meet the requirements of this Supplement; for SF Lenders, this attestation may be provided through the annual Form 582 process described in the SF Guides. Section 4.1 (Actions by Fannie Mae): if disabled, the Company's access and integration to Fannie Mae systems will be restored only after the Company provides an attestation as to the security and safety of its systems; such an attestation may be provided to Fannie Mae either by an independent third-party investigative service provider the Company has retained or be a signed statement by two officers of the Company attesting to the safety and security of its systems."}
{"question": "A Company outsources document storage to a vendor that keeps Fannie Mae Confidential Information unencrypted at rest, and the vendor is breached. Under the Supplement, who is responsible for this failure, what should the Company's contract with the vendor have addressed, and what encryption-related control was violated?", "answer": "Under Section 3.14, the Company is responsible for its vendors' compliance with the Supplement and for all failures by a vendor to comply to the same extent as if the Company itself had committed the failure, so the Company bears responsibility. Section 3.14 also requires the Company to have an agreement in place with each vendor covering adequate considerations for security and resiliency consistent and in compliance with the Supplement, and to run a vendor risk management program ensuring vendor controls are at least as protective as the Company's own Information Security Program. The violated control is in Section 3.10, which requires encrypting data in-transit and at-rest as part of the controls to identify, classify and protect Confidential Information. Because the vendor breach involves unauthorized access to Confidential Information, it is a Cybersecurity Incident at the Company's third party, triggering the Section 4 reporting duties on the Company.", "context": "Section 3.14 (Supply Chain Risk Management): the Company must develop, document and implement a formal vendor risk management program to ensure the controls of new and existing vendors align with and are at least as protective as the Company's Information Security Program and those required in this Supplement; the Company must have an agreement in place with each vendor that covers adequate considerations for security and resiliency consistent and in compliance with the requirements noted in this Supplement; the Company is responsible for its vendors' compliance with the obligations in this Supplement and for all failures by a vendor to comply to the same extent as if such failure were committed by the Company. Section 3.10 (Data Protection and System Security): controls include encrypting end user devices and encrypting data in-transit and at-rest. Section 2 (Definitions) - Cybersecurity Incident includes accidental or unauthorized access to Confidential Information occurring at the Company or at the Company's third party(ies)."}
{"question": "Fannie Mae issues an electronic bulletin amending the Supplement, and the amended term conflicts with a provision in the Company's Lender Contract. Walk through the timeline for the bulletin taking effect, the Company's window and consequence for rejecting it, what happens if the Company says nothing but keeps selling loans, and which document controls when the conflict arises.", "answer": "Under Section 6, Fannie Mae must issue each bulletin at least 20 calendar days before its effective date, and the amendment applies prospectively on the date specified in the bulletin. The Company may reject the bulletin by written notice to Fannie Mae within 15 calendar days after receipt, but in that case Fannie Mae may terminate the Company's Underlying Agreements effective as of the bulletin's effective date or the written notice. If the Company provides no rejection notice within that period, its continued business engagements and continued sale to or servicing of loans for Fannie Mae constitute acknowledgment of acceptance. When the amended term conflicts with the Lender Contract, which is one of the Underlying Agreements, the Conflicts provision states that the Supplement's term takes precedence with respect to its subject matter, while all non-conflicting terms of the Underlying Agreements remain in full force.", "context": "Section 6 (General Requirements) - Issued Bulletins; Amendments: Fannie Mae may issue hard-copy or electronic bulletins amending the Supplement on a prospective basis, effective on the date specified by Fannie Mae in the bulletin; Fannie Mae will issue each bulletin at least 20 calendar days before its effective date; the Company may reject any bulletin by providing written notice to Fannie Mae within 15 calendar days after receipt, in which case Fannie Mae may terminate the Company's Underlying Agreements effective as of the effective date of the bulletin or written notice; unless the Company provides such rejection notice, continued business engagements and continued sale to or servicing of loans for Fannie Mae is an acknowledgment of acceptance. Conflicts: if there is any conflict between a term in this Supplement and the Underlying Agreements, the term in this Supplement takes precedence with respect to the subject matter stated in this Supplement; all other terms of the Underlying Agreements not in conflict remain in full force and effect. Section 2 (Definitions) - Underlying Agreements include the Integration Agreement, Lender Contract, MF Guide, Requirements for Document Custodians and SF Guides."}
{"question": "An employee with a privileged administrator account resigns, and six months later it is discovered that the account was still active and had been used from a remote location. Identify every control in the Access Management and Physical and Environmental sections that this scenario implicates, and explain which authentication requirement is mandatory rather than conditional for such an account.", "answer": "Section 3.1 requires access procedures that remove access upon termination or transfer, evaluate inactivity, and determine when access is no longer needed, and it requires privileged account processes that monitor role changes and enforce revocation to ensure timely removal of dormant accounts and access no longer required, such as when a user has terminated employment. It also requires that all human and non-human accounts be reviewed and certified by the entitlement owner at least annually, which would have caught a six-month-old orphaned account, and requires remote access requirements with mechanisms to monitor and control remote access. Section 3.5 separately requires steps to remove physical and logical access immediately upon the voluntary or involuntary departure of an authorized individual. On authentication, Section 3.1 states that MFA requirements apply 'where applicable' in general, but it is unconditional that MFA must be implemented for access to privileged accounts, so this administrator account was required to have MFA.", "context": "Section 3.1 (Access Management): procedures must include steps for approving access, removing access upon termination or transfer, evaluating inactivity, and determining when access is otherwise no longer needed; access for all human and non-human (system) user accounts should be reviewed and certified by the entitlement owner or user manager at least annually; requirements and processes for managing privileged accounts, including monitoring changes to roles or attributes and enforcing an access revocation process to ensure timely removal of dormant accounts and access no longer required such as when the user has terminated employment; requirements for the use of Multi-Factor Authentication where applicable; remote access requirements including mechanisms to monitor and control remote access; MFA must be implemented for access to privileged accounts. Section 3.5 (Physical and Environmental Controls): steps to remove physical and logical access immediately upon the voluntary or involuntary departure of an authorized individual."}
{"question": "A Company's contract with its subservicer, which handles servicing of Fannie Mae loans, is about to expire without renewal. Using the definition of Business Continuity Procedures together with Sections 5.1 and 5.2, explain why this is a business continuity event under the Supplement and what the Company must already have in place to address it.", "answer": "The definition of Business Continuity Procedures explicitly covers continuing operations upon the termination or expiration of a contract that is material to the Company's ability to service Fannie Mae loans or comply with the Lender Contract, so a lapsing subservicer contract is a covered disruption, not merely a commercial matter. Section 5.1 requires the Business Continuity Plan to ensure the Company can recover critical business operations specifically in the event of the expiration or termination of any contract material to the sale or servicing of Fannie Mae loans. Section 5.2 then requires the Company to establish and document contingencies for third-party and fourth-party vendor relationships and to confirm it can recover critical operations if third-party vendors such as subservicers experience the expiration or termination of such a material contract, suffer business failure, or fail to maintain their own Business Continuity or Disaster Recovery Procedures. The plan must also include third-party dependency information and a crisis communication strategy with points of contact for vendor dependencies.", "context": "Section 2 (Definitions) - Business Continuity Procedures: procedures to continue operations if adverse conditions occur, such as a storm, a fire, a crime, a disruption of critical servicing functions, or the termination or expiration of a contract that is material to Company's ability to originate loans for sale to Fannie Mae, service Fannie Mae loans, or comply with the Lender Contract. Section 5.1 (Business Continuity Plan): ensure the Company's ability to recover critical business operations if (A) there is a disruption or disaster, including to back-up systems, and (B) in the event of the expiration or termination of any contract that is material for the Company's sale or servicing of Fannie Mae loans; the plan must include third party dependency information and a crisis management plan with points of contact for Fannie Mae and vendor dependencies. Section 5.2 (Additional Requirements): establish and document contingencies for third party and fourth party vendor and service provider relationships; confirm the Company can recover critical business operations if any third party vendors such as subservicers, third-party originators and outsourcing firms fail to maintain Business Continuity or Disaster Recovery Procedures, suffer complete business failure or dissolution, or experience the expiration or termination of a material contract."}
{"question": "A Company decides to move its loan origination system, which stores Fannie Mae Confidential Information, to a public cloud provider. Which three distinct sections impose obligations on this decision, and how do the identity-and-access requirements for the cloud relate back to the Access Management section?", "answer": "Three sections apply. Section 3.13 is triggered because the Company leverages Cloud Computing to store Confidential Information, so it must establish cloud computing policies defining its cloud strategy, understand its security posture, concentration risk and dependencies, use providers on current regularly upgraded technology, use secure-by-design principles, and secure access to the cloud environment consistent with the Supplement. Section 3.14 applies because the cloud provider is a vendor, so it must be assessed under the formal vendor risk management program, covered by an agreement addressing security and resiliency, and the Company remains responsible for the provider's compliance. Section 3.10 continues to apply, requiring encryption in-transit and at-rest and data loss prevention over public communications. For identity and access, Section 3.13 requires the Company to implement identity access management procedures consistent with the Supplement for accessing cloud data and services, which means the Section 3.1 requirements such as unique user IDs, least privilege, annual access certification, lockout and timeout rules, and mandatory MFA for privileged accounts extend to the cloud environment.", "context": "Section 3.13 (Cloud Computing): if the Company leverages Cloud Computing to store, process, access or transmit Confidential Information or connect to a Fannie Mae system, it must establish cloud computing policies defining the Company's cloud strategy; understand its security posture, concentration risk and related dependencies; use cloud providers that base their services on current technology with regular upgrades tested for security; implement identity access management procedures consistent with this Supplement for accessing data and services in the cloud; secure access to the cloud environment consistent with this Supplement; and utilize secure-by-design principles. Section 3.14 (Supply Chain Risk Management): formal vendor risk management program, agreements with each vendor covering security and resiliency, Company responsible for vendor failures. Section 3.10 (Data Protection and System Security): encrypting data in-transit and at-rest; data loss prevention for outgoing transmissions over public communication. Section 3.1 (Access Management): unique user IDs, least privilege, annual access review, lockout and timeout, MFA for privileged accounts."}
{"question": "List every recurring activity in the Supplement that must occur at least annually, grouping them by whether they belong to the Information Security Program, the Cybersecurity Incident program, or the Business Continuity program, and identify which of them also have an additional event-driven trigger beyond the annual cycle.", "answer": "Information Security Program (Section 3): review of the Information Security Program itself; the written attestation by a corporate officer; certification of access for all human and non-human accounts; security awareness training; the independent security assessment, which is also event-triggered upon any Cybersecurity Incident; the independent third-party penetration test; review of individuals with physical access to facilities; and, if wireless is offered, review of approved wireless networks. Cybersecurity Incident program (Section 3.6): the test of the incident response plan with lessons learned incorporated. Business Continuity program (Section 5): testing of the Business Continuity Plan, which is also triggered when major changes occur to the plan; the annual threat risk assessment; review and approval of the BCP by the governing body, which is also triggered when material changes occur; and the annual tabletop or similar exercise of business and technology continuity documentation. The three with additional event-driven triggers are the independent security assessment (any Cybersecurity Incident), BCP testing (major changes), and governing-body approval of the BCP (material changes).", "context": "Section 3: at least annually review the Information Security Program; annually provide a written attestation. Section 3.1: access reviewed and certified at least annually. Section 3.2: at least annually conduct information security awareness training. Section 3.3: independent security assessment upon any Cybersecurity Incident and at least annually. Section 3.4: independent third-party penetration test at least annually. Section 3.5: periodic review of physical access at least annually. Section 3.12: at least annually review approved wireless networks. Section 3.6: complete on at least an annual basis a test of the incident response plan and incorporate lessons learned. Section 5.1: testing of the Business Continuity Plan annually at a minimum or when major changes occur. Section 5.2: conduct an annual threat risk assessment; governing body ensures the BCP is reviewed and approved at least annually or when material changes occur; exercise continuity planning documentation annually through tabletop or similar exercises."}
{"question": "A denial-of-service attack against one of the Company's hosting providers degrades the Company's ability to deliver loan files to Fannie Mae, but no Confidential Information is accessed or exfiltrated. Does the Supplement treat this as a reportable event, and under which two separate notification requirements, with what content and timing?", "answer": "Yes. The definition of Cybersecurity Incident covers any occurrence affecting the availability of Confidential Information, and it lists a denial of service attack that may affect delivery of services to Fannie Mae as an example, explicitly including events at the Company's third parties, so exfiltration is not required. Two notification requirements apply. Under Section 3.6, the Company must notify Fannie Mae of any disruptions, including technology-related ones, that could impair business conducted with Fannie Mae or the Company's ability to perform its contractual obligations, and it states that disruptions rising to the level of a Cybersecurity Incident must be reported per Section 4. Under Section 4, within 36 hours of identification the Company must e-mail privacy_office@fanniemae.com all known details, including investigations and technical indicators of compromise, tactics, techniques and procedures and timing, whether Fannie Mae systems or Confidential Information were accessed, names of third-party incident response providers, any law enforcement engaged, contact information for the investigation lead, and the extent to which Confidential Information was accessed, taken or exposed, which in this case would be reported as none.", "context": "Section 2 (Definitions) - Cybersecurity Incident: any occurrence affecting the confidentiality, integrity, or availability of Confidential Information; examples include a denial of service attack which may affect the delivery of the services to Fannie Mae, occurring at the Company or at the Company's third party(ies). Section 3.6 (Cyber Incident Management and Response): provide notification to Fannie Mae of any disruptions, including those related to technology, which may impact operations that could potentially impair or impact business conducted with Fannie Mae or the Company's ability to perform its obligations; disruptions rising to the level of a Cybersecurity Incident must be reported in accordance with Section 4. Section 4: within 36 hours provide all known details including related investigations and technical indicators of compromise, all tactics, techniques and procedures, timing, whether Fannie Mae systems or Confidential Information have been accessed, names of third-party incident response providers, law enforcement agencies engaged, contact information for the investigation lead, and the extent to which Fannie Mae Confidential Information was accessed, taken or exposed."}
{"question": "A Company runs a batch processing service account that several operations staff use interchangeably to run jobs touching Fannie Mae data. Reconcile the Supplement's unique-user-ID rule with its allowance for system accounts, and explain what the Audit and Accountability section requires that determines whether this arrangement is permissible.", "answer": "Section 3.1 requires that every user have a unique user ID and prohibits shared accounts, but carves out built-in and system accounts where individual usage can be tracked. A non-human service account is therefore permitted only if the Company can attribute each use to the specific individual who invoked it. Section 3.3 supplies the test: the Company must maintain written logging and monitoring requirements that ensure logs retain relevant, usable and timely information sufficient to identify user access and system activities, protect those logs from unauthorized modification or deletion, and monitor them for unauthorized disclosure or modification of Confidential Information. If staff share the service account's credentials without any mechanism that ties each job run to an identified person, individual usage cannot be tracked and the arrangement violates Section 3.1. Additionally, Section 3.1 requires non-human IDs to authenticate through a private and controlled method and to be included in the annual access certification by the entitlement owner.", "context": "Section 3.1 (Access Management): every user must have a unique user ID; no shared accounts may be used beyond built-in and system accounts where individual usage can be tracked; access for all human and non-human (system) user accounts should be reviewed and certified by the entitlement owner or user manager at least annually; the Company must require employees and system accounts (non-human ids) to authenticate through a private and controlled method. Section 3.3 (Audit and Accountability): develop and maintain written requirements for logging and monitoring of activities within information systems, including log retention periods, to ensure logs retain relevant, usable and timely information sufficient to identify user access and system activities; implement controls to protect logs from access, modification or deletion by unauthorized personnel; establish processes to monitor logs for unauthorized disclosure, modification, deletion or replication of Confidential Information."}
{"question": "An employee's laptop containing Fannie Mae borrower data is stolen from a car; the laptop had full-disk encryption disabled. Trace this event through the definition of Cybersecurity Incident, the specific Data Protection control that was not followed, the general reporting obligation, and the additional obligations that arise because the data is NPI.", "answer": "The definition of Cybersecurity Incident includes the loss of, or unauthorized access to, Confidential Information, and Fannie Mae Confidential Information expressly includes information about Fannie Mae borrowers, so the stolen laptop is a Cybersecurity Incident. The control not followed is in Section 3.10, which requires encrypting end user devices such as laptops, tablets and cell phones specifically to protect data if devices are lost or stolen, under a formal data management and Encryption use policy. Under Section 4, the Company must promptly investigate, correct and mitigate at its own expense, identify all Fannie Mae Confidential Information affected, and within 36 hours e-mail privacy_office@fanniemae.com all known details including law enforcement engaged and the extent of exposure. Because borrower data is nonpublic personal information, Section 4.3 adds obligations: report the number of individuals and jurisdictions, a description of the NPI and the root cause; provide timely written notice to affected borrowers and regulators under applicable law and retain a copy in each loan file; obtain Privacy Office permission before naming Fannie Mae in notices; and bear all Losses from third-party claims and Fannie Mae's compliance costs since the incident was caused by Company personnel.", "context": "Section 2 (Definitions) - Cybersecurity Incident: loss of, or accidental or unauthorized access to, Confidential Information; Fannie Mae Confidential Information means Confidential Information about Fannie Mae borrowers, collateral, Fannie Mae data and servicing data. Section 3.10 (Data Protection and System Security): maintaining a formal data management and Encryption use policy; encrypting end user devices (laptops, tablets, cell phones) to protect data if devices are lost or stolen. Section 4: promptly investigate, correct and mitigate at the Company's expense, identifying all Fannie Mae Confidential Information affected; within 36 hours provide all known details via privacy_office@fanniemae.com. Section 4.3: inform the Privacy Office of number of individuals impacted, jurisdictions, description of NPI and root cause; timely written notice to affected borrowers and agencies, copy in loan file; request permission to use Fannie Mae's name; Company pays all Losses for incidents relating to NPI caused by Company personnel."}
{"question": "Explain how the Supplement determines what counts as Fannie Mae Confidential Information for an MF Lender versus an SF Lender, and trace the chain of defined terms that makes the Consolidated Technology Guide relevant to both.", "answer": "The definition of Fannie Mae Confidential Information gives a common core covering information about Fannie Mae borrowers, collateral, Fannie Mae data and servicing data, Fannie Mae technologies, products and services, and then adds that it includes anything defined as Confidential Information under the SF Guides for SF Lenders and under the MF Guide for MF Lenders. The term SF Guides is defined as the SF Selling Guide, SF Servicing Guide and the Consolidated Technology Guide collectively, while MF Guide is defined as the Multifamily Selling and Servicing Guide, Multifamily Program Rules and the Consolidated Technology Guide. The Consolidated Technology Guide, defined as the Fannie Mae Consolidated Technology Guide as amended from time to time, therefore sits inside both guide definitions, so confidentiality definitions from it feed into Fannie Mae Confidential Information for both lender types. The SF Lender and MF Lender terms themselves are defined by reference to the SF Selling Guide and MF Guide respectively, and both guides are also Underlying Agreements over which the Supplement takes precedence in a conflict.", "context": "Section 2 (Definitions) - Fannie Mae Confidential Information: Confidential Information about Fannie Mae borrowers, collateral securing a Fannie Mae loan, Fannie Mae data including servicing data, existing or proposed Fannie Mae technologies, products or services and Fannie Mae's business; Fannie Mae Confidential Information includes anything that would be defined as Confidential Information under the SF Guides for SF Lenders and under the MF Guide for MF Lenders. SF Guides: collectively the SF Selling Guide, SF Servicing Guide and Consolidated Technology Guide. MF Guide: collectively the Multifamily Selling and Servicing Guide, Multifamily Program Rules and the Consolidated Technology Guide. Consolidated Technology Guide: the Fannie Mae Consolidated Technology Guide, as amended, restated, supplemented or otherwise modified from time to time. SF Lender: a lender as described in the SF Selling Guide, including sellers, servicers and seller/servicers. MF Lender: a Lender as defined in the MF Guide. Underlying Agreements include the Lender Contract, MF Guide and SF Guides; Section 6 Conflicts: Supplement takes precedence."}
{"question": "A Company provides a guest Wi-Fi network in its offices and allows underwriters to work remotely over that network and from home. Combine the Network Security and Access Management requirements to describe the full set of controls that must be in place, and identify which requirement is conditional on the Company offering wireless at all.", "answer": "From Section 3.12, the Company must have firewalls blocking inbound and outbound public traffic unless permitted by policy, restrict ports, protocols and services to approved ones, monitor for anomalous traffic, review firewall rules on a defined frequency and on notable network changes, and control, secure and monitor wireless access points. Because the Company offers wireless networks, the conditional block of Section 3.12 also applies: a WLAN authentication method meeting or exceeding current industry best practice on encryption strength, at least annual reviews of approved wireless networks to verify authorized users and access points, and password-protected restriction of router and network device access to authorized administrators. From Section 3.1, remote work by underwriters triggers remote access requirements with mechanisms to monitor and control remote access, MFA where applicable, account lockout after failed logins, inactivity timeouts, and least-privilege access. Section 3.10 additionally requires encrypting data in transit and encrypting the end user devices used remotely. The WLAN authentication, annual wireless review and router-protection requirements are the ones conditional on the Company offering wireless networks.", "context": "Section 3.12 (Network Security and Management): implement firewalls to block inbound traffic from and outbound traffic to public networks unless permitted; manage and restrict ports, protocols and services to those required and approved; monitor network traffic and detect anomalous traffic; review and approve firewall rules on a defined frequency and whenever a notable change in network technology occurs; control, secure and monitor wireless access points. In addition, if the Company offers wireless networks for network users, it must implement a WLAN Authentication method meeting or exceeding current industry best practices on Encryption strength; at least annually review approved wireless networks to validate authorized users and access points; and implement controls including password protection to restrict router and network device access to authorized users and administrators. Section 3.1 (Access Management): remote access requirements including mechanisms to monitor and control remote access; MFA where applicable; locked accounts after multiple failed logins and timeout for inactivity; least privilege. Section 3.10: encrypting end user devices and data in-transit."}
{"question": "An independent auditor's annual security assessment identifies a critical vulnerability in a server that transmits loan data to Fannie Mae, and the vendor patch has been available for two months. Which three sections together define what the Company must do, what the standard for timing is, and how the finding must be tracked?", "answer": "Section 3.3 obliges the Company to remediate findings raised in any independent security assessment that are relevant to services provided to or on behalf of Fannie Mae or loans sold to or serviced for Fannie Mae, with remediation completed within a commercially reasonable timeframe consistent with standard industry practices. Section 3.4 requires the finding to be entered into the vulnerability management program's inventory of all identified Vulnerabilities with a risk rating and remediation status tracked against a risk-based schedule, so a critical rating should drive priority. Section 3.9 requires the Company's written patch management program to perform timely software updates and patches based on the vulnerability management program and to maintain a process for testing and consistently applying patches as they become available without compromising the confidentiality, integrity and availability of Fannie Mae Confidential Information; a two-month-old unapplied patch on a system transmitting Fannie Mae data is exactly what this provision targets. Section 3.8 further requires that applying the patch follow the formal change management process with tracking, testing and documented back-out procedures.", "context": "Section 3.3 (Audit and Accountability): the Company must remediate findings relevant to the services provided to or on behalf of Fannie Mae or loans being sold to or serviced for Fannie Mae raised in any independent security assessment; remediation must be completed within a commercially reasonable timeframe and consistent with standard industry practices. Section 3.4 (Vulnerability Management): a process to identify, analyze and remediate identified Vulnerabilities within commercially reasonable timelines, including an inventory of all identified Vulnerabilities, a risk rating of their level of risk, and their remediation status based on a risk-based schedule. Section 3.9 (Patch Management): implement and maintain a written patch management program; perform timely software updates and patches based on the Company's Vulnerability management program, and maintain a process for testing and ensuring updates and patches are consistently applied as they become available without compromising the confidentiality, integrity and availability of Fannie Mae Confidential Information. Section 3.8: all changes to the technology environment including patches must follow a formal change management process including tracking, testing and documented back-out procedures."}
{"question": "A Fannie Mae relationship manager verbally tells a Company during a call that it need not perform the annual penetration test this year; the Company skips it, and later Fannie Mae raises the omission as a breach. Using the General Requirements, explain whether the verbal statement protects the Company, what form a valid waiver must take, and whether a prior written waiver of a different breach would help.", "answer": "The verbal statement does not protect the Company. The No Implied Waiver provision in Section 6 states that no term of the Supplement will be deemed waived and no breach excused unless the waiver or excuse is in writing and executed by a duly authorized representative of Fannie Mae, so an oral assurance from a relationship manager has no effect on the Section 3.4 obligation to conduct an independent third-party penetration test at least annually. A valid waiver would need to be a written instrument signed by an authorized Fannie Mae representative; Section 6 also notes that outside the bulletin process, amendments require a writing executed by a duly authorized representative of each party. A prior written waiver of a different breach would not help either, because the same provision states that any waiver by Fannie Mae of a breach does not constitute consent to, waiver of, or excuse for any different or subsequent breach.", "context": "Section 6 (General Requirements) - No Implied Waiver: no term, provision or clause of this Supplement will be deemed waived, and no breach excused unless such waiver or excuse is in writing and executed by a duly authorized representative of Fannie Mae; any waiver by Fannie Mae of a breach by the Company does not constitute a consent to, waiver of, or excuse for any different or subsequent breach. Amendments to this Supplement: Fannie Mae may at any time alter or waive any requirements; if a bulletin is not used, the terms may be amended by a writing executed by a duly authorized representative of each party. Section 3.4 (Vulnerability Management): an independent third-party penetration test conducted at least annually on systems used to store, access, process or transmit Confidential Information or connect to a Fannie Mae system."}
{"question": "After a tabletop exercise reveals that the Company's Business Continuity Plan was unknown to the operations staff expected to execute it and that no one had approved the current version, which requirements in Sections 5.1 and 5.2 were not met, and what must happen with the exercise results afterward?", "answer": "Section 5.1 requires the Business Continuity Plan to be available to all resources or staff required to support execution of the plan, so staff being unaware of it is a direct failure, and it requires governance to ensure compliance with BCP requirements. Section 5.2 requires a governing body to provide guidance and support of the plan and to ensure it is reviewed, formally socialized with leadership, and approved at least annually or when material changes occur, so an unapproved current version breaches this. The exercise itself satisfies the Section 5.2 requirement to exercise continuity documentation annually through tabletop or similar exercises, but Section 5.2 then requires the Company to document lessons learned and after actions, share results with Company senior management, and track the after actions to resolution. Section 5.1 additionally requires the plan to be tested annually or when major changes occur, so the governance and communication fixes themselves should be reflected in the plan's crisis management and communication strategy.", "context": "Section 5.1 (Business Continuity Plan): the Business Continuity Plan must include governance to ensure compliance with Business Continuity Plan requirements, a crisis management plan and communication strategy, and testing annually at a minimum or when major changes occur; the Business Continuity Plan must be available to all resources or staff that are required to support the execution of the plan. Section 5.2 (Additional Requirements): have a governing body in place to provide guidance and support of the Business Continuity Plan, ensuring the plan is reviewed, formally socialized with leadership, and approved at least annually or when material changes occur; exercise its business and technology continuity planning documentation annually through tabletop or other similar exercises; document lessons learned and after actions when plans are tested or activated; results should be shared with Company senior management and after actions should be tracked to resolution."}
{"question": "A Company's parent affiliate runs the group-wide Information Security Program, designates the senior security executive, and signs the annual attestation, while the Company itself only sells loans to Fannie Mae. Under Sections 1 and 3, is this structure permitted, what condition applies, and which standard must the affiliate's program align with?", "answer": "The structure is permitted. Section 1 states that all obligations required of the Company under the Supplement may be performed by a Company affiliate on the Company's behalf, and Section 3 restates this for the Information Security Program specifically, allowing the Company or its affiliate to implement the program to the extent the affiliate's program covers the Company. The condition is coverage: the affiliate's program must actually encompass the Company's systems, personnel and Fannie Mae Confidential Information. Regardless of who operates it, the program must at a minimum include the Supplement's requirements, be aligned with or exceed a current industry standard such as the NIST Framework or ISO 27001, have a designated senior executive responsible for it, be reviewed at least annually, be communicated to all applicable personnel, be made available to Fannie Mae on request, and be covered by an annual written attestation executed by a duly authorized corporate officer, which an SF Lender may deliver through the Form 582 process. The Company also remains bound to exercise at least the same care with Fannie Mae systems and Confidential Information as with its own.", "context": "Section 1 (Introduction): all obligations required to be performed by the Company under this Supplement may be performed by a Company affiliate on the Company's behalf. Section 3 (Information Security Program): the Company, or its affiliate (to the extent the affiliate's program covers the Company), at its own cost and expense, must implement an information security program that at a minimum includes the requirements of this Supplement; align it with or exceed a current industry standard such as the NIST Framework or ISO 27001; designate and keep a senior executive responsible for the program; at least annually review the program; communicate requirements to all applicable employees, contractors and personnel; make the program available to Fannie Mae upon request; and annually provide a written attestation executed by a duly authorized corporate officer, which for SF Lenders may be provided through the annual Form 582 process; the Company must exercise at least the same level of care with Fannie Mae systems and Fannie Mae Confidential Information as it does for its own."}

The Results

The Conclusion

This experiment started with a practical question: can we simplify the lexical side of hybrid RAG without losing the evidence needed to answer questions?

For this workload, the results were encouraging. Both collections retrieved the same five sections for 19 of the 20 questions, with identical ordering for 15. Retrieval scores remained close, while the answer-level differences could not be confidently attributed to the change in BM25 implementation.

Qdrant’s index-native BM25 offers a simpler path when the requirement is classical BM25 over text already stored in the payload. It removes separate sparse-vector encoding and average-document-length bookkeeping from that path. Learned sparse representations such as SPLADE or miniCOIL serve different needs and still require sparse vectors.

My takeaway is to treat simplification as something to validate. The dataset, embedding model, text processing, and questions all influence the outcome. Before making the switch in your own RAG system, compare the retrieved evidence and evaluate the answers built from it.

In this experiment, the simpler path held up. Your workload deserves the same check.

Does Index-Native BM25 Change Retrieval Quality? A Two-Collection Experiment in Qdrant 1.20 was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Architecture

Our architecture brings together LlamaIndex, Qdrant, Ollama, MLflow, and Jev to build a complete medical RAG pipeline with hybrid retrieval, automated evaluation, trace monitoring, and result analysis. The workflow is divided into four major components.

1. Data Preparation and Ingestion

We start with the MedQuAD Context GroupedQA dataset from Hugging Face, which contains medical contexts and their corresponding question-answer pairs. A Python script (create_testset.py) randomly selects 50 question-answer pairs and creates a JSONL test set containing questions, ground-truth answers, and context IDs.

A separate ingestion script (ingest_contexts.py) processes the medical contexts without chunking. Ollama's EmbeddingGemma generates dense embeddings, while FastEmbed BM25 generates sparse vectors. Both representations are stored in Qdrant, enabling hybrid retrieval with context-level traceability.

2. RAG Pipeline

The RAG pipeline is built using LlamaIndex, which handles retrieval, response generation, and integration with Qdrant. When a question arrives, hybrid retrieval combines dense semantic search and sparse keyword matching using Reciprocal Rank Fusion (RRF).

The retrieved contexts are passed to Ollama’s Gemma 3, which generates the final answer. Each response includes the generated answer, retrieved context IDs, and source references, allowing us to evaluate both retrieval quality and answer correctness.

3. Evaluation with Jev and LLM Judges

This is where the architecture becomes interesting. Using MLflow GenAI Evaluation, we evaluate every RAG response against its ground-truth answer and retrieved contexts.

We use Jev to evaluate answer coverage and contradictions, while Gemma 3 serves as the traditional LLM judge for the same criteria. Custom retrieval scorers check whether the expected medical context was retrieved and at what rank.

This allows us to compare Jev and LLM-based evaluation on the same questions, responses, and evidence.

4. MLflow Tracking, Monitoring, and Analysis

MLflow acts as the central observability, AI Gateway and evaluation platform. It records experiment configurations, Model Routing, RAG execution traces, model interactions, retrieval results, evaluation scores, and judge feedback.

Jev is accessed through the MLflow AI Gateway, while the Analyzer processes evaluation traces to measure judge agreement, identify disagreements, and produce structured results. The architecture also includes GPT-4o-mini for trace-level issue detection.

Finally, the results are exported into a JSONL report, giving us a clear picture of retrieval performance, answer quality, and how Jev compares with the traditional LLM judge.

5. What Makes This Architecture Interesting?

Instead of evaluating RAG using a single LLM judge, we introduce Jev as an alternative evaluation mechanism and compare both approaches against the same ground-truth medical dataset.

The goal is not just to generate accurate medical answers, but to understand how reliably we can evaluate those answers, detect failures, and trace problems back to retrieval or generation, all within a reproducible MLflow experiment.

The Implementation

create_test_set.py — building the test set

TestsetConfig. Every knob (dataset name, split, sample size, seed, output path) is read from .env into a frozen dataclass. The script has no hard-coded choices, and freezing the config means a run is fully described by one immutable object, which is what you want when you later ask “which test set did this run use?”

context_id. Each MedQuAD context gets a 16-character SHA-1 of its text. This is the join key for the whole experiment: the same function is reused by the ingestion script, so a question in the test set and a document in Qdrant can be matched without ever comparing text.

flatten. The dataset groups several question–answer pairs under one context. We explode that grouping so every QA object becomes its own record, carrying its parent context and context id. Malformed or empty rows are skipped rather than failing the run; a test set should be built from clean examples only.

sample. We shuffle with a fixed seed and take the first N records, allowing at most one record per context. The seed makes the set reproducible; the one-per-context rule spreads the 50 questions across 50 distinct documents so retrieval is tested on breadth, not on five questions about the same page.

write. Records are written as flat JSONL (id, context_id, question, answer, context) with sequential ids. Keeping the shape flat keeps the file reusable; mapping to MLflow’s inputs/expectations structure is the evaluation script’s job.

from __future__ import annotations

import hashlib
import json
import os
import random
from dataclasses import dataclass
from pathlib import Path

from datasets import load_dataset
from dotenv import load_dotenv


@dataclass(frozen=True)
class TestsetConfig:
    dataset: str
    split: str
    size: int
    seed: int
    path: Path

    @classmethod
    def from_env(cls) -> "TestsetConfig":
        load_dotenv()
        return cls(
            dataset=os.getenv("HF_DATASET", "pavanmantha/MedQuAD_Context_GroupedQA"),
            split=os.getenv("HF_SPLIT", "train"),
            size=int(os.getenv("TESTSET_SIZE", "50")),
            seed=int(os.getenv("TESTSET_SEED", "42")),
            path=Path(os.getenv("TESTSET_PATH", "src/data/testset.jsonl")),
        )


class TestsetBuilder:
    def __init__(self, config: TestsetConfig) -> None:
        self.config = config

    @staticmethod
    def context_id(context: str) -> str:
        return hashlib.sha1(context.encode("utf-8")).hexdigest()[:16]

    def load(self):
        return load_dataset(self.config.dataset, split=self.config.split)

    def flatten(self, dataset) -> list[dict]:
        records: list[dict] = []
        for row in dataset:
            context = (row.get("context") or "").strip()
            raw_pairs = row.get("question_answer_pairs")
            if not context or not raw_pairs:
                continue
            try:
                pairs = json.loads(raw_pairs) if isinstance(raw_pairs, str) else raw_pairs
            except json.JSONDecodeError:
                continue
            if not isinstance(pairs, list):
                continue
            cid = self.context_id(context)
            for pair in pairs:
                question = (pair.get("question") or "").strip()
                answer = (pair.get("answer") or "").strip()
                if question and answer:
                    records.append({"context_id": cid, "question": question, "answer": answer, "context": context})
        return records

    def sample(self, records: list[dict]) -> list[dict]:
        rng = random.Random(self.config.seed)
        shuffled = records[:]
        rng.shuffle(shuffled)
        seen: set[str] = set()
        chosen: list[dict] = []
        for rec in shuffled:
            if rec["context_id"] in seen:
                continue
            seen.add(rec["context_id"])
            chosen.append(rec)
            if len(chosen) == self.config.size:
                break
        if len(chosen) < self.config.size:
            raise RuntimeError(f"Only {len(chosen)} distinct contexts; cannot sample {self.config.size}.")
        return chosen

    def write(self, records: list[dict]) -> None:
        self.config.path.parent.mkdir(parents=True, exist_ok=True)
        with self.config.path.open("w", encoding="utf-8") as fh:
            for i, rec in enumerate(records, start=1):
                fh.write(json.dumps({"id": f"q{i:03d}", **rec}, ensure_ascii=False) + "\n")

    def run(self) -> None:
        dataset = self.load()
        records = self.flatten(dataset)
        chosen = self.sample(records)
        self.write(chosen)
        print(f"{len(dataset)} rows -> {len(records)} QA records -> {len(chosen)} written to {self.config.path} (seed={self.config.seed})")


if __name__ == "__main__":
    TestsetBuilder(TestsetConfig.from_env()).run()

qdrant_ingestion.py — populating the hybrid collection

IngestConfig. Qdrant, Ollama, the sparse model and batch sizes come from .env, plus a RECREATE_COLLECTION flag so a rebuild is an explicit decision rather than a default.

ContextIngestor.init. Three clients are wired once: the Qdrant client, an Ollama embedding model for dense vectors, and a QdrantVectorStore with enable_hybrid=True so LlamaIndex also computes a BM25 sparse vector through fastembed. Hybrid is deliberate for medical text, where exact terms (drug names, gene symbols) matter as much as semantic similarity.

node_id. The Qdrant point id is a UUID5 derived from the context id. Deterministic ids make ingestion idempotent: re-running the script upserts the same points instead of duplicating them.

load_nodes. The full dataset is read, deduplicated by context id, and each context becomes a single TextNode. There is no chunking: each row is already one coherent document. The context_id and source go into payload but are excluded from the embedded text, so metadata never contaminates the vector.

ingest. Nodes are handed to VectorStoreIndex with a Qdrant-backed storage context. LlamaIndex embeds in batches, computes the sparse vectors, and upserts to the collection; we never touch the Qdrant API directly for writes.

run. Optionally drop the collection, load nodes, report the point count before and after. The before/after print is the quickest sanity check that the upsert path behaved.

from __future__ import annotations

import os
import uuid
from dataclasses import dataclass

from datasets import load_dataset
from dotenv import load_dotenv
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.schema import TextNode
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient

from create_test_set import TestsetBuilder

NODE_NAMESPACE = uuid.UUID("6f2b9d4e-8a1c-4c7e-9f3a-2d5e7b1c0a44")


@dataclass(frozen=True)
class IngestConfig:
    dataset: str
    split: str
    qdrant_url: str
    qdrant_api_key: str | None
    collection: str
    ollama_base_url: str
    embed_model: str
    sparse_model: str
    embed_batch_size: int
    insert_batch_size: int
    recreate: bool

    @classmethod
    def from_env(cls) -> "IngestConfig":
        load_dotenv()
        return cls(
            dataset=os.getenv("HF_DATASET", "pavanmantha/MedQuAD_Context_GroupedQA"),
            split=os.getenv("HF_SPLIT", "train"),
            qdrant_url=os.getenv("QDRANT_URL", "http://localhost:6333"),
            qdrant_api_key=os.getenv("QDRANT_API_KEY") or None,
            collection=os.getenv("QDRANT_COLLECTION", "medquad_contexts"),
            ollama_base_url=os.getenv("OLLAMA_BASE_URL", "http://localhost:11434"),
            embed_model=os.getenv("OLLAMA_EMBED_MODEL", "embeddinggemma-2:latest"),
            sparse_model=os.getenv("SPARSE_MODEL", "Qdrant/bm25"),
            embed_batch_size=int(os.getenv("EMBED_BATCH_SIZE", "16")),
            insert_batch_size=int(os.getenv("INSERT_BATCH_SIZE", "64")),
            recreate=os.getenv("RECREATE_COLLECTION", "false").lower() == "true",
        )


class ContextIngestor:
    def __init__(self, config: IngestConfig) -> None:
        self.config = config
        self.client = QdrantClient(url=config.qdrant_url, api_key=config.qdrant_api_key)
        self.embed_model = OllamaEmbedding(
            model_name=config.embed_model,
            base_url=config.ollama_base_url,
            embed_batch_size=config.embed_batch_size,
        )
        self.vector_store = QdrantVectorStore(
            client=self.client,
            collection_name=config.collection,
            enable_hybrid=True,
            fastembed_sparse_model=config.sparse_model,
            batch_size=config.insert_batch_size,
        )

    @staticmethod
    def node_id(context_id: str) -> str:
        return str(uuid.uuid5(NODE_NAMESPACE, context_id))

    def load_nodes(self) -> list[TextNode]:
        dataset = load_dataset(self.config.dataset, split=self.config.split)
        seen: set[str] = set()
        nodes: list[TextNode] = []
        for row in dataset:
            context = (row.get("context") or "").strip()
            if not context:
                continue
            cid = TestsetBuilder.context_id(context)
            if cid in seen:
                continue
            seen.add(cid)
            nodes.append(
                TextNode(
                    id_=self.node_id(cid),
                    text=context,
                    metadata={"context_id": cid, "source": self.config.dataset},
                    excluded_embed_metadata_keys=["context_id", "source"],
                    excluded_llm_metadata_keys=["source"],
                )
            )
        return nodes

    def prepare_collection(self) -> None:
        if self.config.recreate and self.client.collection_exists(self.config.collection):
            self.client.delete_collection(self.config.collection)

    def existing_count(self) -> int:
        if not self.client.collection_exists(self.config.collection):
            return 0
        return self.client.count(self.config.collection, exact=True).count

    def ingest(self, nodes: list[TextNode]) -> None:
        storage_context = StorageContext.from_defaults(vector_store=self.vector_store)
        VectorStoreIndex(
            nodes=nodes,
            storage_context=storage_context,
            embed_model=self.embed_model,
            insert_batch_size=self.config.insert_batch_size,
            show_progress=True,
        )

    def run(self) -> None:
        self.prepare_collection()
        nodes = self.load_nodes()
        before = self.existing_count()
        print(f"{len(nodes)} unique contexts | collection '{self.config.collection}' has {before} points")
        self.ingest(nodes)
        print(f"done | collection now has {self.existing_count()} points | dense={self.config.embed_model} sparse={self.config.sparse_model}")


if __name__ == "__main__":
    ContextIngestor(IngestConfig.from_env()).run()

rag_index.py — the index as an MLflow model

This file is a model-from-code definition: MLflow executes it both when logging and when loading the model, so the whole RAG configuration lives in one reproducible place.

IndexConfig. The same Qdrant/Ollama settings as ingestion, plus the generator LLM, temperature, timeout and an optional thinking level. thinking is None when blank, because Ollama rejects the option outright for models that have no thinking mode.

IndexFactory.configure_settings. The global LlamaIndex Settings are set to the Ollama embedding model and Ollama LLM. MLflow serialises Settings with the model, so the logged artifact records exactly which models answered the questions.

IndexFactory.build. Rather than re-indexing, the index is opened from the existing hybrid Qdrant collection with from_vector_store. The last two lines build it and call mlflow.models.set_model, which is how MLflow knows which object this file produces.

from __future__ import annotations

import os
from dataclasses import dataclass

import mlflow
from dotenv import load_dotenv
from llama_index.core import Settings, VectorStoreIndex
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient


@dataclass(frozen=True)
class IndexConfig:
    qdrant_url: str
    qdrant_api_key: str | None
    collection: str
    ollama_base_url: str
    embed_model: str
    llm_model: str
    thinking: str | None
    temperature: float
    request_timeout: float
    sparse_model: str

    @classmethod
    def from_env(cls) -> "IndexConfig":
        load_dotenv()
        return cls(
            qdrant_url=os.getenv("QDRANT_URL", "http://localhost:6333"),
            qdrant_api_key=os.getenv("QDRANT_API_KEY") or None,
            collection=os.getenv("QDRANT_COLLECTION", "medquad_contexts"),
            ollama_base_url=os.getenv("OLLAMA_BASE_URL", "http://localhost:11434"),
            embed_model=os.getenv("OLLAMA_EMBED_MODEL", "embeddinggemma-2:latest"),
            llm_model=os.getenv("OLLAMA_LLM_MODEL", "granite4.2:latest"),
            thinking=os.getenv("OLLAMA_THINKING", "").strip().lower() or None,
            temperature=float(os.getenv("OLLAMA_TEMPERATURE", "0.0")),
            request_timeout=float(os.getenv("OLLAMA_REQUEST_TIMEOUT", "300")),
            sparse_model=os.getenv("SPARSE_MODEL", "Qdrant/bm25"),
        )


class IndexFactory:
    def __init__(self, config: IndexConfig) -> None:
        self.config = config

    def configure_settings(self) -> None:
        Settings.embed_model = OllamaEmbedding(model_name=self.config.embed_model, base_url=self.config.ollama_base_url)
        llm_kwargs = dict(
            model=self.config.llm_model,
            base_url=self.config.ollama_base_url,
            temperature=self.config.temperature,
            request_timeout=self.config.request_timeout,
        )
        if self.config.thinking:
            llm_kwargs["thinking"] = self.config.thinking
        Settings.llm = Ollama(**llm_kwargs)

    def build(self) -> VectorStoreIndex:
        self.configure_settings()
        client = QdrantClient(url=self.config.qdrant_url, api_key=self.config.qdrant_api_key)
        store = QdrantVectorStore(
            client=client,
            collection_name=self.config.collection,
            enable_hybrid=True,
            fastembed_sparse_model=self.config.sparse_model,
        )
        return VectorStoreIndex.from_vector_store(vector_store=store)


index = IndexFactory(IndexConfig.from_env()).build()
mlflow.models.set_model(index)

rag_eval.py — answering, judging, analysing

The two instruction strings. Both judges receive the question (inputs), the candidate answer (outputs) and the reference answer (expectations). No-contradiction asks whether anything in the candidate conflicts with the reference and explicitly says not to penalise omissions. Coverage asks the opposite question: are the essential points present. Splitting correctness into these two signals matters because MedQuAD references are long, and a concise RAG answer can be perfectly correct yet incomplete.

EvalConfig. Tracking URI, experiment name, retrieval depths, the Jev endpoint (addressed as gateway:/<endpoint> so the TypeSafe key stays in the MLflow AI Gateway), the Ollama text judge (ollama:/<model>), the routing band, and an optional EVAL_LIMIT for smoke runs.

HybridRag. Wraps the index as a hybrid query engine (dense top-k plus sparse top-k, fused by Qdrant). answer returns a dict rather than a string: the answer text, the ordered list of retrieved context ids, and their scores. Returning the ids is what lets us measure retrieval separately from generation.

Judges.build. Four make_judge scorers from two instruction strings and two models, so Jev and the text judge see identical prompts. Two code scorers sit beside them: retrieval_hit (was the gold context among the retrieved ones) and retrieval_rank (at which position, 0 for a miss). If a judge fails an answer, these tell you whether the retriever or the generator is to blame.

Testset. Loads the JSONL and maps each row to MLflow’s shape: the question goes into inputs, the reference answer and gold context id into expectations. The reference is visible to judges but never to the RAG.

Analyzer. After evaluation, every trace of the run is pulled back from MLflow. For each assessment we keep the verdict, Jev’s typesafe.probability from the feedback metadata, and the text judge’s rationale. Then, per criterion, three metrics: raw Jev-vs-text agreement, the fraction of Jev verdicts inside the uncertain band, and agreement after routing banded cases to the text judge. This reproduces the “accept when confident, escalate when unsure” experiment from the MLflow post, with the text judge standing in for human labels. Per-example rows are written to JSONL for the report.

RagEvaluation.run. One MLflow run wraps everything: parameters are logged, the index is logged from rag_index.py as a query engine, mlflow.genai.evaluate drives predict_fn across the test set with all six scorers and links the results to the logged model, and the Analyzer’s metrics are logged on top. The run page in MLflow then holds the data, the model, the traces and every judgment together.

from __future__ import annotations

import json
import os
from dataclasses import dataclass
from pathlib import Path

import mlflow
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex
from mlflow.genai import scorer
from mlflow.genai.judges import make_judge

from rag_index import index as rag_index

NO_CONTRADICTION_INSTRUCTIONS = (
    "The user asked the question in {{ inputs }}. A trusted reference answer is in {{ expectations }} "
    "under 'expected_response'. The candidate answer is the 'answer' field in {{ outputs }}. "
    "Is the candidate answer free of any claim that contradicts or is materially inconsistent with the reference answer? "
    "Answer yes if every claim in the candidate is consistent with the reference. "
    "Answer no if at least one claim is contradicted by the reference or would mislead the user. "
    "Do not penalize omissions, brevity, or stylistic differences."
)

COVERAGE_INSTRUCTIONS = (
    "The user asked the question in {{ inputs }}. A trusted reference answer is in {{ expectations }} "
    "under 'expected_response'. The candidate answer is the 'answer' field in {{ outputs }}. "
    "Does the candidate answer cover the key points of the reference answer needed to answer the question? "
    "Answer yes if the essential facts are present in substance, even if worded differently or more briefly. "
    "Answer no if an essential point is missing or replaced by unrelated content."
)


@dataclass(frozen=True)
class EvalConfig:
    tracking_uri: str
    experiment: str
    testset_path: Path
    results_path: Path
    top_k: int
    sparse_top_k: int
    jev_model: str
    llm_judge_model: str
    band_low: float
    band_high: float
    limit: int | None

    @classmethod
    def from_env(cls) -> "EvalConfig":
        load_dotenv()
        limit = os.getenv("EVAL_LIMIT")
        return cls(
            tracking_uri=os.getenv("MLFLOW_TRACKING_URI", "http://127.0.0.1:5000"),
            experiment=os.getenv("MLFLOW_EXPERIMENT", "jev-rag-medquad"),
            testset_path=Path(os.getenv("TESTSET_PATH", "data/testset.jsonl")),
            results_path=Path(os.getenv("RESULTS_PATH", "data/eval_results.jsonl")),
            top_k=int(os.getenv("TOP_K", "5")),
            sparse_top_k=int(os.getenv("SPARSE_TOP_K", "20")),
            jev_model=f"gateway:/{os.getenv('JEV_GATEWAY_ENDPOINT', 'jev-decision-model')}",
            llm_judge_model=f"ollama:/{os.getenv('OLLAMA_JUDGE_MODEL', 'granite4.2:latest')}",
            band_low=float(os.getenv("JEV_BAND_LOW", "0.2")),
            band_high=float(os.getenv("JEV_BAND_HIGH", "0.8")),
            limit=int(limit) if limit else None,
        )


class HybridRag:
    def __init__(self, index: VectorStoreIndex, top_k: int, sparse_top_k: int) -> None:
        self.engine = index.as_query_engine(
            vector_store_query_mode="hybrid",
            similarity_top_k=top_k,
            sparse_top_k=sparse_top_k,
        )

    def answer(self, question: str) -> dict:
        response = self.engine.query(question)
        sources = [
            {"context_id": n.node.metadata.get("context_id"), "score": round(float(n.score or 0.0), 4)}
            for n in response.source_nodes
        ]
        return {
            "answer": str(response).strip(),
            "context_ids": [s["context_id"] for s in sources],
            "sources": sources,
        }


class Judges:
    def __init__(self, jev_model: str, llm_model: str) -> None:
        self.jev_model = jev_model
        self.llm_model = llm_model

    def build(self) -> list:
        return [
            make_judge(name="jev_no_contradiction", model=self.jev_model, instructions=NO_CONTRADICTION_INSTRUCTIONS, feedback_value_type=bool),
            make_judge(name="jev_coverage", model=self.jev_model, instructions=COVERAGE_INSTRUCTIONS, feedback_value_type=bool),
            make_judge(name="llm_no_contradiction", model=self.llm_model, instructions=NO_CONTRADICTION_INSTRUCTIONS, feedback_value_type=bool),
            make_judge(name="llm_coverage", model=self.llm_model, instructions=COVERAGE_INSTRUCTIONS, feedback_value_type=bool),
            self.retrieval_hit,
            self.retrieval_rank,
        ]

    @staticmethod
    @scorer
    def retrieval_hit(outputs: dict, expectations: dict) -> bool:
        return expectations["expected_context_id"] in outputs["context_ids"]

    @staticmethod
    @scorer
    def retrieval_rank(outputs: dict, expectations: dict) -> int:
        ids = outputs["context_ids"]
        cid = expectations["expected_context_id"]
        return ids.index(cid) + 1 if cid in ids else 0


class Testset:
    def __init__(self, path: Path, limit: int | None) -> None:
        self.path = path
        self.limit = limit

    def load(self) -> list[dict]:
        rows = [json.loads(line) for line in self.path.open(encoding="utf-8")]
        return rows[: self.limit] if self.limit else rows

    @staticmethod
    def to_eval_data(rows: list[dict]) -> list[dict]:
        return [
            {
                "inputs": {"question": r["question"]},
                "expectations": {"expected_response": r["answer"], "expected_context_id": r["context_id"]},
            }
            for r in rows
        ]


class Analyzer:
    PAIRS = (("jev_no_contradiction", "llm_no_contradiction"), ("jev_coverage", "llm_coverage"))

    def __init__(self, config: EvalConfig, rows: list[dict]) -> None:
        self.config = config
        self.by_question = {r["question"]: r["id"] for r in rows}

    @staticmethod
    def _question(trace) -> str:
        try:
            return json.loads(trace.data.request)["question"]
        except Exception:
            return str(trace.data.request)

    def _record(self, trace) -> dict:
        rec = {"id": self.by_question.get(self._question(trace)), "trace_id": trace.info.trace_id}
        for a in trace.info.assessments:
            if not hasattr(a, "value") or a.name not in {n for pair in self.PAIRS for n in pair} | {"retrieval_hit", "retrieval_rank"}:
                continue
            rec[a.name] = a.value
            if a.metadata and "typesafe.probability" in a.metadata:
                rec[f"{a.name}_prob"] = float(a.metadata["typesafe.probability"])
            if getattr(a, "rationale", None):
                rec[f"{a.name}_rationale"] = a.rationale
        return rec

    def run(self, run_id: str) -> dict:
        traces = mlflow.search_traces(run_id=run_id, return_type="list", max_results=1000)
        records = [self._record(t) for t in traces]
        metrics: dict[str, float] = {}
        for jev, llm in self.PAIRS:
            scored = [r for r in records if jev in r and llm in r]
            if not scored:
                continue
            agree = sum(r[jev] == r[llm] for r in scored)
            band = [r for r in scored if f"{jev}_prob" in r and self.config.band_low <= r[f"{jev}_prob"] <= self.config.band_high]
            routed_agree = sum((r[llm] if r in band else r[jev]) == r[llm] for r in scored)
            metrics[f"{jev}/agreement_with_llm"] = agree / len(scored)
            metrics[f"{jev}/uncertain_fraction"] = len(band) / len(scored)
            metrics[f"{jev}/agreement_after_routing"] = routed_agree / len(scored)
        self.config.results_path.parent.mkdir(parents=True, exist_ok=True)
        with self.config.results_path.open("w", encoding="utf-8") as fh:
            for r in sorted(records, key=lambda x: x["id"] or ""):
                fh.write(json.dumps(r, ensure_ascii=False) + "\n")
        return metrics


class RagEvaluation:
    def __init__(self, config: EvalConfig) -> None:
        self.config = config

    def run(self) -> None:
        mlflow.set_tracking_uri(self.config.tracking_uri)
        mlflow.set_experiment(self.config.experiment)
        mlflow.llama_index.autolog()

        rows = Testset(self.config.testset_path, self.config.limit).load()
        rag = HybridRag(rag_index, self.config.top_k, self.config.sparse_top_k)

        def predict_fn(question: str) -> dict:
            return rag.answer(question)

        with mlflow.start_run(run_name="hybrid-rag-jev-eval") as run:
            mlflow.log_params({"top_k": self.config.top_k, "sparse_top_k": self.config.sparse_top_k, "n_examples": len(rows)})
            model_info = mlflow.llama_index.log_model("rag_index.py", name="rag_index", engine_type="query", input_example=rows[0]["question"])
            result = mlflow.genai.evaluate(
                data=Testset.to_eval_data(rows),
                predict_fn=predict_fn,
                scorers=Judges(self.config.jev_model, self.config.llm_judge_model).build(),
                model_id=model_info.model_id,
            )
            extra = Analyzer(self.config, rows).run(run.info.run_id)
            mlflow.log_metrics(extra)

        print(json.dumps({**result.metrics, **extra}, indent=2))
        print(f"per-example results -> {self.config.results_path}")


if __name__ == "__main__":
    RagEvaluation(EvalConfig.from_env()).run()

eval_report_generator.py — the HTML report

TEMPLATE. A complete, self-contained page (styles, markup, a small script) with a handful of __PLACEHOLDER__ tokens. Keeping the template inside the script means the report tool is one file with one dependency.

FIELDS. The whitelist of columns pulled from each JSONL row: ids, retrieval metrics, the four verdicts, the two Jev probabilities and the two rationales. Anything else in the row is dropped, so the embedded payload stays small and the page’s expectations are explicit.

ReportConfig.from_args. The input path is the one positional argument; output defaults to the same name with .html. Title, eyebrow and band edges are flags with .env fallbacks, so a bare python eval_report_generator.py after each evaluation regenerates the report with the run’s own settings.

EvalReport.load / intro / render. load reads and trims the rows; intro writes the header paragraph from the data (trace count, scored count, band) instead of hard-coding run facts; render embeds the rows as JSON and fills the placeholders. The only subtlety is escaping </ inside the embedded JSON so a rationale can never close the script tag early.

The page itself. All analytics are computed in the browser from the embedded rows: scored vs unscored counts, Jev pass rates, retrieval rank distribution, agreement matrices, routing-band effects, probability strips and a filterable per-trace ledger with rationales. That means the Python side stays trivial and the same report logic applies to any run size.

from __future__ import annotations

import argparse
import json
import os
from dataclasses import dataclass
from pathlib import Path

from dotenv import load_dotenv

TEMPLATE = r"""<meta charset="utf-8">
<title>__TITLE__</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Sans:wght@400;500;600&family=IBM+Plex+Mono:wght@400;500&display=swap">
<style>
/* Layout: summary tiles → judge-probability strips → agreement matrices → per-trace ledger. Clinical slate + cobalt, monospace for numbers. */
:root{
  --bg:#f3f5f8; --surface:#ffffff; --fg:#17202b; --muted:#5b6878; --line:#d7dde6;
  --accent:#1f5fd6; --accent-soft:#dde8fb;
  --good:#1f8a5b; --good-soft:#dcf3e8; --bad:#c43d3d; --bad-soft:#f9dede; --warn:#b7791f; --warn-soft:#fbeccf; --none:#8a94a3; --none-soft:#e8ebf0;
  --sans:"IBM Plex Sans",system-ui,-apple-system,"Segoe UI",sans-serif;
  --mono:"IBM Plex Mono",ui-monospace,SFMono-Regular,Menlo,monospace;
}
@media (prefers-color-scheme: dark){:root:not([data-theme="light"]){
  --bg:#0f141b; --surface:#171e27; --fg:#e6ebf2; --muted:#98a4b5; --line:#2a3441;
  --accent:#6f9cf5; --accent-soft:#1c2a45;
  --good:#4cc08a; --good-soft:#163526; --bad:#ef7676; --bad-soft:#3f1d1d; --warn:#e0a94a; --warn-soft:#3b2d12; --none:#7d8795; --none-soft:#232b36; color-scheme:dark}}
:root[data-theme="dark"]{
  --bg:#0f141b; --surface:#171e27; --fg:#e6ebf2; --muted:#98a4b5; --line:#2a3441;
  --accent:#6f9cf5; --accent-soft:#1c2a45;
  --good:#4cc08a; --good-soft:#163526; --bad:#ef7676; --bad-soft:#3f1d1d; --warn:#e0a94a; --warn-soft:#3b2d12; --none:#7d8795; --none-soft:#232b36; color-scheme:dark}
*{box-sizing:border-box}
body{background:var(--bg);color:var(--fg);font-family:var(--sans);font-size:15px;line-height:1.5;padding-inline:clamp(16px,4vw,40px);padding-block:28px 56px}
h1,h2,h3{margin:0;text-wrap:balance;font-weight:600}
h1{font-size:clamp(1.5rem,3vw,2rem);letter-spacing:-.01em}
h2{font-size:1.05rem}
.eyebrow{font-family:var(--mono);font-size:.72rem;letter-spacing:.08em;text-transform:uppercase;color:var(--muted)}
.wrap{max-width:1180px;margin:0 auto;display:grid;gap:28px}
header{display:grid;gap:6px}
header p{margin:0;color:var(--muted);max-width:70ch}
.num{font-family:var(--mono);font-variant-numeric:tabular-nums}
.tiles{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:12px}
.tile{background:var(--surface);border:1px solid var(--line);border-radius:10px;padding:14px 16px;display:grid;gap:4px;min-width:0}
.tile .v{font-family:var(--mono);font-size:1.7rem;line-height:1.1;font-variant-numeric:tabular-nums}
.tile .s{font-size:.8rem;color:var(--muted)}
.tile.good .v{color:var(--good)} .tile.bad .v{color:var(--bad)} .tile.warn .v{color:var(--warn)} .tile.accent .v{color:var(--accent)}
.note{border-left:3px solid var(--warn);background:var(--warn-soft);padding:12px 16px;border-radius:0 8px 8px 0;display:grid;gap:4px}
.note p{margin:0}
section{display:grid;gap:12px}
.card{background:var(--surface);border:1px solid var(--line);border-radius:10px;padding:16px;min-width:0}
.two{display:grid;grid-template-columns:repeat(auto-fit,minmax(320px,1fr));gap:16px}
.strip{display:grid;gap:10px}
.strip svg{width:100%;height:auto;display:block}
.legend{display:flex;flex-wrap:wrap;gap:14px;font-size:.8rem;color:var(--muted)}
.legend span::before{content:"";display:inline-block;width:10px;height:10px;border-radius:50%;margin-right:6px;vertical-align:-1px;background:var(--c)}
.cm{display:grid;grid-template-columns:auto 1fr 1fr;gap:6px;align-items:stretch;font-size:.85rem}
.cm .h{color:var(--muted);font-size:.75rem;text-align:center;align-self:end;padding-bottom:4px}
.cm .r{color:var(--muted);font-size:.75rem;display:flex;align-items:center;justify-content:flex-end;padding-right:6px}
.cm .c{border-radius:8px;padding:14px 10px;text-align:center;font-family:var(--mono);font-size:1.4rem;background:var(--none-soft)}
.cm .c small{display:block;font-family:var(--sans);font-size:.72rem;color:var(--muted)}
.cm .c.agree{background:var(--good-soft);color:var(--good)} .cm .c.dis{background:var(--bad-soft);color:var(--bad)}
.bars{display:grid;gap:8px;font-size:.85rem}
.bar{display:grid;grid-template-columns:110px 1fr 44px;align-items:center;gap:10px}
.bar .t{height:14px;border-radius:4px;background:var(--accent);min-width:2px}
.bar .t.miss{background:var(--bad)} .bar .t.none{background:var(--none)}
.bar .lab{color:var(--muted)}
.toolbar{display:flex;flex-wrap:wrap;gap:8px;align-items:center}
.toolbar button{font:inherit;font-size:.82rem;padding:6px 12px;border-radius:999px;border:1px solid var(--line);background:var(--surface);color:var(--fg);cursor:pointer}
.toolbar button[aria-pressed="true"]{background:var(--accent);border-color:var(--accent);color:#fff}
.toolbar button:focus-visible{outline:2px solid var(--accent);outline-offset:2px}
.tablewrap{overflow-x:auto;border:1px solid var(--line);border-radius:10px;background:var(--surface)}
table{border-collapse:collapse;width:100%;font-size:.85rem;min-width:720px}
th,td{padding:9px 12px;text-align:left;border-bottom:1px solid var(--line);vertical-align:top}
th{font-size:.72rem;letter-spacing:.06em;text-transform:uppercase;color:var(--muted);font-weight:500;position:sticky;top:0;background:var(--surface)}
tr:last-child td{border-bottom:0}
.pill{display:inline-flex;align-items:center;gap:6px;padding:2px 9px;border-radius:999px;font-size:.75rem;font-weight:500;white-space:nowrap}
.pill.t{background:var(--good-soft);color:var(--good)} .pill.f{background:var(--bad-soft);color:var(--bad)} .pill.n{background:var(--none-soft);color:var(--none)}
.pill .p{font-family:var(--mono);font-weight:400;opacity:.85}
.pill.band{box-shadow:inset 0 0 0 1.5px var(--warn)}
td.tid{font-family:var(--mono);font-size:.78rem;color:var(--muted)}
tr.dis td{background:color-mix(in srgb,var(--bad-soft) 35%,transparent)}
details{margin-top:6px}
details summary{cursor:pointer;color:var(--accent);font-size:.78rem}
details div{margin-top:6px;display:grid;gap:6px;font-size:.8rem;color:var(--muted);max-width:70ch}
details b{color:var(--fg);font-weight:500}
.foot{color:var(--muted);font-size:.8rem}
@media (prefers-reduced-motion:no-preference){.tile,.card{transition:border-color .15s}}
</style>

<div class="wrap">
<header>
  <div class="eyebrow">__EYEBROW__</div>
  <h1>__TITLE__</h1>
  <p>__INTRO__</p>
</header>

<div class="tiles" id="tiles"></div>

<div class="note" id="dq"></div>

<section>
  <h2>Jev decision probabilities</h2>
  <div class="card strip">
    <div class="legend"><span style="--c:var(--good)">Jev and text judge agree</span><span style="--c:var(--bad)">disagree</span><span style="--c:var(--none)">text judge missing</span><span style="--c:var(--warn)">__BAND_LOW__-__BAND_HIGH__ routing band</span></div>
    <div id="strips"></div>
  </div>
</section>

<section>
  <h2>Agreement with the text judge</h2>
  <div class="two" id="cms"></div>
</section>

<section>
  <h2>Retrieval</h2>
  <div class="two">
    <div class="card"><div class="eyebrow" style="margin-bottom:10px">Rank of the gold context among top-5</div><div class="bars" id="ranks"></div></div>
    <div class="card"><div class="eyebrow" style="margin-bottom:10px">Judge pass rates (scored traces only)</div><div class="bars" id="passrates"></div></div>
  </div>
</section>

<section>
  <h2>Per-trace ledger</h2>
  <div class="toolbar" id="filters"></div>
  <div class="tablewrap"><table>
    <thead><tr><th>Trace</th><th>Gold rank</th><th>No contradiction · Jev</th><th>No contradiction · text</th><th>Coverage · Jev</th><th>Coverage · text</th></tr></thead>
    <tbody id="rows"></tbody>
  </table></div>
  <p class="foot">Rationales come from the text judge; Jev emits only the probability. Highlighted rows are Jev–text disagreements.</p>
</section>
</div>

<script id="data" type="application/json">__DATA__</script>
<script>
const rows = JSON.parse(document.getElementById('data').textContent);
const BAND=[__BAND_LOW__,__BAND_HIGH__];
const scored = rows.filter(r=>r.jev_no_contradiction!==null);
const cnt=(arr,f)=>arr.filter(f).length;
const pct=(a,b)=>b?Math.round(100*a/b)+'%':'–';
const pairs=[["jev_no_contradiction","llm_no_contradiction","No contradiction"],["jev_coverage","llm_coverage","Coverage"]];

function agreement(j,l){
  const both=rows.filter(r=>r[j]!==null&&r[l]!==null);
  const agree=cnt(both,r=>r[j]===r[l]);
  const band=both.filter(r=>r[j+'_prob']>=BAND[0]&&r[j+'_prob']<=BAND[1]);
  const routed=cnt(both,r=>(band.includes(r)?r[l]:r[j])===r[l]);
  return {both,agree,band,routed,tt:cnt(both,r=>r[j]&&r[l]),tf:cnt(both,r=>r[j]&&!r[l]),ft:cnt(both,r=>!r[j]&&r[l]),ff:cnt(both,r=>!r[j]&&!r[l])};
}
const A={}; pairs.forEach(([j,l])=>A[j]=agreement(j,l));

const tiles=[
  ['Scored traces',`${scored.length}<span style="font-size:1rem;color:var(--muted)">/${rows.length}</span>`,'with any judge output','warn'],
  ['Jev · no contradiction',pct(cnt(scored,r=>r.jev_no_contradiction),scored.length),`${cnt(scored,r=>r.jev_no_contradiction)} of ${scored.length} pass`,'good'],
  ['Jev · coverage',pct(cnt(scored,r=>r.jev_coverage),scored.length),`${cnt(scored,r=>r.jev_coverage)} of ${scored.length} pass`,'accent'],
  ['Gold context retrieved',pct(cnt(scored,r=>r.retrieval_hit),scored.length),`${cnt(scored,r=>r.retrieval_rank===1)} at rank 1`,'good'],
  ['Agreement · contradiction',pct(A.jev_no_contradiction.agree,A.jev_no_contradiction.both.length),`${A.jev_no_contradiction.agree}/${A.jev_no_contradiction.both.length} · ${A.jev_no_contradiction.routed}/${A.jev_no_contradiction.both.length} after routing`,'accent'],
  ['Agreement · coverage',pct(A.jev_coverage.agree,A.jev_coverage.both.length),`${A.jev_coverage.agree}/${A.jev_coverage.both.length} · ${A.jev_coverage.routed}/${A.jev_coverage.both.length} after routing`,'accent'],
];
document.getElementById('tiles').innerHTML=tiles.map(([k,v,s,c])=>`<div class="tile ${c}"><div class="eyebrow">${k}</div><div class="v">${v}</div><div class="s">${s}</div></div>`).join('');

const unscored=rows.length-scored.length, noLLM=cnt(scored,r=>r.llm_no_contradiction===null||r.llm_coverage===null);
document.getElementById('dq').innerHTML=`<div class="eyebrow" style="color:var(--warn)">Data quality</div>
<p><b>${unscored} of ${rows.length} traces carry no scores at all</b> (no judge or retrieval value), so every rate on this page is computed over the ${scored.length} scored traces, not 50. On ${noLLM} scored traces the gemma3 text judge returned nothing for at least one criterion, which shrinks the agreement sample to ${A.jev_no_contradiction.both.length} and ${A.jev_coverage.both.length}. Every record also has a null <span class="num">id</span>: the question-to-id mapping in the analyzer did not resolve, so rows are keyed by trace id.</p>`;

function strip(j,l,label){
  const W=900,H=96,L=70,R=30,y=52;
  const x=p=>L+(W-L-R)*p;
  const pts=rows.filter(r=>r[j+'_prob']!==undefined&&r[j+'_prob']!==null).map(r=>({p:r[j+'_prob'],c:r[l]===null?'var(--none)':(r[j]===r[l]?'var(--good)':'var(--bad)'),r}));
  const dots=pts.map(d=>`<circle cx="${x(d.p).toFixed(1)}" cy="${y}" r="7" fill="${d.c}" fill-opacity=".75" stroke="var(--surface)" stroke-width="1.5"><title>${d.r.trace_id.slice(3,11)} · p=${d.p}</title></circle>`).join('');
  const ticks=[0,.25,.5,.75,1].map(t=>`<line x1="${x(t)}" x2="${x(t)}" y1="${y-16}" y2="${y+16}" stroke="var(--line)"/><text x="${x(t)}" y="${y+34}" text-anchor="middle" font-size="11" fill="var(--muted)" font-family="var(--mono)">${t}</text>`).join('');
  return `<svg viewBox="0 0 ${W} ${H}" role="img" aria-label="${label} probabilities">
   <rect x="${x(BAND[0])}" y="${y-22}" width="${x(BAND[1])-x(BAND[0])}" height="44" fill="var(--warn)" fill-opacity=".14" rx="4"/>
   <line x1="${x(0)}" x2="${x(1)}" y1="${y}" y2="${y}" stroke="var(--line)" stroke-width="2"/>
   ${ticks}
   <line x1="${x(.5)}" x2="${x(.5)}" y1="${y-24}" y2="${y+24}" stroke="var(--fg)" stroke-dasharray="3 3"/>
   <text x="8" y="${y+4}" font-size="12" fill="var(--fg)" font-weight="600">${label}</text>
   <text x="8" y="${y+18}" font-size="10" fill="var(--muted)">n=${pts.length}</text>
   ${dots}</svg>`;
}
document.getElementById('strips').innerHTML=pairs.map(([j,l,n])=>strip(j,l,n)).join('');

document.getElementById('cms').innerHTML=pairs.map(([j,l,n])=>{const a=A[j];return `<div class="card"><div class="eyebrow" style="margin-bottom:10px">${n} · ${a.both.length} traces with both verdicts · ${a.band.length} in routing band</div>
<div class="cm"><div></div><div class="h">text judge: pass</div><div class="h">text judge: fail</div>
<div class="r">Jev: pass</div><div class="c agree">${a.tt}<small>both pass</small></div><div class="c dis">${a.tf}<small>Jev accepts, text rejects</small></div>
<div class="r">Jev: fail</div><div class="c dis">${a.ft}<small>Jev rejects, text accepts</small></div><div class="c agree">${a.ff}<small>both fail</small></div></div></div>`}).join('');

const rk=[['Rank 1',cnt(rows,r=>r.retrieval_rank===1),''],['Rank 2',cnt(rows,r=>r.retrieval_rank===2),''],['Rank 3',cnt(rows,r=>r.retrieval_rank===3),''],['Missed',cnt(rows,r=>r.retrieval_rank===0),'miss'],['Unscored',cnt(rows,r=>r.retrieval_rank===null),'none']];
const mx=Math.max(...rk.map(x=>x[1]));
document.getElementById('ranks').innerHTML=rk.map(([k,v,c])=>`<div class="bar"><span class="lab">${k}</span><div class="t ${c}" style="width:${100*v/mx}%"></div><span class="num">${v}</span></div>`).join('');
const pr=[['Jev · no contr.',cnt(scored,r=>r.jev_no_contradiction),scored.length],['Text · no contr.',cnt(rows,r=>r.llm_no_contradiction===true),cnt(rows,r=>r.llm_no_contradiction!==null)],['Jev · coverage',cnt(scored,r=>r.jev_coverage),scored.length],['Text · coverage',cnt(rows,r=>r.llm_coverage===true),cnt(rows,r=>r.llm_coverage!==null)]];
document.getElementById('passrates').innerHTML=pr.map(([k,v,n])=>`<div class="bar"><span class="lab">${k}</span><div class="t" style="width:${n?100*v/n:0}%"></div><span class="num">${v}/${n}</span></div>`).join('');

const pill=(v,p)=>{if(v===null)return `<span class="pill n">none</span>`;const band=p!==undefined&&p!==null&&p>=BAND[0]&&p<=BAND[1];return `<span class="pill ${v?'t':'f'}${band?' band':''}">${v?'pass':'fail'}${p!==undefined&&p!==null?`<span class="p">${p.toFixed(2)}</span>`:''}</span>`};
const isDis=r=>pairs.some(([j,l])=>r[j]!==null&&r[l]!==null&&r[j]!==r[l]);
const views={all:r=>true,scored:r=>r.jev_no_contradiction!==null,disagree:isDis,band:r=>pairs.some(([j])=>r[j+'_prob']>=BAND[0]&&r[j+'_prob']<=BAND[1]),unscored:r=>r.jev_no_contradiction===null};
const labels={all:'All',scored:'Scored',disagree:'Disagreements',band:'In routing band',unscored:'Unscored'};
let view='scored';
function render(){
  document.getElementById('filters').innerHTML=Object.keys(views).map(k=>`<button id="f-${k}" aria-pressed="${k===view}" data-v="${k}">${labels[k]} <span class="num">${cnt(rows,views[k])}</span></button>`).join('');
  document.querySelectorAll('#filters button').forEach(b=>b.onclick=()=>{view=b.dataset.v;render();});
  const list=rows.filter(views[view]).sort((a,b)=>(a.jev_no_contradiction===null)-(b.jev_no_contradiction===null)||(a.jev_no_contradiction_prob??2)-(b.jev_no_contradiction_prob??2));
  document.getElementById('rows').innerHTML=list.map(r=>{
    const rk=r.retrieval_rank===null?'<span class="pill n">none</span>':r.retrieval_rank===0?'<span class="pill f">miss</span>':`<span class="num">${r.retrieval_rank}</span>`;
    const rat=(k,t)=>r[k]?`<details><summary>rationale</summary><div><b>${t}</b>${r[k]}</div></details>`:'';
    return `<tr class="${isDis(r)?'dis':''}"><td class="tid">${r.trace_id.slice(3,13)}</td><td>${rk}</td>
      <td>${pill(r.jev_no_contradiction,r.jev_no_contradiction_prob)}</td><td>${pill(r.llm_no_contradiction)}${rat('llm_no_contradiction_rationale','Text judge · no contradiction')}</td>
      <td>${pill(r.jev_coverage,r.jev_coverage_prob)}</td><td>${pill(r.llm_coverage)}${rat('llm_coverage_rationale','Text judge · coverage')}</td></tr>`}).join('')||`<tr><td colspan="6" style="color:var(--muted)">No traces match this view.</td></tr>`;
}
render();
</script>
"""

FIELDS = (
    "id", "trace_id", "retrieval_hit", "retrieval_rank",
    "jev_no_contradiction", "jev_no_contradiction_prob", "llm_no_contradiction", "llm_no_contradiction_rationale",
    "jev_coverage", "jev_coverage_prob", "llm_coverage", "llm_coverage_rationale",
)


@dataclass(frozen=True)
class ReportConfig:
    input_path: Path
    output_path: Path
    title: str
    eyebrow: str
    band_low: float
    band_high: float

    @classmethod
    def from_args(cls) -> "ReportConfig":
        load_dotenv()
        p = argparse.ArgumentParser(description="Build an HTML evaluation report from an eval_results JSONL file.")
        p.add_argument("input", nargs="?", default=os.getenv("RESULTS_PATH", "data/eval_results.jsonl"))
        p.add_argument("-o", "--output")
        p.add_argument("--title", default=os.getenv("REPORT_TITLE", "Jev Judge Evaluation"))
        p.add_argument("--eyebrow", default=os.getenv("REPORT_EYEBROW", f"MLflow · {os.getenv('MLFLOW_EXPERIMENT', 'experiment')} · {os.getenv('OLLAMA_LLM_MODEL', 'LLM')} over Qdrant hybrid retrieval"))
        p.add_argument("--band-low", type=float, default=float(os.getenv("JEV_BAND_LOW", "0.2")))
        p.add_argument("--band-high", type=float, default=float(os.getenv("JEV_BAND_HIGH", "0.8")))
        a = p.parse_args()
        inp = Path(a.input)
        return cls(
            input_path=inp,
            output_path=Path(a.output) if a.output else inp.with_suffix(".html"),
            title=a.title,
            eyebrow=a.eyebrow,
            band_low=a.band_low,
            band_high=a.band_high,
        )


class EvalReport:
    def __init__(self, config: ReportConfig) -> None:
        self.config = config

    def load(self) -> list[dict]:
        rows = [json.loads(line) for line in self.config.input_path.open(encoding="utf-8") if line.strip()]
        return [{k: r.get(k) for k in FIELDS} for r in rows]

    def intro(self, rows: list[dict]) -> str:
        scored = sum(1 for r in rows if r["jev_no_contradiction"] is not None)
        return (
            f"{len(rows)} traces in {self.config.input_path.name}, {scored} with judge output. "
            f"Each answer is scored by two Jev decision judges (no-contradiction, coverage) and two text judges on the same criteria. "
            f"Jev returns a probability; the pass threshold is 0.5 and the {self.config.band_low}-{self.config.band_high} band marks verdicts that would be routed for a second opinion."
        )

    def render(self, rows: list[dict]) -> str:
        return (
            TEMPLATE.replace("__DATA__", json.dumps(rows, ensure_ascii=False, separators=(",", ":")).replace("</", "<\\/"))
            .replace("__TITLE__", self.config.title)
            .replace("__EYEBROW__", self.config.eyebrow)
            .replace("__INTRO__", self.intro(rows))
            .replace("__BAND_LOW__", str(self.config.band_low))
            .replace("__BAND_HIGH__", str(self.config.band_high))
        )

    def run(self) -> Path:
        rows = self.load()
        self.config.output_path.parent.mkdir(parents=True, exist_ok=True)
        self.config.output_path.write_text(self.render(rows), encoding="utf-8")
        print(f"{len(rows)} rows -> {self.config.output_path}")
        return self.config.output_path


if __name__ == "__main__":
    EvalReport(ReportConfig.from_args()).run()

The Execution and Results:

Detect Issues: let the traces review themselves

Large number of traces is already too many to read one by one. few hundreds is impossible. Detect Issues flips the job around: instead of you hunting for failures, an LLM reads every trace in the experiment, spots the ones that went wrong, and clusters them into named issues with a severity, a description, a count of affected traces, and tags for the failure category (latency, correctness, adherence, relevance).

You pick the reviewing model from any provider or AI Gateway endpoint (here, gemma via the gateway), select the traces, see the estimated cost up front, and run. The result lands as an evaluation run with an Issues tab you triage like a bug tracker: accept, reject, or resolve each one.

On this RAG experiment it surfaced two issues without any judge or label being involved: Execution timeout errors across 13 traces, and Incorrect medical information and timeouts on 4. The first is exactly the 22 unscored rows the eval report flagged; the second overlaps with the answers both Jev and the text judge rejected. Two different routes to the same conclusion, one of them hands-off.

The point is not that it replaces judges. Judges score what you asked about; Detect Issues tells you what you forgot to ask.

MLflow’s experiment Overview has three tabs. Usage answers “did it run?”, Quality answers “was it any good?”, and Tool calls (empty here, since this RAG makes no tool calls) covers agents.

Usage tab. Three panels, all from the autologged traces. Traces shows volume over time: one bar of 50 on 11/10, the single evaluation run. Latency plots p50/p90/p99 per day; the average of 239 s per question is the cost of running a 12B generator plus two Ollama judges on a laptop, and the three percentiles lying on top of each other means every question was uniformly slow rather than a few outliers. Errors is the number to act on: 22 traces failed, a 44 % error rate. That is the same 22 rows that arrived with no scores in the eval report, now visible as a platform-level signal rather than something you infer from nulls.

Quality tab, summary table. One row per scorer attached to the run: the two Jev judges, the two text judges, and the two retrieval code scorers, each with 50 recorded assessments. Average value is blank because the verdicts are booleans and ranks, not continuous scores.

Quality tab, per-scorer charts. Each scorer gets a histogram of its values, and the null bar is the honest part of every chart: it is the 22 errored traces showing up again. Read past it and the story is consistent:

• jev_no_contradiction — 25 true, 3 false: Jev rarely finds a claim that conflicts with the reference.
• jev_coverage — 21 true, 7 false: the stricter criterion, as expected; concise answers miss key points more often than they contradict them.
• llm_no_contradiction and llm_coverage — the gemma3 text judge has a taller null bar (31–32) than Jev (22) because it also timed out on some traces that generated fine. Where it did answer, its pass/fail split mirrors Jev’s.
• retrieval_hit — 27 true, 1 false: the gold context was in the top-5 almost every time.
• retrieval_rank — 22 at rank 1, 4 at rank 2, 1 at rank 3, 1 miss: hybrid retrieval put the right document first in about four out of five scored questions.

The takeaway for the blog: before any judge comparison, the Overview already tells you that retrieval is not the problem, generation latency is, and nearly half the run never reached the judges at all.

Jev decision probabilities. Each dot is one answer, placed at the probability Jev assigned; the dashed line at 0.5 is the pass threshold and the shaded band (0.2–0.8) marks verdicts we’d escalate. Green means Jev and the text judge agreed, red means they disagreed, grey means the text judge never returned. No contradiction is clustered hard against 1.0: Jev is confident most answers don’t conflict with the reference, and only 5 of 28 fall in the band. Coverage is spread across the whole axis with 12 of 28 in the band: a much less certain question, which makes sense since “did it cover the key points” is a judgment of degree, not a yes/no fact. The two red dots at 0.76 and 0.85 are the interesting ones: answers Jev accepted that the text judge rejected, one of them outside the band.

Agreement with the text judge. Confusion matrices on the traces where both judges answered. No contradiction: 17 of 19 agree, and both disagreements are Jev-accepts / text-rejects. Jev never rejected something the text judge passed. Coverage: 16 of 18 agree, one miss each way. Routing the banded cases to the text judge lifts both to 18/19 and 18/18.

Retrieval. Of 28 scored questions the gold context was at rank 1 in 22, rank 2 in 4, rank 3 in 1, missed in 1. Retrieval is not where quality is lost. The grey “unscored 22” bar is the errored traces.

Judge pass rates. Jev passes 25/28 on contradiction and 21/28 on coverage; the text judge passes 15/19 and 14/18, almost the same rates on its smaller sample. Jev is not systematically more lenient, just faster and present on every trace.

The Conclusion

This article asked one question: can a System-One decision model like Jev stand in for a text-generating LLM as a RAG judge? We built a hybrid LlamaIndex + Qdrant + Ollama pipeline over MedQuAD, wired Jev in through the MLflow AI Gateway with make_judge, and ran it side by side with a text judge on the same criteria.

The answer, on this run, is a qualified yes. On the 28 traces that completed, Jev agreed with the text judge on 17/19 contradiction verdicts and 16/18 coverage verdicts, and routing its uncertain band to the text judge closed almost all of the gap. Jev answered every trace; the text judge timed out on a third of them. Retrieval was not the weak link: the gold context ranked first in 22 of 28 questions.

Two caveats matter. Jev’s misses were confident acceptances of answers that were nearly right, and one sat above the routing band, so a fixed threshold is not a safety net. And 22 of 50 traces never reached the judges at all, a generation-latency problem that MLflow’s error rate and Detect Issues surfaced before any judge did.

The practical pattern: Jev for cheap, fast, always-on scoring; a text judge for rationale on the cases Jev is unsure about; MLflow to keep the whole thing observable.

Evaluating Medical RAG with Qdrant, Jev and MLflow — A Production Ready Setup was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Architecture:

The pipeline has one shared foundation and three phases built on top of it.

The foundation: one MLflow server. Every model the pipeline uses is served from a single MLflow server. Three models are registered in the model registry: Qwen3-Embedding-4B for embeddings, Qwen3-Reranker-0.6B for reranking, and Laya for claim verification. The fourth component, the LLM that writes the answer, sits behind the MLflow AI Gateway, which exposes a locally hosted Ollama model through an OpenAI-compatible endpoint. The pipeline code never loads a model from a path. It asks for a model by name, so swapping a model is a registry change, not a code change.

Phase 1: Ingest (offline). The source is a 431-page state treatment guidelines manual. We parse the PDF to markdown, split it into chunks of 512 tokens with 100 tokens of overlap, and embed each chunk twice. The dense vector comes from Qwen3-Embedding-4B (2,560 dimensions), and the sparse vector comes from BM25. Both are written to Qdrant in batches of 16, as two named vectors on the same point, with stable IDs so a re-run never creates duplicates. Keeping both vectors on one point is what makes hybrid search possible later.

Phase 2: Retrieve and generate (online). A question is embedded with the same Qwen model, this time in query mode with a task instruction. Qdrant runs the dense and sparse searches and fuses the two rankings with reciprocal rank fusion, returning 30 candidates. The reranker then reads the question and each candidate together and scores how well they match, and the top five are kept. Those five are numbered and handed to the LLM with one instruction: write one short statement per line, and end each line with the number of the passage it came from.

Phase 3: Verdict gating. This is where the pipeline differs from a standard RAG setup. The answer is split into claims, one per line, and each claim is paired with the chunk it cites. Laya reads the pair and labels the claim supported, unsupported or contradicted. The gate then acts on those labels. If every claim is supported, the answer is released. If some are not, they are dropped and the rest are released. If nothing is supported, the pipeline abstains instead of answering.

The Core Algorithm:

The algorithm has four parts: turn the answer into claims, attach evidence to each claim, get a verdict, and decide what to release.

1. Claim extraction. The generator is told to write one short statement per line and to end each line with a single citation such as [2]. We treat each line as one claim. We strip any bullet or list marker, read the citation numbers, remove duplicates and delete the markers from the text, so the verifier sees only the statement. A line saying the guidelines do not cover the question is not a claim, so it is skipped. If no claims are left, there is nothing to verify and the gate abstains.

2. Evidence selection. The evidence for a claim is the full text of the chunk it cites. If a claim cites two chunks, both are joined. If a citation points to a chunk that does not exist, it is ignored. If a claim has no valid citation, it is checked against all five retrieved chunks and marked as uncited.

3. The verdict. Laya receives the evidence and the claim together and one question: is the claim supported by the evidence? It must choose one of three options, each with a short definition: supported (the evidence states the claim), unsupported (the evidence does not mention it), or contradicted (the evidence says the opposite). This is one forward pass per claim, and Laya returns one label.

4. The gate. The gate is a small policy over the labels:

• Release: every claim is supported, so the answer goes out as written.
• Revise: some claims are not supported. Those claims are dropped, and the supported ones are released with their citations.
• Abstain: no claim is supported, or there was nothing to check. The system says it could not verify an answer from the guidelines.

A worked example. For the question “How is benign paroxysmal positional vertigo treated?”, the model wrote eight claims. Seven were checked against the chunks they cited and came back supported. One, a sentence about repositioning maneuvers, cited a different chunk from the one that contains it, and Laya marked it unsupported. The gate dropped that line and released the other seven, each with its citation.

What this check tests. The verdict says whether a claim is backed by the passage it cites. It does not say whether the passage is relevant to the question or appropriate for a given patient. Those are separate checks, and a clinician stays responsible for the final decision. The released answer comes with its per-claim verdicts and sources so that the decision can be audited.

The Implementation

The scafolding looks as below

.
├── README.md
├── data
│   ├── standard-treatment-guidelines.md
│   └── standard-treatment-guidelines.pdf
├── data_ingestion
│   ├── __init__.py
│   └── ingest.py
├── data_retriever
│   ├── __init__.py
│   ├── claim_check.py
│   ├── retriever_and_generate.py
│   └── verdict_gate.py
├── model_registry
│   ├── __init__.py
│   ├── laya_model.py
│   ├── qwen_embedding_model.py
│   ├── qwen_reranking_model.py
│   ├── register_embedding_model.py
│   ├── register_laya.py
│   ├── register_reranking_model.py
│   └── smoke_test_all_models.py
├── pyproject.toml
└── uv.lock

The dependency toml for the project looks as below

[project]
name = "rag-with-decisive-model"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "mlflow>=3.16.1",
    "mlflow-skinny>=3.16.1",
    "sentence-transformers>=6.1.0",
    "laya>=0.3.23",
    "psycopg>=3.3.6",
    "psycopg-binary>=3.3.6",
    "liteparse>=2.15.1",
    "llama-index>=0.14.25",
    "llama-index-vector-stores-qdrant>=0.10.3",
    "llama-index-embeddings-ollama>=0.10.0",
    "qdrant-client>=1.19.1",
    "fastembed>=0.8.1",
]

Let us start with our core foundation which is Mlflow where all the models like embedding, rerankers, decision models and the LLM (AI Gateway) are first configured. The following scripts are written for respective model to get registered with Mlflow and finally all are tested using one single smoke test script.

import os

os.environ["USE_TF"] = "0"

import laya
import mlflow.pyfunc


class LayaModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        self.agent = laya.load(context.artifacts["checkpoint"])

    def predict(self, context, model_input, params=None):
        state = model_input["state"]
        questions = model_input["questions"]
        max_len = model_input.get("max_len", 8192)
        return self.agent.predict(state, questions, max_len=max_len)

import mlflow.pyfunc
import torch
from sentence_transformers import SentenceTransformer

DEFAULT_INSTRUCTION = "Given a clinical question, retrieve relevant passages from standard treatment guidelines"


def pick_device():
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"


class QwenEmbeddingModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        self.model = SentenceTransformer(
            context.artifacts["checkpoint"],
            device=pick_device(),
            model_kwargs={"torch_dtype": torch.float16},
        )

    def predict(self, context, model_input, params=None):
        texts = model_input["texts"]
        mode = model_input.get("mode", "document")
        instruction = model_input.get("instruction", DEFAULT_INSTRUCTION)
        prompt = f"Instruct: {instruction}\nQuery: " if mode == "query" else None
        embeddings = self.model.encode(
            texts,
            prompt=prompt,
            normalize_embeddings=True,
            batch_size=model_input.get("batch_size", 8),
            truncate_dim=model_input.get("dim"),
        )
        return embeddings.tolist()

import mlflow.pyfunc
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

DEFAULT_INSTRUCTION = "Given a clinical question, retrieve relevant passages from standard treatment guidelines"

PREFIX = (
    "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the "
    'Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
)
SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"


def pick_device():
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"


class QwenRerankerModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        path = context.artifacts["checkpoint"]
        self.device = pick_device()
        self.tokenizer = AutoTokenizer.from_pretrained(path, padding_side="left")
        self.model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.float16).to(self.device).eval()
        self.yes_id = self.tokenizer.convert_tokens_to_ids("yes")
        self.no_id = self.tokenizer.convert_tokens_to_ids("no")

    def predict(self, context, model_input, params=None):
        query = model_input["query"]
        documents = model_input["documents"]
        instruction = model_input.get("instruction", DEFAULT_INSTRUCTION)
        batch_size = model_input.get("batch_size", 4)

        scores = []
        for i in range(0, len(documents), batch_size):
            texts = [
                f"{PREFIX}<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}{SUFFIX}"
                for doc in documents[i : i + batch_size]
            ]
            inputs = self.tokenizer(texts, padding=True, return_tensors="pt").to(self.device)
            with torch.no_grad():
                logits = self.model(**inputs).logits[:, -1, :]
            pair = torch.stack([logits[:, self.no_id], logits[:, self.yes_id]], dim=1).float()
            scores += torch.softmax(pair, dim=1)[:, 1].tolist()
        return scores

# register decision model: convaiinnovations/laya
import mlflow
from huggingface_hub import snapshot_download

from laya_model import LayaModel

mlflow.set_tracking_uri("http://localhost:5001")

local_dir = snapshot_download("convaiinnovations/laya", allow_patterns="multilingual/*", local_dir="laya_checkpoint")

with mlflow.start_run(run_name="laya-multilingual"):
    mlflow.pyfunc.log_model(
        name="laya-multilingual",
        python_model=LayaModel(),
        artifacts={"checkpoint": f"{local_dir}/multilingual"},
        code_paths=["laya_model.py"],
        pip_requirements=["laya", "mlflow"],
        registered_model_name="laya-multilingual",
    )

# register reranking model: Qwen/Qwen3-Reranker-0.6B
import mlflow
from huggingface_hub import snapshot_download

from qwen_reranking_model import QwenRerankerModel

mlflow.set_tracking_uri("http://localhost:5001")

local_dir = snapshot_download("Qwen/Qwen3-Reranker-0.6B", local_dir="qwen3_reranker_0.6b")

with mlflow.start_run(run_name="qwen3-reranker-0.6b"):
    mlflow.pyfunc.log_model(
        name="qwen3-reranker-0-6b",
        python_model=QwenRerankerModel(),
        artifacts={"checkpoint": local_dir},
        code_paths=["qwen_reranking_model.py"],
        pip_requirements=["mlflow", "torch", "transformers>=4.51.0"],
        registered_model_name="qwen3-reranker-0-6b",
    )

# register embedding model: Qwen/Qwen3-Embedding-4B
import mlflow
from huggingface_hub import snapshot_download

from qwen_embedding_model import QwenEmbeddingModel

mlflow.set_tracking_uri("http://localhost:5001")

local_dir = snapshot_download("Qwen/Qwen3-Embedding-4B", local_dir="qwen3_embedding_4b")

with mlflow.start_run(run_name="qwen3-embedding-4b"):
    mlflow.pyfunc.log_model(
        name="qwen3-embedding-4b",
        python_model=QwenEmbeddingModel(),
        artifacts={"checkpoint": local_dir},
        code_paths=["qwen_embedding_model.py"],
        pip_requirements=["mlflow", "torch", "transformers>=4.51.0", "sentence-transformers>=3.0.0"],
        registered_model_name="qwen3-embedding-4b",
    )

Phase1: Extract & Ingest

Phase 1 runs once, before any question is asked. It turns the PDF into a searchable Qdrant collection in two steps.

Extract. The guidelines manual is a 431-page PDF. A parser called LiteParse from LlamaIndex reads it and writes out a markdown file, keeping headings, tables and links. This step is run only the first time or when the source document changes, so the script keeps it as a separate function that you switch on when you need it.

Ingest. The ingest step reads the markdown file and works in four stages:

1. Chunk. The text is split into pieces of about 512 tokens, with 100 tokens of overlap between neighbouring pieces. The overlap keeps a sentence that falls on a boundary from losing its context.
2. Create the collection. A Qdrant collection is created with two named vectors per point: a dense vector of 2,560 dimensions compared by cosine similarity, and a sparse vector for BM25 keyword matching. The sparse vector uses Qdrant’s IDF modifier, so rare words count for more. If the collection already exists, it is deleted and recreated, so every run starts clean.
3. Embed. Chunks are processed 16 at a time. The dense vector comes from the Qwen3-Embedding-4B model in the MLflow registry, called in document mode. The sparse vector comes from BM25, running locally.
4. Store. Each chunk becomes one Qdrant point holding both vectors and the chunk text. The point ID is derived from the chunk’s own ID, so the same chunk always gets the same ID and a re-run cannot create duplicates.

Storing the text on the point means the later phases never have to go back to the PDF. Whatever search returns can go straight to the reranker, the LLM and the claim checker. Keeping the dense and sparse vectors on the same point is what lets Phase 2 run both searches in one query and fuse the results.

# ingest.py
import uuid
import logging
import mlflow
from fastembed import SparseTextEmbedding
from liteparse import LiteParse
from llama_index.core import SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.vector_stores.utils import node_to_metadata_dict
from qdrant_client import QdrantClient, models

logger = logging.getLogger(__name__)

COLLECTION = "standard-treatment-guidelines"
DENSE_SIZE = 2560
BATCH_SIZE = 16

client = QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y")


def extract(src_path: str, dst_path: str) -> None:
    logger.info(f"extracting: {src_path} to {dst_path}")
    parser = LiteParse(
        output_format="markdown",
        extract_images=True,
        image_mode="embed",
        extract_links=True,
        extract_text_metadata=True,
    )
    result = parser.parse(src_path)
    with open(dst_path, "w", encoding="utf-8") as f:
        f.write(result.text)
    logger.info(f"extracted: {dst_path}")


def create_collection() -> None:
    logger.info(f"creating collection: {COLLECTION}")
    if client.collection_exists(COLLECTION):
        client.delete_collection(COLLECTION)
    client.create_collection(
        COLLECTION,
        vectors_config={"text-dense": models.VectorParams(size=DENSE_SIZE, distance=models.Distance.COSINE)},
        sparse_vectors_config={"text-sparse": models.SparseVectorParams(modifier=models.Modifier.IDF)},
    )


def ingest() -> None:
    logger.info(f"ingesting: {COLLECTION}")
    mlflow.set_tracking_uri("http://localhost:5001")
    embedder = mlflow.pyfunc.load_model("models:/qwen3-embedding-4b/latest")
    sparse_model = SparseTextEmbedding("Qdrant/bm25")

    documents = SimpleDirectoryReader(input_dir="../data", required_exts=[".md"]).load_data()
    nodes = SentenceSplitter(chunk_size=512, chunk_overlap=100).get_nodes_from_documents(documents)
    print(f"{len(nodes)} chunks")

    logger.info(f"nodes: {len(nodes)}")
    create_collection()

    for start in range(0, len(nodes), BATCH_SIZE):
        batch = nodes[start:start + BATCH_SIZE]
        texts = [n.get_content() for n in batch]
        logger.debug(f"texts: {texts}")
        dense = embedder.predict({"texts": texts, "mode": "document"})
        sparse = list(sparse_model.embed(texts))

        points = [
            models.PointStruct(
                id=str(uuid.uuid5(uuid.NAMESPACE_URL, n.node_id)),
                vector={
                    "text-dense": d,
                    "text-sparse": models.SparseVector(indices=s.indices.tolist(), values=s.values.tolist()),
                },
                payload=node_to_metadata_dict(n, remove_text=False, flat_metadata=True),
            )
            for n, d, s in zip(batch, dense, sparse)
        ]
        logger.debug(f"points in collection: {len(points)}")
        client.upsert(COLLECTION, points=points)
        print(f"{min(start + BATCH_SIZE, len(nodes))}/{len(nodes)}", flush=True)


if __name__ == "__main__":
    # extract("../data/standard-treatment-guidelines.pdf", "../data/standard-treatment-guidelines.md")
    ingest()

Phase2: Retrieve and generate

Phase 2 runs every time someone asks a question. It has three stages: find candidates, narrow them down, and write a cited answer.

1. Hybrid search. The question is searched two ways at once. For the dense search, it is embedded with the same Qwen3 model used at ingest, this time in query mode, which prepends a short instruction (“given a clinical question, retrieve relevant passages from standard treatment guidelines”). This matches the meaning of the question to the meaning of the chunks. For the sparse search, BM25 matches the exact words, which helps with drug names, doses and clinical terms that embeddings can blur. Each search returns its top 30 chunks from Qdrant. Qdrant then merges the two lists with reciprocal rank fusion, which combines positions in each list rather than raw scores, so the two scoring scales never have to be compared. The result is one list of 30 candidates.

2. Rerank. Search ranks chunks by their similarity to the question. The reranker reads the question and each candidate side by side and scores how well the chunk answers it, as a probability between 0 and 1. We use Qwen3-Reranker-0.6B from the registry and keep the top five. In our BPPV test the five scores were 0.988, 0.676, 0.662, 0.514 and 0.034, so the last one was clearly a poor match, even though it had made the top 30.

3. Generate. The five chunks are numbered and given to the LLM through the MLflow AI Gateway, with a system prompt that sets the rules:

• Use only the numbered passages.
• Write one short factual statement per line.
• End every line with one citation, such as [2].
• Do not quote the passages, and add no introduction or summary.
• If the passages do not answer the question, write only “Not covered in the provided guidelines.”

Temperature is set to 0, so the same question gives the same answer. The format rules are not cosmetic. They are what makes Phase 3 possible, because they turn the answer into lines that can be checked one by one.

The output of Phase 2 is the answer plus the five chunks, with their IDs and scores. Phase 3 needs all of it.

# retrieve_and_generate.py
import json
import logging
import sys
import time

import mlflow
from fastembed import SparseTextEmbedding
from openai import OpenAI
from qdrant_client import QdrantClient, models

COLLECTION = "standard-treatment-guidelines"
TRACKING_URI = "http://localhost:5001"
GATEWAY_URL = "http://localhost:5001/gateway/mlflow/v1"
LLM_ENDPOINT = "ollama"
CANDIDATES = 30
TOP_K = 5

SYSTEM_PROMPT = (
    "You are a clinical assistant answering from standard treatment guidelines. "
    "Use only the numbered context passages. "
    "Write one short factual statement per line. "
    "End every line with exactly one citation such as [2]. "
    "Do not quote the passages, and do not add an introduction, a summary or any comment about the context. "
    "If the context does not answer the question, write only: Not covered in the provided guidelines."
)

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-7s %(name)s | %(message)s", datefmt="%H:%M:%S")
log = logging.getLogger("retriever")


def timed_load(label, loader):
    start = time.perf_counter()
    log.info("loading %s", label)
    obj = loader()
    log.info("loaded %s in %.1fs", label, time.perf_counter() - start)
    return obj


mlflow.set_tracking_uri(TRACKING_URI)
qdrant = timed_load("qdrant client", lambda: QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y"))
embedder = timed_load("embedder qwen3-embedding-4b", lambda: mlflow.pyfunc.load_model("models:/qwen3-embedding-4b/latest"))
reranker = timed_load("reranker qwen3-reranker-0-6b", lambda: mlflow.pyfunc.load_model("models:/qwen3-reranker-0-6b/latest"))
sparse_model = timed_load("sparse model Qdrant/bm25", lambda: SparseTextEmbedding("Qdrant/bm25"))
llm = OpenAI(base_url=GATEWAY_URL, api_key="no-api-key")
log.info("llm gateway=%s endpoint=%s", GATEWAY_URL, LLM_ENDPOINT)


def hybrid_search(query: str, limit: int = CANDIDATES):
    start = time.perf_counter()
    log.info("hybrid_search: query=%r limit=%d", query, limit)
    dense = embedder.predict({"texts": [query], "mode": "query"})[0]
    log.info("hybrid_search: dense embedding dim=%d (%.2fs)", len(dense), time.perf_counter() - start)
    sparse = next(iter(sparse_model.query_embed(query)))
    log.info("hybrid_search: sparse terms=%d", len(sparse.indices))
    response = qdrant.query_points(
        COLLECTION,
        prefetch=[
            models.Prefetch(query=dense, using="text-dense", limit=limit),
            models.Prefetch(
                query=models.SparseVector(indices=sparse.indices.tolist(), values=sparse.values.tolist()),
                using="text-sparse",
                limit=limit,
            ),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=limit,
        with_payload=True,
    )
    log.info("hybrid_search: %d points returned (%.2fs total)", len(response.points), time.perf_counter() - start)
    return response.points


def to_chunk(point) -> dict:
    node = json.loads(point.payload["_node_content"])
    return {"id": str(point.id), "text": node["text"], "metadata": node.get("metadata", {}), "fusion_score": point.score}


def retrieve(query: str, top_k: int = TOP_K) -> list[dict]:
    chunks = [to_chunk(p) for p in hybrid_search(query)]
    if not chunks:
        log.warning("retrieve: no candidates found")
        return []
    start = time.perf_counter()
    log.info("retrieve: reranking %d candidates", len(chunks))
    scores = reranker.predict({"query": query, "documents": [c["text"] for c in chunks]})
    for chunk, score in zip(chunks, scores):
        chunk["rerank_score"] = float(score)
    top = sorted(chunks, key=lambda c: c["rerank_score"], reverse=True)[:top_k]
    log.info("retrieve: reranked in %.2fs, top scores=%s", time.perf_counter() - start, [round(c["rerank_score"], 3) for c in top])
    return top


def generate(query: str, chunks: list[dict]) -> str:
    context = "\n\n".join(f"[{i}] {c['text']}" for i, c in enumerate(chunks, 1))
    prompt = f"Context:\n{context}\n\nQuestion: {query}"
    start = time.perf_counter()
    log.info("generate: endpoint=%s chunks=%d prompt_chars=%d", LLM_ENDPOINT, len(chunks), len(prompt))
    response = llm.chat.completions.create(
        model=LLM_ENDPOINT,
        messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": prompt}],
        temperature=0,
        extra_body={"think": False},
    )
    choice = response.choices[0]
    content = (choice.message.content or "").strip()
    log.info("generate: finish_reason=%s usage=%s answer_chars=%d (%.2fs)", choice.finish_reason, response.usage, len(content), time.perf_counter() - start)
    if not content:
        log.error("generate: empty content, raw message=%s", choice.message)
        raise RuntimeError(f"LLM returned empty content, finish_reason={choice.finish_reason}")
    return content


def answer(query: str) -> dict:
    log.info("answer: start")
    chunks = retrieve(query)
    result = {"query": query, "answer": generate(query, chunks), "chunks": chunks}
    log.info("answer: done")
    return result

# uncomment to test it standalone
# if __name__ == "__main__":
#     question = " ".join(sys.argv[1:]) or "How is benign paroxysmal positional vertigo treated?"
#     result = answer(question)
#     for i, c in enumerate(result["chunks"], 1):
#         print(f"[{i}] rerank={c['rerank_score']:.3f} fusion={c['fusion_score']:.3f} id={c['id']}")
#         print("    " + c["text"][:160].replace("\n", " "))
#     print("\n" + result["answer"])

Phase 3: claim check and verdict gate

Phase 3 takes the answer and the five chunks from Phase 2 and decides what is safe to release. It lives in two small files: one checks claims, the other makes the release decision.

The claim check. Laya is loaded from the MLflow registry the first time it is needed and then kept in memory, so only the first claim pays the loading cost of a few seconds. The check follows four steps:

1. Split. Each line of the answer becomes a claim. The bullet marker and the [n] citation are removed from the text, and the citation numbers are kept separately. A line saying the guidelines don't cover the question is skipped.
2. Collect the evidence. The evidence is the text of the chunk or chunks the claim cites. A citation to a chunk that doesn’t exist is ignored with a warning. If a claim has no valid citation, it is checked against all five chunks and marked as uncited.
3. Ask Laya. Laya receives “Evidence: … Claim: …” with one question, “is the claim supported by the evidence?”, and three defined options: supported (the evidence states the claim), unsupported (it does not mention it), or contradicted (it says the opposite). It returns one choice, in a single forward pass.
4. Record. Each claim keeps its text, its citations, the IDs of the evidence chunks, and its verdict.

The verdict gate. The gate runs the whole pipeline for one question and applies a simple rule to the verdicts:

• Release: every claim is supported, so the answer goes out as written.
• Revised: some claims are not supported. They are removed, and the supported claims are rebuilt into the final answer with their citations.
• Abstain: no claim is supported, or there is nothing to check. The response is a fixed message saying the answer could not be verified from the guidelines.

The gate returns more than the final text. It also returns the original LLM draft, every claim with its verdict, and the five source chunks with their scores, so a reader can see what was removed and why. In our BPPV test, eight claims were checked. One, a sentence about repositioning maneuvers, was marked unsupported against the chunk it cited, and the gate removed it. The other seven were released with their citations.

What the check tests. A “supported” verdict means the claim matches the passage it cites. It does not mean the passage is relevant to the question or right for a particular patient.

# claim_check.py
import logging
import re
import sys
import time

import mlflow

TRACKING_URI = "http://localhost:5001"

QUESTIONS = {
    "verdict": {
        "type": "choice",
        "instructions": "Is the claim supported by the evidence?",
        "criteria": {
            "supported": "the evidence states the claim",
            "unsupported": "the evidence does not mention the claim",
            "contradicted": "the evidence says the opposite",
        },
    }
}

CITATION = re.compile(r"\[(\d+)\]")
LIST_MARKER = re.compile(r"^\s*(?:[-*•]|\d+[.)])\s+")
NOT_COVERED = "not covered"

log = logging.getLogger("claim_check")
_laya = None


def load_laya():
    global _laya
    if _laya is None:
        start = time.perf_counter()
        log.info("loading laya-multilingual")
        mlflow.set_tracking_uri(TRACKING_URI)
        _laya = mlflow.pyfunc.load_model("models:/laya-multilingual/latest")
        log.info("loaded laya-multilingual in %.1fs", time.perf_counter() - start)
    return _laya


def split_claims(answer: str) -> list[dict]:
    claims = []
    for line in answer.splitlines():
        line = LIST_MARKER.sub("", line).strip()
        if not line or NOT_COVERED in line.lower():
            continue
        cited = list(dict.fromkeys(int(n) for n in CITATION.findall(line)))
        text = re.sub(r"\s+([.!?,;])", r"\1", re.sub(r"\s+", " ", CITATION.sub("", line))).strip()
        if len(text) > 3:
            claims.append({"claim": text, "citations": cited})
    log.info("split_claims: %d claims from %d lines", len(claims), len(answer.splitlines()))
    return claims


def check_claim(claim: str, evidence: str) -> str:
    start = time.perf_counter()
    result = load_laya().predict({"state": f"Evidence: {evidence}\nClaim: {claim}", "questions": QUESTIONS})
    verdict = result["answers"]["verdict"]["choice"]
    log.info("check_claim: verdict=%s evidence_chars=%d (%.2fs)", verdict, len(evidence), time.perf_counter() - start)
    log.debug("check_claim: raw result=%s", result)
    return verdict


def check_answer(answer: str, chunks: list[dict]) -> list[dict]:
    log.info("check_answer: start, %d chunks available", len(chunks))
    results = []
    for i, item in enumerate(split_claims(answer), 1):
        valid = list(dict.fromkeys(n for n in item["citations"] if 1 <= n <= len(chunks)))
        if item["citations"] and len(valid) != len(item["citations"]):
            log.warning("claim %d cites out-of-range chunks %s", i, item["citations"])
        if not valid:
            log.warning("claim %d has no valid citation, checking against all chunks", i)
        sources = valid or list(range(1, len(chunks) + 1))
        log.info("claim %d: %r citations=%s", i, item["claim"][:80], sources)
        evidence = "\n\n".join(chunks[n - 1]["text"] for n in sources)
        results.append({
            **item,
            "evidence_chunks": [chunks[n - 1]["id"] for n in sources],
            "cited": bool(valid),
            "verdict": check_claim(item["claim"], evidence),
        })
    verdicts = [r["verdict"] for r in results]
    log.info("check_answer: done, verdicts=%s", {v: verdicts.count(v) for v in set(verdicts)})
    return results


# if __name__ == "__main__":
#     logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-7s %(name)s | %(message)s", datefmt="%H:%M:%S")
#     from retriever_and_generate import answer
#
#     """
#     1. How is benign paroxysmal positional vertigo treated?
#     2. Compare the treatment of BPPV and vestibular migraine.
#     """
#     question = " ".join(sys.argv[1:]) or "How is benign paroxysmal positional vertigo treated?"
#     response = answer(question)
#     print(response["answer"], "\n")
#     for r in check_answer(response["answer"], response["chunks"]):
#         source = "cited" if r["cited"] else "uncited"
#         print(f"{r['verdict']:<13} | {source:<7} | {r['claim']}")

# verdict_gate.py
import logging
import sys

from claim_check import check_answer
from retriever_and_generate import answer

log = logging.getLogger("gate")

ABSTAIN = "I could not verify an answer to this question from the guidelines."


def render(claims: list[dict]) -> str:
    return "\n".join(f"{c['claim']} " + "".join(f"[{n}]" for n in c["citations"]) for c in claims)


def gate(question: str) -> dict:
    result = answer(question)
    draft = result["answer"]
    checks = check_answer(draft, result["chunks"])
    supported = [c for c in checks if c["verdict"] == "supported"]
    dropped = len(checks) - len(supported)

    if not supported:
        log.info("gate: no supported claims, abstaining")
        return {"status": "abstain", "answer": ABSTAIN, "draft": draft, "claims": checks, "chunks": result["chunks"]}

    if dropped == 0:
        log.info("gate: all %d claims supported, releasing", len(checks))
        status = "release"
    else:
        log.info("gate: dropped %d of %d unsupported claims, releasing the rest", dropped, len(checks))
        status = "revised"
    return {"status": status, "answer": render(supported), "draft": draft, "claims": checks, "chunks": result["chunks"]}


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-7s %(name)s | %(message)s", datefmt="%H:%M:%S")
    question = " ".join(sys.argv[1:]) or "How is benign paroxysmal positional vertigo treated?"
    outcome = gate(question)

    print("\n=== LLM draft ===")
    print(outcome["draft"])
    print("\n=== Claim check ===")
    for c in outcome["claims"]:
        cites = "".join(f"[{n}]" for n in c["citations"]) or "[uncited]"
        print(f"{c['verdict']:<13} | {cites:<8} | {c['claim']}")
    print("\n=== Sources ===")
    for i, ch in enumerate(outcome["chunks"], 1):
        print(f"[{i}] rerank={ch['rerank_score']:.3f} id={ch['id']}")
    print(f"\n=== Final response: {outcome['status']} ===")
    print(outcome["answer"])

The Run

06:21:36 INFO    retrieve and generate | loaded sparse model Qdrant/bm25 in 1.2s
06:21:36 INFO    retrieve and generate | llm gateway=http://localhost:5001/gateway/mlflow/v1 endpoint=ollama
06:21:36 INFO    retrieve and generate | answer: start
06:21:36 INFO    retrieve and generate | hybrid_search: query='How is benign paroxysmal positional vertigo treated?' limit=30
Batches: 100%|██████████| 1/1 [00:08<00:00,  8.82s/it]
06:21:45 INFO    retrieve and generate | hybrid_search: dense embedding dim=2560 (8.83s)
06:21:45 INFO    retrieve and generate | hybrid_search: sparse terms=5
06:21:45 INFO    httpx | HTTP Request: POST http://localhost:6333/collections/standard-treatment-guidelines/points/query "HTTP/1.1 200 OK"
06:21:45 INFO    retrieve and generate | hybrid_search: 30 points returned (8.96s total)
06:21:45 INFO    retrieve and generate | retrieve and generate: reranking 30 candidates
06:21:55 INFO    retrieve and generate | retrieve and generate: reranked in 9.47s, top scores=[0.988, 0.676, 0.662, 0.514, 0.034]
06:21:55 INFO    retrieve and generate | generate: endpoint=ollama chunks=5 prompt_chars=7554
06:22:18 INFO    httpx | HTTP Request: POST http://localhost:5001/gateway/mlflow/v1/chat/completions "HTTP/1.1 200 OK"
06:22:18 INFO    retrieve and generate | generate: finish_reason=stop usage=CompletionUsage(completion_tokens=248, prompt_tokens=1936, total_tokens=2184, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cache_write_tokens=None, cached_tokens=1930)) answer_chars=781 (23.65s)
06:22:18 INFO    retrieve and generate | answer: done


06:22:18 INFO    claim_check | loading laya-multilingual
Downloading artifacts: 100%|██████████| 11/11 [00:01<00:00,  8.52it/s]
/Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/LLMs/rag_with_decisive_model/.venv/lib/python3.13/site-packages/mlflow/pyfunc/utils/data_validation.py:187: UserWarning: Add type hints to the `predict` method to enable data validation and automatic signature inference during model logging. Check https://mlflow.org/docs/latest/model/python_model.html#type-hint-usage-in-pythonmodel for more details.
  color_warning(
06:22:32 INFO    claim_check | loaded laya-multilingual in 13.1s
06:22:32 INFO    claim_check | check_claim: verdict=unsupported evidence_chars=1445 (13.43s)
06:22:32 INFO    claim_check | claim 2: 'The nystagmus beats upward and torsionally with posterior canal BPPV.' citations=[1]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=1687 (0.08s)
06:22:32 INFO    claim_check | claim 3: 'Horizontal canal BPPV resulting in a horizontal nystagmus.' citations=[1]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=1687 (0.08s)
06:22:32 INFO    claim_check | claim 4: 'Antihistamines like Tab. meclizine (25-50 mg 3 times daily), Tab. dimenhydrinate' citations=[1]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=1687 (0.08s)
06:22:32 INFO    claim_check | claim 5: 'Benzodiazepines like Tab. diazepam (2.5 mg 1-3 times daily) and Tab. clonazepam ' citations=[1]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=1687 (0.08s)
06:22:32 INFO    claim_check | claim 6: 'Tab. baclofen 15-240 mg/d in divided doses is effective.' citations=[3]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=808 (0.05s)
06:22:32 INFO    claim_check | claim 7: 'Tab. diazepam 2-4 mg at bed time acts by a similar mechanism and is useful for l' citations=[3]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=808 (0.04s)
06:22:32 INFO    claim_check | claim 8: 'Tab. Tizanidine 2-8 mg TDS is another option.' citations=[3]
06:22:32 INFO    claim_check | check_claim: verdict=supported evidence_chars=808 (0.04s)
06:22:32 INFO    claim_check | check_answer: done, verdicts={'unsupported': 1, 'supported': 7}


06:22:32 INFO    gate | gate: dropped 1 of 8 unsupported claims, releasing the rest

ACTUAL REPONSE  === LLM draft ===
With repositioning maneuvers that utilize gravity to remove the otoconia from the semicircular canal [4].
The nystagmus beats upward and torsionally with posterior canal BPPV [1].
Horizontal canal BPPV resulting in a horizontal nystagmus [1].
Antihistamines like Tab. meclizine (25-50 mg 3 times daily), Tab. dimenhydrinate (50 mg 1-2 times daily) and Tab. promethazine (25 mg 2-3 times daily, also can be given rectally and IM) are used [1].
Benzodiazepines like Tab. diazepam (2.5 mg 1-3 times daily) and Tab. clonazepam (0.25 mg 1 - 3 times daily [1].
Tab. baclofen 15-240 mg/d in divided doses is effective [3].
Tab. diazepam 2-4 mg at bed time acts by a similar mechanism and is useful for leg spasms that interrupt sleep [3].
Tab. Tizanidine 2-8 mg TDS is another option [3].

=== Claim check ===
unsupported   | [4]      | With repositioning maneuvers that utilize gravity to remove the otoconia from the semicircular canal.
supported     | [1]      | The nystagmus beats upward and torsionally with posterior canal BPPV.
supported     | [1]      | Horizontal canal BPPV resulting in a horizontal nystagmus.
supported     | [1]      | Antihistamines like Tab. meclizine (25-50 mg 3 times daily), Tab. dimenhydrinate (50 mg 1-2 times daily) and Tab. promethazine (25 mg 2-3 times daily, also can be given rectally and IM) are used.
supported     | [1]      | Benzodiazepines like Tab. diazepam (2.5 mg 1-3 times daily) and Tab. clonazepam (0.25 mg 1 - 3 times daily.
supported     | [3]      | Tab. baclofen 15-240 mg/d in divided doses is effective.
supported     | [3]      | Tab. diazepam 2-4 mg at bed time acts by a similar mechanism and is useful for leg spasms that interrupt sleep.
supported     | [3]      | Tab. Tizanidine 2-8 mg TDS is another option.

=== Sources ===
[1] rerank=0.988 id=372cd729-4e4a-5d1c-b462-04d6880bf691
[2] rerank=0.676 id=2fa8b876-7caf-5263-94a5-dd4b21c5a170
[3] rerank=0.662 id=73570d2b-02ae-51d3-8b96-f4850a5e62fc
[4] rerank=0.514 id=f2c0bb7c-bace-5ddd-9925-309ff2b72d94
[5] rerank=0.034 id=117ff53f-ff65-5deb-b3d2-029ce56a89c0

=== Final response: revised ===
The nystagmus beats upward and torsionally with posterior canal BPPV. [1]
Horizontal canal BPPV resulting in a horizontal nystagmus. [1]
Antihistamines like Tab. meclizine (25-50 mg 3 times daily), Tab. dimenhydrinate (50 mg 1-2 times daily) and Tab. promethazine (25 mg 2-3 times daily, also can be given rectally and IM) are used. [1]
Benzodiazepines like Tab. diazepam (2.5 mg 1-3 times daily) and Tab. clonazepam (0.25 mg 1 - 3 times daily. [1]
Tab. baclofen 15-240 mg/d in divided doses is effective. [3]
Tab. diazepam 2-4 mg at bed time acts by a similar mechanism and is useful for leg spasms that interrupt sleep. [3]
Tab. Tizanidine 2-8 mg TDS is another option. [3]

Process finished with exit code 0

The Conclusion

We started with a simple question: after a RAG system writes an answer, how do you know each statement in it is backed by the passage it cites? The answer here was to keep the pipeline small and make every step inspectable. One MLflow server serves the models, Qdrant combines meaning-based and keyword search, and the answer is written in a format that can be split into claims and checked one at a time by a small, fast model called Laya. The gate then releases, trims or withholds the answer. What you take away is a pattern more than a product. A verifier like this tells you whether a claim matches its cited passage, and it does not tell you whether that passage is the right one for the question or for the patient. The base Laya checkpoint we used is not fine-tuned, so before relying on it for anything real you should test it with claims you know are wrong and calibrate it on your own documents. A natural next step is to weight the checks by risk, with stricter handling for doses and contraindications, and to add a relevance check beside faithfulness. A clinician stays in charge of the final decision.

Catching Hallucinations Before They Ship: A Claim-Level Verdict Gate for Clinical RAG was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few months ago I loaded the MedQuAD corpus into Qdrant to build a patient-education assistant. Nothing fancy: 7,936 medical explainer pages, one point each, with a topic, a summary and some keywords sitting in the payload next to the vector. The search part worked on day one. What kept eating my evenings was everything around the search.

Two requests in particular kept coming back. A content reviewer wanted “every page about childhood cancers” so she could check them against updated guidelines. I had a keyword index on the topic field, but keyword indexes match whole values, so “Childhood” alone found nothing and “Childhood Leukemia” found exactly one page. I ended up writing a small script that scrolled all 7,936 payloads and did a startswith in Python. It worked. It was also embarrassing.

The second request came from the evaluation side. We were about to switch embedding models and I wanted two things: to re-embed the whole collection in parallel without two workers ever touching the same page, and to keep a fixed ten percent of the pages as an evaluation set so that recall numbers from this week could honestly be compared with next week’s. Qdrant’s random sampling gave me a different ten percent on every call, and splitting IDs across workers meant hand-maintaining ranges that broke the moment someone added a page.

Qdrant 1.19 shipped two filter conditions that made both scripts disappear. One is prefix matching on keyword indexes. The other is a slice condition that carves any collection into fixed, non-overlapping pieces using nothing but the point ID. Neither is glamorous. Both are the kind of feature you only appreciate after you’ve built the workaround yourself.

This post walks through how each of them works under the hood, with the medical corpus as the running example. By the end you’ll know when a prefix index beats a text index, why a hash of the point ID is a better sampling tool than a random number, and how to combine the two to get a reproducible sample of exactly the slice of data you care about. All the code is in the linked notebook; the numbers you’ll see in the post are real runs against the collection.

Introduction

Vector search is very good at one question: “which points are closest to this query?” It is silent on every other question you might have about your data. Which of these results belong to the childhood oncology section? Which were published after the guideline change? Which pages am I even allowed to show this user? None of that lives in the vector. It lives in the payload, and the only way to bring it into the search is a filter.

Payload metadata is what turns a pile of embeddings into a system you can actually operate. In the MedQuAD collection every point carries a topic, a short summary, ten keywords and the original text. The vector answers “what is this page about in a fuzzy sense”. The topic field answers “what is this page about, exactly”. Both are needed. A user asking about treatment options for a child gets a semantically close page about adult chemotherapy dosing unless something narrows the candidate set first, and that something is a filter on the payload.

Qdrant treats filters as first-class rather than a post-processing step. A filter condition is evaluated during the HNSW traversal, so the graph walk only ever visits points that satisfy it, and the same condition works for scroll, count and recommend. The catch is that a filter is only as fast as the index behind it. A keyword index gives you constant-time exact matches. A text index gives you tokenised full-text matches. Until 1.19, “give me everything whose topic starts with Childhood” fell between those two: too loose for keyword, too strict for text, so it degraded to a scan.

There is also a class of questions that no payload field can answer, because they are about the point itself rather than its content. “Give me a fixed quarter of this collection” is one. “Give me the same ten percent I evaluated last week” is another. These need a filter that derives its answer from the point ID and nothing else, deterministically, with no index to maintain. That is exactly the gap the slice condition fills, and it’s the one I’ll start with.

Slice Filter:

The slice condition takes two integers, index and total, and answers a single yes-or-no question for each point: does hash(point_id) mod total equal index? That is the entire mechanism. No payload is read, no index is built, and nothing is written to disk. The hash is fixed, so a given ID lands in the same bucket on every run and on every Qdrant version, and because every integer has exactly one remainder mod total, the buckets 0 through total − 1 are disjoint and together cover the whole collection. When I asked four workers to each scroll slice(i, 4) over the medical corpus, they came back with 2,011, 1,957, 1,998 and 1,970 points; the sum is 7,936, the union is 7,936, and no ID appeared twice. Each worker only knows its own index. There is no queue, no ID range to hand out and nothing to reconcile if one of them dies and restarts.

The same property turns slicing into a sampling tool that random sampling can’t match. Asking for slice(0, 10) twice returned the identical 794 IDs, which is what you want when this week's retrieval recall has to be compared with last week's. Shrinking the sample is safe too: slice(0, 4) is a strict subset of slice(0, 2), because a hash divisible by four is also divisible by two, so nothing you have already evaluated drops out when you halve the set. And since slice is an ordinary filter condition, it composes with everything else. Combining slice(0, 5) with a topic prefix of "Childhood" gave me the same fifth of the childhood cancer pages every time, a reproducible stratified sample ready for a canary run. One honest caveat: the condition is checked per candidate in every shard, so it divides the work across your clients rather than reducing the server's total work. For a collection this size that is exactly the trade you want.

Replace my four worker counts and the 794 with the numbers from your actual run so they match the notebook.

import asyncio
import logging
from qdrant_client import AsyncQdrantClient, models

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("slice")


class SliceScroller:
    def __init__(self, client: AsyncQdrantClient, collection: str, show: int = 3):
        self.client = client
        self.collection = collection
        self.show = show

    def print_points(self, label: str, points: dict[int, dict]):
        print(f"\n{label}: {len(points)} points")
        for pid, p in list(points.items())[: self.show]:
            print(f"  [{pid}] {p['topic']}\n      {p['text'][:150]}...")

    async def slice_points(self, index: int, total: int,
                           extra: list[models.Condition] | None = None) -> dict[int, dict]:
        conditions = [models.SliceCondition(slice=models.Slice(index=index, total=total))]
        points, offset = {}, None
        while True:
            records, offset = await self.client.scroll(
                collection_name=self.collection,
                scroll_filter=models.Filter(must=conditions + (extra or [])),
                limit=1000,
                with_payload=["topic", "text"],
                with_vectors=False,
                offset=offset,
            )
            points |= {r.id: r.payload for r in records}
            if offset is None:
                break
        log.info("slice %d/%d -> %d points", index, total, len(points))
        return points

    async def parallel_scroll(self, workers: int):
        log.info("scrolling with %d workers", workers)
        slices = await asyncio.gather(*(self.slice_points(i, workers) for i in range(workers)))
        for i, s in enumerate(slices):
            self.print_points(f"worker {i} (slice {i}/{workers})", s)
        union = set().union(*(s.keys() for s in slices))
        total = sum(map(len, slices))
        log.info("sum=%d  union=%d  overlap=%s", total, len(union), total != len(union))

    async def reproducible_sample(self, index: int, total: int):
        first = await self.slice_points(index, total)
        second = await self.slice_points(index, total)
        self.print_points(f"sample slice {index}/{total}", first)
        log.info("same sample on repeat: %s", first.keys() == second.keys())

    async def nested_samples(self):
        half = await self.slice_points(0, 2)
        quarter = await self.slice_points(0, 4)
        log.info("slice 0/4 is a subset of slice 0/2: %s", quarter.keys() <= half.keys())

    async def stratified_sample(self, index: int, total: int, prefix: str):
        points = await self.slice_points(index, total, extra=[
            models.FieldCondition(key="topic", match=models.MatchPrefix(prefix=prefix))
        ])
        self.print_points(f"stratified slice {index}/{total} within {prefix!r}", points)
        log.info("all topics start with %r: %s", prefix,
                 all(p["topic"].startswith(prefix) for p in points.values()))


async def main():
    s = SliceScroller(AsyncQdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y"), "medquad")
    await s.parallel_scroll(workers=4)
    await s.reproducible_sample(index=0, total=10)
    await s.nested_samples()
    await s.stratified_sample(index=0, total=5, prefix="Childhood")


if __name__ == "__main__":
    asyncio.run(main())

Prefix Match:

A keyword index in Qdrant is a dictionary: each distinct value maps to the list of point IDs that carry it. That design is why exact matches are fast and why identifiers, URLs and category names belong in a keyword field rather than a text field. It is also why “starts with” used to be awkward. A text index would let me find pages containing the token “Childhood”, but it tokenises values and stops guaranteeing that “Bile Duct Cancer” is one thing, and it would happily match “Childhood” inside “Adult Survivors of Childhood Cancer”. A keyword index gave exactness but only for whole values. Creating the index with prefix: true in 1.19 closes that gap by keeping the dictionary keys in sorted order in a dedicated structure, so every value sharing a prefix sits in one contiguous run. A prefix query is a binary search to the first key at or after the prefix, followed by a walk forward until the keys stop matching, then a union of the ID lists it passed. The cost depends on the number of matching values and the size of the dictionary, not on the number of points in the collection.

In practice this changed two things for me. The reviewer’s request became a single scroll with MatchPrefix(prefix="Childhood") on the topic field, which returned all 219 pages across 41 distinct childhood cancer topics with no Python loop and no full payload read. More usefully for the assistant itself, the same condition sits inside query_points as a filter, so a question like "what are the treatment options" is scored only against the childhood subset and never surfaces an adult dosing page, however close the vectors happen to be. Two things to keep in mind: matching is exact and case-sensitive because the dictionary stores values verbatim, so consistent casing in your metadata matters, and prefix support must be enabled when the index is created, it's not something you toggle on an existing keyword index.

Swap the 219 and 41 for the counts from your run. Say proceed for the conclusion.

import logging
from openai import OpenAI
from qdrant_client import QdrantClient, models
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("prefix")


class PrefixSearch:
    def __init__(self, client: QdrantClient, openai: OpenAI, collection: str,
                 embedder: str = "text-embedding-3-small"):
        self.client = client
        self.openai = openai
        self.collection = collection
        self.embedder = embedder

    def prefix_filter(self, prefix: str) -> models.Filter:
        return models.Filter(must=[
            models.FieldCondition(key="topic", match=models.MatchPrefix(prefix=prefix))
        ])

    def topics_starting_with(self, prefix: str) -> list[str]:
        log.info("scrolling topics with prefix %r", prefix)
        topics, offset = [], None
        while True:
            records, offset = self.client.scroll(
                collection_name=self.collection,
                scroll_filter=self.prefix_filter(prefix),
                limit=256,
                with_payload=["topic"],
                with_vectors=False,
                offset=offset,
            )
            topics += [r.payload["topic"] for r in records]
            if offset is None:
                break
        log.info("%d topics start with %r", len(topics), prefix)
        return sorted(set(topics))

    def search(self, query: str, prefix: str, limit: int = 5) -> list[models.ScoredPoint]:
        log.info("semantic search %r within prefix %r", query, prefix)
        vector = self.openai.embeddings.create(model=self.embedder, input=query).data[0].embedding
        hits = self.client.query_points(
            collection_name=self.collection,
            query=vector,
            query_filter=self.prefix_filter(prefix),
            limit=limit,
            with_payload=["topic", "summary"],
        ).points
        for h in hits:
            log.info("  %.3f  %s", h.score, h.payload["topic"])
        return hits


if __name__ == "__main__":
    ps = PrefixSearch(QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y"), OpenAI(), "medquad")

    for t in ps.topics_starting_with("Childhood"):
        print(t)

    ps.search("what are the treatment options", prefix="Childhood", limit=5)

The ingestion of the data looks as below. The dataset is my own regrouped version of MedQuAD, published on Hugging Face as pavanmantha/MedQuAD_Context_GroupedQA. MedQuAD itself is a collection of medical question-answer pairs scraped from NIH websites such as the National Cancer Institute and the Genetic and Rare Diseases centre, so the language is patient-facing and the topics run from common cancers to rare genetic conditions. The original release is question-centric, one row per question. For retrieval work I found that inconvenient: many questions share the same source passage, and what a RAG system actually indexes is the passage, not the question. So I regrouped it by context. Each row is one source passage plus the list of every question and answer that was written against it. There are 7,936 rows; passages range from a single sentence of 77 characters to long treatment overviews of over 30,000 characters.

For this experiment I only used the context column. The question-answer pairs never enter the collection, which keeps the payload clean and avoids leaking evaluation questions into the index. Each context became one point in Qdrant with four payload fields: topic, the single condition or subject the passage is about; key_words, the ten most important terms; summary, one to three sentences; and text, the passage itself. The first three were generated with an LLM through the OpenAI Batch API, which processed all 7,936 passages overnight at half the price of synchronous calls. The embeddings came from the same batch pipeline, one request per passage, joined back to the metadata by custom_id. The topic field is the one that matters for this post: it's what the keyword index with prefix support is built on, and it's what makes "everything starting with Childhood" a meaningful question to ask.

import json
import logging
import time
from pathlib import Path

from datasets import load_dataset
from openai import OpenAI
from pydantic import BaseModel, ConfigDict
from qdrant_client import QdrantClient, models
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("medquad")


class Metadata(BaseModel):
    model_config = ConfigDict(extra="forbid")
    topic: str
    key_words: list[str]
    summary: str


class MedQuADBatchIndexer:
    PROMPT = (
        "Read the medical text and return: "
        "topic (the single condition or subject it is about), "
        "key_words (the 10 most important terms), "
        "summary (1 to 3 sentences).\n\n{text}"
    )

    def __init__(self, client: QdrantClient, openai: OpenAI, collection: str,
                 llm: str = "gpt-4o-mini",
                 embedder: str = "text-embedding-3-small", dim: int = 1536,
                 workdir: str = "batch"):
        self.client = client
        self.openai = openai
        self.collection = collection
        self.llm = llm
        self.embedder = embedder
        self.dim = dim
        self.workdir = Path(workdir)
        self.workdir.mkdir(exist_ok=True)
        log.info("indexer ready: collection=%s llm=%s embedder=%s", collection, llm, embedder)

    def create_collection(self):
        log.info("creating collection %s (dim=%d)", self.collection, self.dim)
        self.client.create_collection(
            collection_name=self.collection,
            vectors_config=models.VectorParams(size=self.dim, distance=models.Distance.COSINE),
        )
        self.client.create_payload_index(
            collection_name=self.collection,
            field_name="topic",
            field_schema=models.KeywordIndexParams(type="keyword", prefix=True),
        )
        log.info("keyword index with prefix=True created on 'topic'")

    def chat_request(self, idx: int, text: str) -> dict:
        return {
            "custom_id": str(idx),
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": self.llm,
                "messages": [{"role": "user", "content": self.PROMPT.format(text=text)}],
                "response_format": {
                    "type": "json_schema",
                    "json_schema": {"name": "metadata", "strict": True,
                                    "schema": Metadata.model_json_schema()},
                },
            },
        }

    def embed_request(self, idx: int, text: str) -> dict:
        return {
            "custom_id": str(idx),
            "method": "POST",
            "url": "/v1/embeddings",
            "body": {"model": self.embedder, "input": text},
        }

    def submit(self, name: str, endpoint: str, requests: list[dict]) -> str:
        path = self.workdir / f"{name}.jsonl"
        path.write_text("\n".join(json.dumps(r) for r in requests))
        log.info("[%s] wrote %d requests to %s", name, len(requests), path)

        file = self.openai.files.create(file=path.open("rb"), purpose="batch")
        batch = self.openai.batches.create(
            input_file_id=file.id, endpoint=endpoint, completion_window="24h",
            metadata={"job": name},
        )
        log.info("[%s] batch %s submitted (status=%s)", name, batch.id, batch.status)
        return batch.id

    def wait(self, batch_id: str, every: int = 60) -> dict:
        while True:
            b = self.openai.batches.retrieve(batch_id)
            c = b.request_counts
            log.info("[%s] %s  %d/%d done, %d failed", batch_id, b.status, c.completed, c.total, c.failed)
            if b.status in ("completed", "failed", "expired", "cancelled"):
                if b.status != "completed":
                    raise RuntimeError(f"batch {batch_id} ended with status {b.status}")
                if b.error_file_id:
                    log.warning("[%s] error file: %s", batch_id, b.error_file_id)
                return self.results(b.output_file_id)
            time.sleep(every)

    def results(self, output_file_id: str) -> dict:
        lines = self.openai.files.content(output_file_id).text.splitlines()
        out = {}
        for line in lines:
            r = json.loads(line)
            if r["response"]["status_code"] == 200:
                out[r["custom_id"]] = r["response"]["body"]
            else:
                log.warning("request %s failed: %s", r["custom_id"], r.get("error"))
        log.info("parsed %d successful results", len(out))
        return out

    def index(self, batch_size: int = 64) -> int:
        log.info("loading dataset")
        ds = load_dataset("pavanmantha/MedQuAD_Context_GroupedQA", split="train")
        texts = {i: row["context"] for i, row in enumerate(ds)}
        log.info("%d contexts loaded", len(texts))

        meta_id = self.submit("metadata", "/v1/chat/completions",
                              [self.chat_request(i, t) for i, t in texts.items()])
        emb_id = self.submit("embeddings", "/v1/embeddings",
                             [self.embed_request(i, t) for i, t in texts.items()])

        meta, emb = self.wait(meta_id), self.wait(emb_id)

        points = []
        for i, text in texts.items():
            k = str(i)
            if k not in meta or k not in emb:
                log.warning("point %d skipped: missing metadata or embedding", i)
                continue
            m = Metadata.model_validate_json(meta[k]["choices"][0]["message"]["content"])
            points.append(models.PointStruct(
                id=i,
                vector=emb[k]["data"][0]["embedding"],
                payload={**m.model_dump(), "text": text},
            ))

        log.info("uploading %d points in batches of %d", len(points), batch_size)
        self.client.upload_points(self.collection, points, batch_size=batch_size, wait=True)
        count = self.client.count(self.collection).count
        log.info("done: %d points in collection", count)
        return count


if __name__ == "__main__":
    indexer = MedQuADBatchIndexer(
        client=QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y"),
        openai=OpenAI(),
        collection="medquad",
    )
    indexer.create_collection()
    indexer.index()

The Conclusion

These two features in this post look small on a release-notes page and both of them replaced code I had been maintaining for months. The lesson I took away is that filters come in two kinds. Prefix match is a filter that knows the payload: it reads a value you wrote, and its power comes from the index structure behind it, a sorted dictionary that turns “starts with” into a range lookup. Slice is a filter that knows the point: it never reads the payload at all, and its power comes from a hash of the ID that is stable, disjoint and complete by construction. Once that distinction is clear, it becomes obvious which one to reach for. Questions about content go to prefix. Questions about partitioning, sampling and reproducibility go to slice. Questions that need both, like “a fixed fifth of the childhood cancer pages”, go to a filter that combines them.

Where I’d apply these next. Prefix match belongs anywhere a keyword field has natural hierarchy in its string: file paths, URL prefixes, SKU families, ICD-style codes, tenant IDs that share a customer prefix. Anywhere you’re tempted to switch to a text index just to get “starts with”, don’t; enable prefix on the keyword index instead and keep exact matching. Slice belongs in every batch job that touches a whole collection: re-embedding after a model change, backfilling a new payload field, exporting to a data lake, running periodic quality checks. Give each worker an index and let the hash do the coordination. And for evaluation, stop using random sampling for anything you intend to compare over time. A slice is a frozen sample you can shrink without losing history, and combining it with a payload filter gives you stratified samples for free.

Both features need Qdrant 1.19 or later, and prefix support must be set when the keyword index is created. The notebook linked at the top reproduces every number in this post against the MedQuAD collection, including the worker counts and the repeat-sample check. If you try either feature on your own data and hit something I didn’t cover, I’d like to hear about it.

full source code: https://github.com/pavanjava/qdrant_filters_tutorial

Slicing and Prefixing a Medical Knowledge Base: Two Small Qdrant Features That Solve Big… was originally published in Towards Dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

That is the idea behind this architecture. Using Agno, we create a small team of AI agents where a Coding Agent builds the solution and a Critique Agent independently reviews it. With access to Web Search, coding tools, FileSystem, and Shell, the agent can go beyond code generation and actually work with the development environment.

The result is a simple but powerful idea: AI that does not just write code, but iteratively works toward a validated final artifact.

The Architecture:

The architecture starts with a user query. This could be a request to create a new application, implement a feature, fix a bug, modify an existing repository, or generate a complete coding artifact.

The request enters the agent environment built using Agno. Instead of relying on a single general-purpose agent, the architecture separates responsibilities between two specialized agents: the Coding Agent and the Critique Agent.

The Coding Agent is the primary implementation agent. It receives dedicated coding instructions that define how it should understand the requirement, plan the implementation, generate code, and work with the surrounding development environment. Its responsibility is not limited to producing code as text. It can actively work on the solution using the tools available to it.

One of those capabilities is Web Search. When the Coding Agent needs additional technical information, documentation, API references, library usage details, or implementation guidance, it can retrieve that information rather than depending entirely on the model’s existing knowledge.

The Coding Agent also has access to coding tools, which allow it to create, update, and reason over source code. This becomes especially useful when the task contains multiple files or requires changes across an existing codebase.

Another important component is FileSystem & Shell access. This moves the agent from being simply a code-generation assistant toward behaving more like a development agent. It can inspect directories, read existing source files, create new files, execute commands, run applications, install or inspect dependencies, execute tests, and observe runtime errors.

The architecture also supplies environment variables to the agent runtime. These can contain configuration such as API keys, model settings, service endpoints, repository configuration, database connections, or runtime parameters. Keeping these outside the core agent instructions makes the architecture easier to configure across different environments.

Once the Coding Agent produces an implementation, the work is passed to the Critique Agent.

The Critique Agent operates using a separate set of critique and verification instructions. Its purpose is to independently inspect what the Coding Agent has produced. It can look for incorrect implementations, missing requirements, logical problems, incomplete code, or areas that require improvement.

The key part of the architecture is the feedback loop between the Coding Agent and Critique Agent.

The Coding Agent produces an implementation. The Critique Agent reviews it and provides feedback. If changes are required, the feedback is returned to the Coding Agent. The Coding Agent then updates the implementation and sends it back for another review.

Conceptually, the workflow becomes:

User Query → Coding Agent → Implementation → Critique Agent → Feedback → Coding Agent → Validation → Final Artifact

This iterative process can continue until the implementation satisfies the expected requirements.

Finally, once the solution has been implemented and verified, the system produces the Final Artifact. Depending on the original request, this could be source code, a complete application, configuration files, scripts, repository changes, or another executable software deliverable.

At the center of the entire architecture is Agno, which provides the framework for defining the agents, assigning their instructions, integrating tools, and coordinating the interaction between them. So rather than building one large coding agent responsible for everything, this architecture follows a much cleaner separation of responsibilities:

Coding Agent builds. Critique Agent verifies. Tools provide real-world execution capabilities. Agno coordinates the workflow. That combination turns a simple coding assistant into a more practical agentic coding system.

The Implementation

Source code of this project: https://github.com/pavanjava/coding_agents_with_agno

pyproject.toml

The pyproject.toml file defines the project metadata, required Python dependencies, build configuration, and the qdrant-agent CLI entry point. It makes the project installable as a Python package and connects the terminal command directly to the application's main CLI function.

cli.py

The cli.py file is the main entry point of the application. It loads environment variables, lets the user choose models for the coder and critic agents, validates required API keys, collects the Qdrant connection details, builds the agent team, and finally starts the interactive CLI experience.

agent.py

The agent.py file contains the core agent setup and orchestration logic. It creates the Coder Agent and Critic Agent, assigns models, tools, shared Qdrant context, local skills, persistent session memory, and combines everything into an Agno Team that manages the code → review → fix workflow.

instructions.py

The instructions.py file defines the behavioral rules for the entire system. It contains the shared instructions, coder-specific instructions, critic-specific verification rules, and the overall team workflow that ensures one agent writes the code while the other independently reviews it.

skills/

The skills/ directory contains a local offline copy of the Qdrant knowledge base used by both agents. It provides structured guidance across topics such as Qdrant deployment, scaling, monitoring, multitenancy, SDKs, and operations, allowing the agents to rely on documented Qdrant knowledge before using external web search.

CodingTools

The CodingTools component gives the agents direct access to the development environment. The coder receives full capabilities such as reading, writing, editing, searching, listing files, and executing shell commands, while the critic gets a restricted version that allows inspection and execution but prevents modification of the final deliverable.

TavilyTools

The TavilyTools integration provides web search capabilities when local Qdrant skills are not enough. It allows the agents to look up recent documentation, APIs, implementation details, or external technical information needed to complete a task correctly.

JsonDb

The JsonDb component provides lightweight persistent storage for conversation history and agent sessions. Each project directory gets its own stable session identity, allowing the coding workflow and context to continue across application restarts without mixing information between unrelated projects.

Agno Team

The Agno Team is the coordination layer that brings the coder and critic agents together. It manages delegation, passes work between the agents, maintains shared context and history, and continues the write → critique → fix → approve cycle until a validated final result is ready.

The Conclusion

Today we have seen how Agno can be used to build a practical coding agent where software development is treated as an iterative process rather than a single prompt-and-response interaction.

By combining a Coder Agent, an independent Critic Agent, Qdrant skills, CodingTools, Tavily web search, persistent JsonDb sessions, and controlled access to the local development environment, the system can move through a clear write → review → fix → validate workflow.

The important idea here is simple: a useful coding agent should not only generate code. It should understand the project, use the right tools, verify its own work through an independent reviewer, and finally produce a cleaner and more reliable software artifact.

Creating coding agents with Agno was originally published in 𝐀𝐈 𝐦𝐨𝐧𝐤𝐬.𝐢𝐨 on Medium, where people are continuing the conversation by highlighting and responding to this story.

This thought process can be changed by combining domain-specific embedding models, intelligent query classification, vector search, and dynamic LLM routing. Instead of forcing every query through the same path, the application can first understand what the user is asking, select the right knowledge base and embedding model, retrieve the relevant context, and finally route the request to the most appropriate LLM.

In this blog article, we are going to build this entire flow hands-on using MLflow Model Registry, MLflow AI Gateway, Qdrant, Chonkie, Sentence Transformers, Ollama, OpenAI, and Gemma. We will register domain-specific embedding models with a champion alias, ingest and index knowledge into Qdrant, classify incoming queries locally with Ollama, dynamically select the right embedding and collection, retrieve the relevant context, and route the final RAG request to the right LLM.

The result is a RAG application where both retrieval and generation adapt to the user’s intent, rather than depending on one fixed model for everything.

Setting up Mlflow in Raspberry pi 5:

For this experiment, I wanted to decouple the MLflow tracking server from my main development machine and run it on a Raspberry Pi instead, while keeping the actual metadata store on Postgres running on my Mac. The idea was simple on paper — point the Pi’s MLflow server at a Postgres instance over the local network as the backend store — but getting there meant untangling a few networking assumptions that Postgres makes by default. Out of the box, Postgres only listens on localhost and only trusts connections coming from the same machine, which meant the first few attempts to connect from the Pi were met with a flat connection refused, even though both machines could ping each other just fine. The fix involved opening up listen_addresses to accept connections on all interfaces and adding an explicit entry in pg_hba.conf to trust the Pi's IP with password authentication. One detail that cost me more time than it should have: on macOS, Homebrew installs versioned Postgres instances under paths like postgresql@15, and it's easy to end up editing a stale, unused config directory instead of the one the running service actually reads from. A quick SHOW config_file; inside psql settled that once and for all.

1. Locate the correct Postgres config directory on the Mac

If Postgres was installed via Homebrew, it’s likely versioned (e.g. postgresql@15). Confirm which config file the running instance actually uses before editing anything:

psql -U <your_user> -d postgres -c "SHOW config_file;"

This returns the real path, e.g. /opt/homebrew/var/postgresql@15/postgresql.conf. Always edit files in that directory — not a similarly named but unused one like /opt/homebrew/var/postgres/.

2. Allow Postgres to listen on the network, not just localhost

In postgresql.conf:

cd /opt/homebrew/var/postgresql@15
sed -i '' "s/#listen_addresses = 'localhost'/listen_addresses = '*'/" postgresql.conf

Verify:

grep -v "^#" postgresql.conf | grep listen_addresses
# listen_addresses = '*'

3. Whitelist the Pi’s IP in pg_hba.conf

echo "host  all  all  192.168.1.9/32  md5" >> pg_hba.conf

Verify

grep "192.168.1.9" pg_hba.conf

4. Restart Postgres and confirm it’s listening on all interfaces

brew services restart postgresql@15
lsof -i -P | grep LISTEN | grep postgres

You should see *:5432 in the output rather than localhost:5432. If it still shows localhost:5432, you're likely editing the wrong config directory (see Step 1).

5. Create/verify the database user and password

Confirm the role exists and has login privileges:

psql -U <admin_user> -d postgres -c "\du"

Set (or reset) the password for the user MLflow will connect as:

psql -U <admin_user> -d postgres -c "ALTER USER root WITH PASSWORD 'root';"

Confirm the target database exists, or create it:

psql -U <admin_user> -d postgres -c "\l" | grep mlflow
psql -U <admin_user> -d postgres -c "CREATE DATABASE mlflow OWNER root;"

6. Test raw connectivity from the Pi before touching MLflow

nc -zv 192.168.1.8 5432

Once the network layer was sorted and Postgres accepted authenticated connections from 192.168.1.9, the MLflow server on the Pi came up cleanly against the remote backend store.

mlflow server \
  --backend-store-uri postgresql://root:root@192.168.1.8:5432/mlflow \
  --host 0.0.0.0 \
  --port 5000 \
  --allowed-hosts "192.168.1.9:5000" \
  --cors-allowed-origins "http://192.168.1.9:5000"

The next wrinkle showed up not on the database side but in the browser — accessing the MLflow UI from the Pi's address triggered a wall of 403s in the server logs, all flagged as blocked cross-origin requests. This turned out to be a byproduct of MLflow's newer FastAPI-based server, which ships with security middleware from version 3.5.0 onward that explicitly validates allowed hosts and CORS origins rather than accepting requests from anywhere by default. Since the browser was pointed at the Pi's IP directly, that had to be declared as both an allowed host and an allowed CORS origin when starting the server. Once that was in place, validation was straightforward: the UI loaded without console errors, experiment runs and their metrics showed up in near real time, and a quick check against the mlflow database on the Mac confirmed that run metadata, parameters, and the registered medquad-embedder model with its champion alias were all persisting correctly to Postgres rather than to the default local SQLite file. That combination — a lightweight ARM device serving the tracking UI, with the durability of a proper relational backend sitting on a separate machine — turned out to be a genuinely low-overhead way to run MLflow for a home lab setup.

The RAG architecture

Architecture Overview

This architecture brings together MLflow, Qdrant, Ollama, embeddings, and multiple LLMs to build a RAG system that does more than just retrieve documents and send them to a language model. The idea is to manage the complete lifecycle, starting from embedding generation and model registration, all the way to vector search, query understanding, model selection, and response generation.

The architecture is divided into two logical stages. Stage 1 focuses on embeddings, knowledge indexing, and retrieval, while Stage 2 focuses on intelligent LLM routing and response generation.

Stage 1: Building and Managing the Knowledge Layer

The first important component is the embedding model. Instead of embedding documents using an unmanaged model hidden somewhere inside the application code, the embedding model is treated as a first-class component of the system.

The selected embedding model is registered in the MLflow Model Registry. This gives us a central place to manage which embedding model is being used by the application. This becomes particularly useful when we experiment with different embedding models or replace one model with another later.

The same registered embedding model is used for both sides of the retrieval process.

On the knowledge side, documents are converted into vector representations. These vectors capture the semantic meaning of the content rather than just the literal words appearing in the document. The generated dense embeddings, along with the corresponding document metadata, are stored inside Qdrant.

Qdrant becomes the retrieval layer of the architecture.

So instead of storing only something like:

"MLflow provides model management capabilities"

we store its semantic representation as a vector similar to:

[0.34, 0.198, -1.57, ...]

along with the original content and any useful metadata.

This indexing process happens ahead of the user’s query and creates the searchable knowledge base for the RAG application.

What Happens When the User Asks a Question?

When a user submits a question, that query goes through the same embedding model registered in MLflow.

For example, if the user asks:

How can I manage different versions of my ML models?

the query is converted into its own dense vector representation.

That query vector is then sent to Qdrant, where similarity search is performed against the vectors already stored in the collection.

The important point here is that retrieval is based on semantic similarity. Qdrant is not simply searching for documents containing the exact same words. It tries to identify pieces of knowledge whose meaning is close to the user’s question.

The most relevant pieces of information are returned as the search context.

At this point, the traditional RAG part of the architecture is complete.

We now have two important pieces of information:

the original user query and the retrieved knowledge context.

But instead of immediately sending both of them to a fixed LLM, the architecture introduces another layer.

This is where the system becomes more interesting.

Stage 2: Intelligent LLM Routing

Many RAG applications are designed around a single LLM. Every request goes to the same model regardless of whether the question is simple, complex, technical, or computationally expensive.

That approach works, but it is not always efficient.

A simple question may not require a large cloud-hosted model. At the same time, a complex reasoning request might benefit from a much stronger model.

The AI Gateway in this architecture acts as the decision layer between the RAG pipeline and the available language models.

Before routing the request, the user query is classified using a small local model running through Ollama. The classifier tries to understand what kind of query has been submitted.

Based on that classification, the gateway decides which configured LLM should handle the request.

For example, the architecture may have models such as GPT-4.1, a smaller Gemma model, and another locally hosted Gemma model available behind the gateway.

A relatively straightforward request could therefore be handled by a smaller local model, while a question requiring stronger reasoning could be routed to GPT-4.1.

The routing decision happens dynamically rather than being hardcoded into the application.

Bringing Retrieval and Routing Together

The retrieved context from Qdrant and the original user query are then passed to the AI Gateway.

Conceptually, the request reaching the selected LLM looks something like:

User Query + Retrieved Knowledge Context → Selected LLM

The selected model uses the retrieved knowledge as grounding while generating the answer.

This is an important distinction.

The LLM is not expected to answer purely from what it learned during training. The relevant information has already been retrieved from Qdrant and supplied as context, allowing the model to generate a response grounded in the application’s knowledge base.

Once the selected model generates the answer, the response travels back through the gateway and is returned to the user.

Why This Architecture Is Interesting

What makes this architecture different from a basic RAG implementation is that retrieval and generation are treated as separate, manageable layers.

MLflow manages the embedding model and provides the model management layer. Qdrant handles high-performance semantic retrieval. Ollama provides lightweight local intelligence for understanding the incoming query. The AI Gateway decides which LLM is most appropriate, while the final language model focuses on producing the answer.

The resulting flow is essentially:

User Query → Embedding → Qdrant Search → Relevant Context → Query Classification → LLM Routing → Grounded Response

This gives us a RAG architecture where we are not only asking, “What information should I retrieve?”

We are also asking:

“Which model is the right model to answer this particular question?”

That second question becomes increasingly important as applications start working with a mixture of local models, open-source models, and commercial LLM APIs.

Instead of tying the entire application to one model, this architecture creates a flexible RAG layer where knowledge retrieval, embedding management, and LLM selection can evolve independently.

The Implementation

model_registry.py — the setup step. It loads two sentence-transformer models (Arctic-embed for mortgage, MiniLM for leukemia), logs each one to MLflow on the Raspberry Pi, and pins a champion alias to the latest version. So downstream code never hardcodes a version, it just asks for the champion.

import mlflow
import mlflow.sentence_transformers
from mlflow import MlflowClient
from mlflow.models import infer_signature
from sentence_transformers import SentenceTransformer

mlflow.set_tracking_uri("http://192.168.1.9:5000")
mlflow.set_experiment("embedding-models")

EMBEDDERS = {
    "mortgage-embedder": {"source": "Snowflake/snowflake-arctic-embed-m-v1.5"},
    "leukemia-embedder": {"source": "all-MiniLM-L6-v2"},
}

sample_texts = ["MLflow tracks embedding models", "Qdrant stores vectors"]
client = MlflowClient()

for name, cfg in EMBEDDERS.items():
    model = SentenceTransformer(
        cfg["source"],
        trust_remote_code=cfg.get("trust_remote_code", False),
        config_kwargs=cfg.get("config_kwargs"),
        device=cfg.get("device"),
    )
    with mlflow.start_run(run_name=name):
        info = mlflow.sentence_transformers.log_model(
            model=model,
            name="embedder",
            signature=infer_signature(sample_texts, model.encode(sample_texts)),
            input_example=sample_texts,
            registered_model_name=name,
        )
    client.set_registered_model_alias(name, "champion", info.registered_model_version)
    print(f"{name}: v{info.registered_model_version}, dim={model.get_embedding_dimension()}")

ingest_data.py — the data loader. It wipes and rebuilds the leukemia_details collection in Qdrant, then uses a Chonkie pipeline to read PDFs from the parent folder, chunk them (512 tokens, 100 overlap), embed with MiniLM, and store the vectors. One chained call, no ceremony.

import os
from chonkie import Pipeline
from qdrant_client import QdrantClient
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

client = QdrantClient(url="http://localhost:6333", api_key=os.environ["QDRANT_API_KEY"])
client.delete_collection("leukemia_details") # this will delete and recreate the collection.

(Pipeline()
 .fetch_from("file", dir="../", ext=[".pdf"])
 .process_with("liteparse")
 .chunk_with("recursive", chunk_size=512)
 .refine_with("overlap", context_size=100)
 .store_in("qdrant", collection_name="leukemia_details", client=client,
           embedding_model="sentence-transformers/all-MiniLM-L6-v2")
 .run())

count = client.count("leukemia_details", exact=True).count
print(f"Ingested {count} chunks into leukemia_details")

ai_gateway.py — the traffic cop. Every query first goes to a local Ollama model with a strict prompt: “is this LEUKEMIA, MORTGAGE, or OTHER?” Leukemia questions get sent to OpenAI, mortgage questions to Gemma, and anything else gets a polite refusal. All of it flows through the MLflow gateway using the plain OpenAI client, so swapping backends is a config change, not a code change.

from openai import OpenAI
import mlflow

mlflow.set_tracking_uri("http://192.168.1.9:5000")

client = OpenAI(
    base_url="http://192.168.1.9:5000/gateway/mlflow/v1",
    api_key="no-api-key",
)

CLASSIFIER_PROMPT = (
    "Classify the user query into exactly one label: LEUKEMIA, MORTGAGE, or OTHER.\n"
    "LEUKEMIA = questions about leukemia (AML, CML, ALL, CLL), its symptoms, diagnosis, "
    "treatment, prognosis, or related blood cancers.\n"
    "MORTGAGE = questions about home loans, mortgage rates, refinancing, EMI, "
    "eligibility, lenders, or mortgage documents.\n"
    "OTHER = everything else, including general medical questions not about leukemia.\n"
    "Reply with only the label."
)

ROUTES = {
    "LEUKEMIA": "openai",
    "MORTGAGE": "gemma",
}

REJECT_MESSAGE = (
    "Sorry, this setup can only answer Leukemia-specific medical questions "
    "and mortgage-related questions."
)


def classify(query: str) -> str:
    resp = client.chat.completions.create(
        model="ollama",  # local Ollama endpoint does the classification
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": query},
        ],
        temperature=0,
    )
    label = resp.choices[0].message.content.strip().upper()
    for intent in ROUTES:
        if intent in label:
            return intent
    return "OTHER"


def route(query: str) -> str:
    intent = classify(query)
    target = ROUTES.get(intent)
    if target is None:
        print(f"[intent={intent} -> rejected]")
        return REJECT_MESSAGE
    resp = client.chat.completions.create(
        model=target,
        messages=[{"role": "user", "content": query}],
    )
    print(f"[intent={intent} -> model={target}]")
    return resp.choices[0].message.content

#
# if __name__ == "__main__":
#     print(route("What are the first-line treatments for acute myeloid leukemia?"))
#     print(route("How does refinancing affect my monthly mortgage EMI?"))
#     print(route("What are the side effects of metformin?"))   # medical but not leukemia -> rejected
#     print(route("Write a haiku about monsoon in Hyderabad."))  # rejected

smart_rag.py — the actual chatbot. It reuses the classifier, then picks the matching Qdrant collection and embedder for that intent, pulls the top 5 chunks, stuffs them into a system prompt (“answer using only this context”), and sends it to the model the gateway routed to. Wrapped in a simple you> / assistant> loop.

from qdrant_client import QdrantClient
import os

from ai_gateway import client, classify, ROUTES, REJECT_MESSAGE
from embedding_model_inference import EMBEDDERS
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

COLLECTIONS = {
    "LEUKEMIA": {"collection": "leukemia_details", "embedder": "leukemia-embedder"},
    "MORTGAGE": {"collection": "mortgage", "embedder": "mortgage-embedder"},
}
TEXT_FIELD = "text"
EXIT_WORDS = {"bye", "exit", "quit"}


class SmartRAG:
    def __init__(self, qdrant_url="http://localhost:6333", top_k=5):
        self.qdrant = QdrantClient(url=qdrant_url, api_key=os.environ.get("QDRANT_API_KEY"))
        self.top_k = top_k

    def retrieve(self, query, intent):
        cfg = COLLECTIONS[intent]
        embedder = EMBEDDERS[cfg["embedder"]]
        prompt = "query" if "query" in embedder.prompts else None  # arctic needs its query prefix
        vec = embedder.encode(query, prompt_name=prompt, normalize_embeddings=True).tolist()
        hits = self.qdrant.query_points(cfg["collection"], query=vec, limit=self.top_k, with_payload=True).points
        return "\n\n".join(str(h.payload.get(TEXT_FIELD, h.payload)) for h in hits)

    def answer(self, query):
        intent = classify(query)
        if intent not in ROUTES:
            return REJECT_MESSAGE
        context = self.retrieve(query, intent)
        resp = client.chat.completions.create(
            model=ROUTES[intent],
            messages=[
                {"role": "system", "content": f"Answer using only this context:\n{context}"},
                {"role": "user", "content": query},
            ],
        )
        print(f"[intent={intent} -> {COLLECTIONS[intent]['collection']} -> {ROUTES[intent]}]")
        return resp.choices[0].message.content

    def run(self):
        while True:
            query = input("\nyou> ").strip()
            if query.lower() in EXIT_WORDS:
                break
            if query:
                print("\nassistant>", self.answer(query))


if __name__ == "__main__":
    SmartRAG().run()

embedding_model_inference.py pulls both embedders back out of MLflow by alias (models:/leukemia-embedder@champion, same for mortgage) and hands them over as live SentenceTransformer objects in the EMBEDDERS dict. That's exactly what smart_rag.py imports, so the loop is now closed: registry writes the champion, this file reads it.

import mlflow
import mlflow.sentence_transformers

mlflow.set_tracking_uri("http://192.168.1.9:5000")

EMBEDDERS = {
    name: mlflow.sentence_transformers.load_model(f"models:/{name}@champion")
    for name in ("leukemia-embedder", "mortgage-embedder")
}

QUERIES = {
    "leukemia-embedder": "What are the first-line treatments for acute myeloid leukemia?",
    "mortgage-embedder": "How does refinancing affect my monthly mortgage EMI?",
}

for name, embedder in EMBEDDERS.items():
    vecs = embedder.encode([QUERIES[name]], normalize_embeddings=True)
    print(f"{name}: shape={vecs.shape}, prompts={list(embedder.prompts) or None}")

The result

/Users/pavanmantha/.local/bin/uv run /Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/mlflow-tutorials/mlflow_on_edge/.venv/bin/python /Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/mlflow-tutorials/mlflow_on_edge/src/smart_rag.py 
Downloading artifacts: 100%|██████████| 16/16 [00:19<00:00,  1.19s/it]
2026/09/06 12:48:35 INFO mlflow.sentence_transformers: 'models:/leukemia-embedder@champion' resolved as 'mlflow-artifacts:/3/models/m-f568ee233dd748f997ba66763e41653b/artifacts'
Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00, 76.69it/s]
Loading weights: 100%|██████████| 103/103 [00:00<00:00, 15587.15it/s]
Downloading artifacts: 100%|██████████| 16/16 [01:29<00:00,  5.60s/it]
2026/09/06 12:50:06 INFO mlflow.sentence_transformers: 'models:/mortgage-embedder@champion' resolved as 'mlflow-artifacts:/3/models/m-c4221c844dfb45bfa15c6a4ecde56920/artifacts'
Downloading artifacts: 100%|██████████| 1/1 [00:00<00:00, 27.17it/s]
Loading weights: 100%|██████████| 197/197 [00:00<00:00, 8855.57it/s]
[transformers] BertModel LOAD REPORT from: /var/folders/1x/36nz44_569501n4xs0px2_8m0000gn/T/tmpef4y6tj5/model.sentence_transformer
Key                 | Status  | 
--------------------+---------+-
pooler.dense.weight | MISSING | 
pooler.dense.bias   | MISSING | 

Notes:
- MISSING: those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
leukemia-embedder: shape=(1, 384), prompts=['query', 'document']
mortgage-embedder: shape=(1, 768), prompts=['query', 'document']
/Users/pavanmantha/Pavans/PracticeExamples/DataScience_Practice/mlflow-tutorials/mlflow_on_edge/src/smart_rag.py:20: UserWarning: Api key is used with an insecure connection.
  self.qdrant = QdrantClient(url=qdrant_url, api_key=os.environ.get("QDRANT_API_KEY"))

you> What are the phases of treatment for acute lymphoblastic leukemia?
[intent=LEUKEMIA -> leukemia_details -> openai]

assistant> The phases of treatment for acute lymphoblastic leukemia (ALL) are:

1. Induction - achieving remission.
2. Consolidation - eradicating residual disease with different agents to circumvent resistance.
3. Maintenance (in ALL) - prolonged low-dose therapy to suppress re-emergence of the clone.

you> What was 30-day Average SOFR referred to in Series issued before July 2023?
[intent=MORTGAGE -> mortgage -> gemma]

assistant> *   Question: "What was 30-day Average SOFR referred to in Series issued before July 2023?"
    *   Constraint: Use *only* the provided context.

    *   Chunk 1: Discusses "Excess inclusion income", Residual Holders, taxable income, Trust REMIC taxable income, etc. No mention of SOFR or July 2023.
    *   Chunk 2: Discusses "Backup withholding", Non-U.S. Persons, Ginnie Mae Guaranty, State/Local/Foreign tax. No mention of SOFR or July 2023.
    *   Chunk 3: Discusses "Residual Holder's share of expenses", REITs, RICs, phantom income. No mention of SOFR or July 2023.
    *   Chunk 4: Discusses "Principal Distribution Amount", Ginnie Mae Project Loan Certificates, Underlying Certificates. No mention of SOFR or July 2023.
    *   Chunk 5: Discusses "OID on a Teaser Security", "Initial Distribution Period", "Excess Interest Payment", "Accrual of Original Issue Discount". No mention of SOFR or July 2023.

    *   I have scanned all five provided text nodes.

you> 

The Conclusion

Today we have seen how to build a multi-domain, intelligently routed RAG application where the embedding model, vector collection, and final LLM are selected based on the user’s query.

We used MLflow Model Registry to manage domain-specific embedding models through a champion alias, Chonkie and Qdrant to build the knowledge layer, Ollama to classify queries locally, and MLflow AI Gateway to route requests across different LLMs such as OpenAI and Gemma. Finally, we connected everything inside the RAG application so that each query automatically follows the most appropriate retrieval and generation path.

The important takeaway is that a RAG system does not have to be tied to one embedder, one collection, or one LLM. By separating these responsibilities and making the routing decision dynamically, we can build RAG applications that are much more flexible and easier to evolve as new models and domains are introduced.

RAG Beyond Retrieval: Model Registry, Vector Search and Intelligent LLM Routing was originally published in GoPenAI on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is a completely hands-on journey, and the article is paired with a YouTube demo so you can follow the entire flow in action. By the end, the MCP Registry will feel less like a new feature and more like something you can immediately start using in your own agentic AI projects.

The MLflow Platform

MLflow is an open-source platform for managing the lifecycle of machine learning and Generative AI applications. It helps teams track experiments, manage models, evaluate outputs, monitor traces, and compare different versions of an application.

For GenAI development, MLflow is useful when working with LLMs, prompts, RAG pipelines, agents, tools, and MCP servers. It gives developers a central place to understand application behavior, measure quality, troubleshoot failures, and manage changes as the system evolves.

MLflow also offers a growing set of features such as Experiment Tracking, Model Registry, Prompt Management, GenAI Evaluation, Tracing and Observability, AI Gateway capabilities, and MCP Registry support.

In simple terms, MLflow provides the tracking, evaluation, observability, and lifecycle management layer needed to move GenAI applications from experimentation to production

MCP Registry and its Features

As the number of MCP servers in an organization starts growing, managing them through individual configuration files quickly becomes difficult. Which servers are available? Which version should an application use? What tools does a server expose? Where is the endpoint? How should a client connect to it? MLflow MCP Registry is designed to solve exactly this problem.

Introduced in MLflow 3.15.0, the MCP Registry provides a centralized catalog for registering, versioning, discovering, and managing MCP servers. Instead of every developer or AI agent maintaining its own collection of MCP configurations, the registry becomes a common place where MCP servers can be discovered and accessed.

The diagram above represents this idea very simply. We may have Server 1, Server 2, Server 3, and many more MCP servers, but they can all be registered with MLflow. Once registered, MLflow can maintain their metadata, semantic versions and aliases such as production or staging, discover the tools exposed by those servers, and maintain the endpoints required to connect to them.

An important thing to understand is that MLflow MCP Registry is not where your MCP server actually runs. The MCP server can continue running locally, in the cloud, or on another remote infrastructure. MLflow maintains the information required to discover, manage, version, and connect to that server. An access endpoint connects a registered server version or alias with the actual running MCP endpoint.

This is where the registry becomes useful for clients and AI agents. Instead of manually exchanging MCP configuration snippets, developers can register servers through server.json, the MLflow UI, or programmatically, discover their available tools, manage updates, and obtain ready-to-use connection configurations. In simple terms, MLflow MCP Registry becomes the single place between your growing collection of MCP servers and the clients or agents that want to use them.

Note: One thing to keep in mind while experimenting with it is that the MCP Registry is currently marked as an experimental MLflow feature, so its APIs and behavior may evolve in future releases.

The Experiment

As a first step start the mlflow server and back it with some persistant storage in my case its postgresql.

mlflow server --backend-store-uri postgresql://root:root@localhost:5432/mlflow --host 127.0.0.1 --port 5000 

We shall deal with 3 MCP servers

1. Simple MCP Server (custom built)
2. Complex Postgres MCP Server (custom built)
3. A standard Qdrant server provided by Qdrant.tech

Simple Server:

FastMCP("My MCP Server") creates the MCP server instance. The @mcp.tool(...) decorators then expose normal Python functions as MCP tools. In this example, the server exposes two tools: Greeting and Sendoff.

The greet() function accepts a name and returns "Hello, <name>!", while the bye() function returns "Bye, <name>!". Because these functions are decorated with @mcp.tool, an MCP client can discover and invoke them remotely.

Finally, mcp.run(transport="http", port=8000) starts the MCP server using HTTP transport on port 8000. Once it is running, an MCP client or later the MLflow MCP Registry can connect to this server and discover the tools it exposes.

from fastmcp import FastMCP

mcp = FastMCP("My MCP Server")

@mcp.tool(
    name="Greeting",
    description="Greets the incoming message",
    title="Greeting",
)
def greet(name: str) -> str:
    return f"Hello, {name}!"

@mcp.tool(
    name="Sendoff",
    description="Sendoff the incoming message",
    title="Sendoff",
)
def bye(name: str) -> str:
    return f"Bye, {name}!"


if __name__ == "__main__":
    mcp.run(transport="http", port=8000)

PostgreSQL Explorer MCP Server connects to a PostgreSQL instance and exposes database exploration capabilities as MCP tools. The PostgreSQL connection details such as host, port, username, password, SSL mode, and admin database are loaded from environment variables.

The server uses two helper functions, _connect() and _query(), to connect to PostgreSQL, execute read-only SQL queries, return the results as dictionaries, and close the connection after execution. This keeps the actual MCP tools simple and focused.

The important part is the set of MCP tools exposed by the server. ListDatabases discovers available PostgreSQL databases, ListSchemas finds schemas inside a database, ListTables discovers tables and views, and ListTableColumns provides column-level metadata such as datatype, nullability, and default values.

It goes one step further with relationship discovery. ListTableRelations identifies incoming and outgoing foreign-key relationships for an individual table, while ListSchemaRelations discovers relationships across the entire schema. This can even be used to construct a database relationship graph or ER diagram.

Finally, the server runs using HTTP transport on port 8000, exactly like our simple MCP example.

"""
PostgreSQL Explorer MCP Server
================================

Exposes tools to browse a PostgreSQL server's structure:
  - list databases
  - list schemas within a database
  - list tables within a schema
  - list foreign-key relations for a single table
  - list all foreign-key relations (a relation graph) within a schema

Run (same pattern as the sample server):
    fastmcp run postgres_server.py --transport http --port 8000
    # or simply: python postgres_server.py
"""

import os
from typing import Any
from dotenv import load_dotenv, find_dotenv
import psycopg2
import psycopg2.extras
from fastmcp import FastMCP

load_dotenv(find_dotenv())

mcp = FastMCP("PostgreSQL Explorer")

PGHOST = os.environ.get("PGHOST", "localhost")
PGPORT = int(os.environ.get("PGPORT", "5432"))
PGUSER = os.environ.get("PGUSER", "root")
PGPASSWORD = os.environ.get("PGPASSWORD", "root")
PGSSLMODE = os.environ.get("PGSSLMODE", "prefer")
PGADMINDB = os.environ.get("PGADMINDB", "postgres")

SYSTEM_SCHEMAS = ("pg_catalog", "information_schema", "pg_toast")


def _connect(database: str):
    """Open a connection to a specific database on the configured server."""
    return psycopg2.connect(
        host=PGHOST,
        port=PGPORT,
        user=PGUSER,
        password=PGPASSWORD,
        dbname=database,
        sslmode=PGSSLMODE,
    )


def _query(database: str, sql: str, params: tuple = ()) -> list[dict[str, Any]]:
    """Run a read-only query against `database` and return rows as dicts."""
    conn = _connect(database)
    try:
        with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
            cur.execute(sql, params)
            return [dict(row) for row in cur.fetchall()]
    finally:
        conn.close()


@mcp.tool(
    name="ListDatabases",
    description="Lists all non-template databases on the configured PostgreSQL server, with owner and size.",
    title="List Databases",
)
def list_databases() -> list[dict[str, Any]]:
    sql = """
          SELECT
              d.datname AS database,
            pg_get_userbyid(d.datdba) AS owner,
            pg_size_pretty(pg_database_size(d.datname)) AS size
          FROM pg_database d
          WHERE d.datistemplate = false
          ORDER BY d.datname; \
          """
    return _query(PGADMINDB, sql)


@mcp.tool(
    name="ListSchemas",
    description="Lists all user-defined schemas within a given database (excludes system schemas).",
    title="List Schemas",
)
def list_schemas(database: str) -> list[dict[str, Any]]:
    sql = """
          SELECT
              schema_name AS schema,
            schema_owner AS owner
          FROM information_schema.schemata
          WHERE schema_name NOT IN %s
            AND schema_name NOT LIKE 'pg_temp%%'
            AND schema_name NOT LIKE 'pg_toast_temp%%'
          ORDER BY schema_name; \
          """
    return _query(database, sql, (SYSTEM_SCHEMAS,))


@mcp.tool(
    name="ListTables",
    description="Lists all tables (and views) within a given schema of a given database.",
    title="List Tables",
)
def list_tables(database: str, schema: str) -> list[dict[str, Any]]:
    sql = """
          SELECT
              table_name AS table,
            table_type AS type
          FROM information_schema.tables
          WHERE table_schema = %s
          ORDER BY table_name; \
          """
    return _query(database, sql, (schema,))


@mcp.tool(
    name="ListTableColumns",
    description="Lists columns, data types, nullability, and defaults for a given table.",
    title="List Table Columns",
)
def list_table_columns(database: str, schema: str, table: str) -> list[dict[str, Any]]:
    sql = """
          SELECT
              column_name AS column,
            data_type AS type,
            is_nullable AS nullable,
            column_default AS default
          FROM information_schema.columns
          WHERE table_schema = %s AND table_name = %s
          ORDER BY ordinal_position; \
          """
    return _query(database, sql, (schema, table))


@mcp.tool(
    name="ListTableRelations",
    description=(
            "Lists foreign-key relations for a single table: relations where this table "
            "references another table (outgoing), and relations where other tables "
            "reference this table (incoming)."
    ),
    title="List Table Relations",
)
def list_table_relations(database: str, schema: str, table: str) -> dict[str, Any]:
    outgoing_sql = """
                   SELECT
                       tc.constraint_name,
                       kcu.column_name AS local_column,
                       ccu.table_schema AS foreign_schema,
                       ccu.table_name AS foreign_table,
                       ccu.column_name AS foreign_column
                   FROM information_schema.table_constraints tc
                            JOIN information_schema.key_column_usage kcu
                                 ON tc.constraint_name = kcu.constraint_name AND tc.table_schema = kcu.table_schema
                            JOIN information_schema.constraint_column_usage ccu
                                 ON tc.constraint_name = ccu.constraint_name AND tc.table_schema = ccu.table_schema
                   WHERE tc.constraint_type = 'FOREIGN KEY'
                     AND tc.table_schema = %s
                     AND tc.table_name = %s
                   ORDER BY tc.constraint_name; \
                   """
    incoming_sql = """
                   SELECT
                       tc.constraint_name,
                       tc.table_schema AS referencing_schema,
                       tc.table_name AS referencing_table,
                       kcu.column_name AS referencing_column,
                       ccu.column_name AS local_column
                   FROM information_schema.table_constraints tc
                            JOIN information_schema.key_column_usage kcu
                                 ON tc.constraint_name = kcu.constraint_name AND tc.table_schema = kcu.table_schema
                            JOIN information_schema.constraint_column_usage ccu
                                 ON tc.constraint_name = ccu.constraint_name AND tc.table_schema = ccu.table_schema
                   WHERE tc.constraint_type = 'FOREIGN KEY'
                     AND ccu.table_schema = %s
                     AND ccu.table_name = %s
                   ORDER BY tc.constraint_name; \
                   """
    return {
        "table": table,
        "schema": schema,
        "outgoing": _query(database, outgoing_sql, (schema, table)),
        "incoming": _query(database, incoming_sql, (schema, table)),
    }


@mcp.tool(
    name="ListSchemaRelations",
    description=(
            "Lists all foreign-key relations across every table in a schema, as a flat "
            "edge list (from_table.from_column -> to_table.to_column). Useful for "
            "building a full relationship graph / ERD of the schema."
    ),
    title="List Schema Relations",
)
def list_schema_relations(database: str, schema: str) -> list[dict[str, Any]]:
    sql = """
          SELECT
              tc.constraint_name,
              tc.table_name AS from_table,
              kcu.column_name AS from_column,
              ccu.table_name AS to_table,
              ccu.column_name AS to_column
          FROM information_schema.table_constraints tc
                   JOIN information_schema.key_column_usage kcu
                        ON tc.constraint_name = kcu.constraint_name AND tc.table_schema = kcu.table_schema
                   JOIN information_schema.constraint_column_usage ccu
                        ON tc.constraint_name = ccu.constraint_name AND tc.table_schema = ccu.table_schema
          WHERE tc.constraint_type = 'FOREIGN KEY'
            AND tc.table_schema = %s
          ORDER BY tc.table_name, tc.constraint_name; \
          """
    return _query(database, sql, (schema,))


if __name__ == "__main__":
    mcp.run(transport="http", port=8000)

Testing these MCP servers in a standalone mode using FastMCP client is as below. before the client is run we should run the MCP server and it should be up and running.

import asyncio
from typing import Any
import json

from fastmcp import Client

client = Client("http://localhost:8000/mcp")

async def call_tool(tool_name: str):
    async with client:
        tool_result: Any = await client.call_tool(name=tool_name)
        for tool in tool_result.content:
            for obj in json.loads(tool.text):
                print(obj['database'])


asyncio.run(call_tool("ListDatabases"))

Registering MCP Servers:

There are two simple ways to register an MCP server in MLflow MCP Registry: through the UI or programmatically.

Using the MLflow UI, you open the MCP Registry, choose to create a new MCP server, provide the server details such as name, version, description, and remote MCP endpoint, and then register it. This is the easiest option when you want to quickly add and inspect a server manually.

Using the programmatic approach, you can register the MCP server directly from Python using MLflow APIs. This is more suitable for automation, CI/CD pipelines, or environments where MCP servers are created and updated frequently.

import asyncio
from typing import Any

import mlflow

# Point at the same MLflow server your browser UI is connected to.
# Check the URL in your browser's address bar — commonly http://127.0.0.1:5000
mlflow.set_tracking_uri("http://127.0.0.1:5000")

async def register_postgresql_mcp_server():
    version = mlflow.genai.register_mcp_server(
        server_json={
            "name": "io.github.pavanjava/postgresql-server",
            "version": "0.1.0",
            "description": "PostgreSQL FastMCP server exposing DB tools",
            "remotes": [
                {"url": "http://localhost:8000/mcp", "type": "streamable-http"},
            ],
        },
        status="active",
        source="local dev server via fastmcp",
        create_access_endpoints_from_remotes=True,
    )
    print(f"Registered '{version.name}' version {version.version}")

async def register_qdrant_mcp_server():
    version = mlflow.genai.register_mcp_server(
        server_json={
            "name": "io.github.qdrant/mcp-server-qdrant",
            "version": "0.1.0",
            "description": "Qdrant MCP server exposing vector search tools",
            "packages": [
                {
                    "registryType": "pypi",
                    "identifier": "mcp-server-qdrant",
                    "version": "0.8.1",
                    "runtimeHint": "uvx",
                    "transport": {"type": "stdio"},
                    "environmentVariables": [
                        {"name": "QDRANT_URL", "description": "Qdrant endpoint URL", "isRequired": True, "isSecret": False},
                        {"name": "QDRANT_API_KEY", "description": "Qdrant API key", "isRequired": True, "isSecret": True},
                        {"name": "COLLECTION_NAME", "description": "Collection name", "default": "code"},
                        {"name": "EMBEDDING_MODEL", "description": "Embedding model", "default": "BAAI/bge-small-en-v1.5"},
                    ],
                }
            ],
        },
        status="active",
        source="local dev server via uvx mcp-server-qdrant",
    )

    print(f"Registered '{version.name}' version {version.version}")


if __name__ == "__main__":
    # asyncio.run(register_mcp())
    asyncio.run(register_qdrant_mcp_server())

Once registered, MLflow can maintain the server metadata, versions and aliases, discover the tools exposed by the MCP server, and provide the connection information required by clients and agents.

Now lets access these registered MCP servers using the client to see what they have programatically.

import asyncio
from typing import Any

import mlflow

# Point at the same MLflow server your browser UI is connected to.
# Check the URL in your browser's address bar — commonly http://127.0.0.1:5000
mlflow.set_tracking_uri("http://127.0.0.1:5000")


async def discover_tools():
    server_version: Any = mlflow.genai.refresh_mcp_server_version_tools(
        name="io.github.pavanjava/postgresql-server",
        version="0.1.0",
    )
    for tool in server_version.tools:
        print(tool.name)

async def list_endpoints():
    # List all endpoints for a server
    endpoints = mlflow.genai.search_mcp_access_endpoints(
        server_name="io.github.pavanjava/postgresql-server",
    )

    for ep in endpoints:
        resolved = ep.resolved_version.version if ep.resolved_version else None
        print(f"  {ep.url} ({ep.transport_type}) → version {resolved}")

if __name__ == "__main__":
    asyncio.run(discover_tools())
    # asyncio.run(list_endpoints())

The Conclusion:

In this hands-on session, we went beyond just understanding the MLflow MCP Registry. We built custom MCP servers, registered both a standard Qdrant MCP server and our own custom MCP servers, and explored how to manage them through the registry.

We also saw how to register servers using both the MLflow UI and programmatically, access them from MCP clients, and finally integrate the registered MCP server with Claude Code to validate the complete flow end to end.

Overall, the MLflow MCP Registry gives us a practical way to organize, discover, manage, and consume MCP servers as our agentic AI ecosystem grows.

MLflow MCP Registry in Action: An End-to-End Hands-On Guide was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

We will walk through the complete architecture, starting from speech input and speech-to-text processing, moving through semantic retrieval and RAG-based context generation, and finally converting the response back into speech. We will also see how LiveKit enables real-time voice interaction, LlamaIndex manages the knowledge ingestion and retrieval workflow, Qdrant powers semantic search, and MLflow helps trace and observe the agent’s execution.

The goal is to move beyond a simple voice chatbot and build a production-oriented, observable Voice RAG system where every interaction, retrieval step, LLM call, and response can be traced and understood.

The Architecture

The idea behind this project is simple. A user should be able to speak naturally, ask a mortgage-related question, and get a useful answer back in voice. At the same time, the answer should come from trusted mortgage documents rather than only from the LLM’s own knowledge.

The flow starts with the mortgage guideline documents. These documents are processed using LlamaIndex and stored in Qdrant Vector Search Engine in a form that makes semantic search possible. This means the system can find information based on the meaning of the user’s question, not just on exact keyword matches.

When the user starts speaking, LiveKit manages the voice session. Silero VAD detects when the user is speaking, and Cartesia Speech-to-Text converts the speech into text. That text is then passed to the voice agent.

The voice agent uses GPT-4o-mini as its main language model. The agent is instructed to answer only mortgage-related questions and to use the retrieval tool whenever it needs information. This is important because the agent should not simply generate an answer from memory. It should first look for supporting information from the mortgage knowledge base.

For retrieval, the agent calls a tool named query_info. This tool uses LlamaIndex to search the mortgage collection in Qdrant Vector Search Engine. The most relevant pieces of information are returned to the agent as context.

The LLM then uses this retrieved context to prepare a short and conversational answer. Since this is a voice application, the response should sound natural when spoken and should not be unnecessarily long.

The generated text is then passed to Cartesia Text-to-Speech, which converts the answer back into audio. LiveKit sends this audio back to the user, completing the full speech-to-speech interaction.

Alongside this entire flow, MLflow tracing is connected through OpenTelemetry. This gives us visibility into what is happening inside the agent while a conversation is running. Instead of treating the voice agent as a black box, we can trace the agent execution and understand how a request moves through the system.

So, at a high level, the flow is User Speech → Speech-to-Text → Voice Agent → Retrieval → Qdrant Search → Retrieved Context → LLM Response → Text-to-Speech → User. MLflow sits alongside this flow and provides the tracing needed to observe the agent.

The Implementation

The project structure is something as below

.
├── __init__.py
├── .env
├── agent_with_langfuse_observability.py
└── agent_with_mlflow_observability.py

The sample .env as below, The implementation starts by loading the required environment variables using load_dotenv(). These environment variables hold the configuration needed by services such as LiveKit, OpenAI, Cartesia, Qdrant, and OpenTelemetry.

# Cartesia (STT + TTS)
CARTESIA_API_KEY=

# OpenAI
OPENAI_API_KEY=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

# LiveKit local dev server (default dev credentials)
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret

# langfuse
LANGFUSE_SECRET_KEY=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_HOST=http://172.20.10.8:3000

# mlflow
OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:5000
OTEL_EXPORTER_OTLP_HEADERS=x-mlflow-experiment-id=0
OTEL_SERVICE_NAME=livekit-voice-agent

import os
import logging

from livekit.agents import JobContext, JobProcess, WorkerOptions, cli, function_tool
from livekit.agents.voice import Agent, AgentSession
from livekit.agents.telemetry import set_tracer_provider
from livekit.plugins import openai, silero, cartesia

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

from dotenv import load_dotenv, find_dotenv

from llama_index.core import VectorStoreIndex, Settings, SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from qdrant_client import QdrantClient, AsyncQdrantClient

load_dotenv(find_dotenv())

Next, LlamaIndex is configured with embeddinggemma:latest as the embedding model. This model is used when a user query needs to be represented as an embedding for semantic retrieval. The code also configures gemma3:latest as the default LlamaIndex LLM, while the actual voice conversation is handled separately by GPT-4o-mini inside the LiveKit agent.

The code then creates both a QdrantClient and an AsyncQdrantClient. These clients connect the application to the locally running Qdrant Vector Search Engine. The mortgage collection is then connected to LlamaIndex through QdrantVectorStore.

After this connection is created, VectorStoreIndex.from_vector_store() gives LlamaIndex access to the already existing mortgage collection. At this stage, the code is not loading or ingesting the documents again. It is simply connecting to the knowledge that has already been stored in Qdrant.

# configs
Settings.embed_model = OllamaEmbedding(model_name="embeddinggemma:latest")
Settings.llm = Ollama(model="gemma3:latest")

# creates a persistant index to disk
client = QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y")
aclient = AsyncQdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y")

# create our vector store with hybrid indexing enabled
vector_store = QdrantVectorStore(
    "mortgage",
    client=client,
    aclient=aclient
)

# create the index reference to vector store
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("voice-agent")

The query_info() function is the main retrieval function in the project. It is decorated with @function_tool(), which means the voice agent can call it as a tool. When the agent receives a mortgage question, this function creates a retriever, searches for the top relevant results, collects the text from those results, and sends that information back to the agent.

@function_tool()
async def query_info(query: str) -> str:
    """Get more information about a specific topic"""
    print(f"query: {query}")
    retriever = index.as_retriever(similarity_top_k=20)
    nodes_with_scores = retriever.retrieve(query)
    context = ""
    for node in nodes_with_scores:
        print(f"Query result: {node.text}")
        context = context + node.text + ". \n"
    return context

The configure_mlflow_tracing() function is responsible for tracing. It checks whether an OpenTelemetry endpoint has been configured. If it is available, the function creates a tracer provider and connects it to an OTLP exporter. The same tracer is also registered with LiveKit. This is what allows agent activity to be sent to MLflow for tracing and inspection.

def configure_mlflow_tracing():
    """Configure OpenTelemetry to send traces to MLflow."""
    if not os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"):
        logger.warning("OTEL_EXPORTER_OTLP_ENDPOINT not set, tracing disabled")
        return None

    service_name = os.getenv("OTEL_SERVICE_NAME", "livekit-voice-agent")
    resource = Resource.create({SERVICE_NAME: service_name})

    provider = TracerProvider(resource=resource)
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))

    # Set both global and LiveKit tracer provider
    trace.set_tracer_provider(provider)
    set_tracer_provider(provider)

    logger.info("MLflow tracing configured successfully!")
    return provider

The entrypoint() function is where the actual voice agent is created. It first connects to the LiveKit room. It then creates an Agent with a set of instructions that define how the agent should behave. These instructions tell the agent to stay within the mortgage domain, keep answers short, use the retrieval tool, and clearly say when the required information is not available.

async def entrypoint(ctx: JobContext) -> None:
    """Main entrypoint for the agent."""
    logger.info(f"Agent starting for room: {ctx.room.name}")

    # Connect to the room
    await ctx.connect()

    # Create the voice agent with all components
    agent = Agent(
        instructions=(
            "You are a helpful mortgage voice assistant. Your interface "
            "with users will be voice. You should use short and concise "
            "responses, and avoiding usage of unpronouncable punctuation."
            "Always use your tools to get the realtime information."
            "If you dont find any information politely respond the same to user."
            "Never answer anything other than mortgage subject and reject politely."
            "Keep your responses concise and conversational since you're speaking out loud."
            "Be friendly and helpful."
        ),
        stt=cartesia.STT(), # Speech-to-Text
        llm=openai.LLM(model="gpt-4o-mini"), # Language Model
        # voice-id: 9626c31c-bec5-4cca-baa8-f8ba9e84c8bc, f786b574-daa5-4673-aa0c-cbe3e8534c02, 630ed21c-2c5c-41cf-9d82-10a7fd668370
        tts=cartesia.TTS(model="sonic-3", voice="630ed21c-2c5c-41cf-9d82-10a7fd668370"), # Text-to-Speech
        vad=silero.VAD.load(),
        tools=[query_info]
    )

    # Create and start the agent session
    session = AgentSession()
    await session.start(agent, room=ctx.room)

    logger.info("Agent session started! Ready for conversation.")

Inside the same agent configuration, Cartesia STT is used for speech-to-text, GPT-4o-mini is used for reasoning and response generation, Cartesia Sonic-3 is used for text-to-speech, and Silero VAD is used to detect speech activity. The query_info function is also attached as a tool so that the LLM can access the mortgage knowledge base when needed.

The AgentSession() then starts the agent inside the LiveKit room. Once this session starts, the agent is ready to listen to the user, understand the question, retrieve information, generate an answer, and speak the response back.

The prewarm() function runs before the actual conversation starts. It configures the tracing and loads the Silero VAD model ahead of time. Preloading the VAD model helps avoid unnecessary startup delay when the first user joins the session.

def prewarm(proc: JobProcess) -> None:
    """Prewarm function to load models before handling requests."""
    # Configure tracing before anything else
    configure_mlflow_tracing()

    # Preload Silero VAD model for faster startup
    proc.userdata["vad"] = silero.VAD.load()
    logger.info("Prewarmed VAD model")

Finally, cli.run_app() starts the LiveKit worker. It tells LiveKit which function should start the agent and which function should run during prewarming.

if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        )
    )

Overall, the implementation keeps each part of the system focused on one job. LiveKit manages the real-time conversation, Cartesia handles speech, GPT-4o-mini handles reasoning, LlamaIndex handles retrieval logic, Qdrant Vector Search Engine performs semantic search, and MLflow provides traceability.

How to run?

To run the Voice RAG agent locally, first install the LiveKit Server and LiveKit CLI using Homebrew. Start the local LiveKit server with

livekit-server --dev

and keep that terminal running. In your project .env file, configure

LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret

Make sure the other required services, such as Qdrant, Ollama, and the credentials needed for OpenAI, Cartesia, and MLflow tracing, are also available. Then open another terminal in the project directory and start the agent in console mode using

python voice_agent.py console

or the corresponding agent filename in your project. The agent will then use your system’s microphone and speakers directly, allowing you to speak to the Voice RAG agent and receive spoken responses.

The Experiment Results

The below run log shows how the Agent was able to connect to Qdrant Search engine to fetch the related context and metedata like page numbers.

╭─    ~/Pavans/Workshops/s2s_agents/livekit_observability  on   main !3 ?84 ···························································· ✔  s2s_agents   base   at 08:15:41 AM  
╰─ python agent_with_mlflow_observability.py console
/Users/pavanmantha/Pavans/Workshops/s2s_agents/livekit_observability/agent_with_mlflow_observability.py:30: UserWarning: Api key is used with an insecure connection.
  client = QdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y")
/Users/pavanmantha/Pavans/Workshops/s2s_agents/livekit_observability/agent_with_mlflow_observability.py:31: UserWarning: Api key is used with an insecure connection.
  aclient = AsyncQdrantClient(url="http://localhost:6333", api_key="th3s3cr3tk3y")
    Agents   Starting console mode 🚀
              
DEBUG:asyncio:Using selector: KqueueSelector
    08:15:55.965 DEBUG    asyncio            Using selector: KqueueSelector  
INFO:livekit.agents:starting worker
    08:15:55.967 INFO     livekit.agents     starting worker {"version": "1.6.7", "rtc-version": "1.1.13"}
INFO:livekit.agents:plugin registered
                 INFO     livekit.agents     plugin registered {"plugin": "livekit.plugins.openai", "version": "1.6.7"}
INFO:livekit.agents:plugin registered
    08:15:55.968 INFO     livekit.agents     plugin registered {"plugin": "livekit.plugins.silero", "version": "1.6.7"}
INFO:livekit.agents:plugin registered
                 INFO     livekit.agents     plugin registered {"plugin": "livekit.plugins.cartesia", "version": "1.6.7"}
INFO:livekit.agents:HTTP server listening on :63907
    08:15:55.969 INFO     livekit.agents     HTTP server listening on :63907  
/Users/pavanmantha/Pavans/Workshops/s2s_agents/.venv/lib/python3.12/site-packages/jwt/api_jwt.py:147: InsecureKeyLengthWarning: The HMAC key is 6 bytes long, which is below the minimum recommended length of 32 bytes for SHA256. See RFC 7518 Section 3.2.
  return self._jws.encode(
WARNING:livekit.agents:no warmed process available for job, waiting for one to be created
    08:15:55.985 WARNING  livekit.agents     no warmed process available for job, waiting for one to be created {"job_id": "mock-job-dfd2d7a8afa9"}
INFO:livekit.agents:initializing job runner
    08:15:55.986 INFO     livekit.agents     initializing job runner {"tid": 27303873}
INFO:voice-agent:MLflow tracing configured successfully!
    08:15:55.988 INFO     voice-agent        MLflow tracing configured successfully!  
INFO:voice-agent:Prewarmed VAD model
    08:15:56.059 INFO     voice-agent        Prewarmed VAD model  
DEBUG:asyncio:Using selector: KqueueSelector
INFO:livekit.agents:job runner initialized
    08:15:56.060 DEBUG    asyncio            Using selector: KqueueSelector  
                 INFO     livekit.agents     job runner initialized {"tid": 27303873, "elapsed_time": 0.07}
INFO:voice-agent:Agent starting for room: console
    08:15:56.078 INFO     voice-agent        Agent starting for room: console  
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.469 DEBUG    livekit.agents     http_session(): creating a new httpclient ctx  
/Users/pavanmantha/Pavans/Workshops/s2s_agents/.venv/lib/python3.12/site-packages/jwt/api_jwt.py:147: InsecureKeyLengthWarning: The HMAC key is 6 bytes long, which is below the minimum 
recommended length of 32 bytes for SHA256. See RFC 7518 Section 3.2.
  return self._jws.encode(
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.513 DEBUG    livekit.agents     using audio io: `Console` -> `AgentSession` -> `TranscriptSynchronizer` -> `AudioSinkProxy` -> `Console`  
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.514 DEBUG    livekit.agents     using transcript io: `AgentSession` -> `TranscriptSynchronizer`  
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.515 INFO     voice-agent        Agent session started! Ready for conversation.  
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.673 DEBUG    livekit.….cartesia Established new Cartesia STT WebSocket connection {"cartesia_request_id": "55dd9277-c7ff-4aaf-994f-c5e234200b90"}
   MacBook Pro Microphone   ▁ ▁ ▁ ▂ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.687 DEBUG    livekit.….cartesia Established new Cartesia TTS WebSocket connection {"cartesia_request_id": "b0e833e0-1838-4282-a812-5e1d0c3eaa45"}
   MacBook Pro Microphone   ▁ ▁ ▁ ▆ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:15:56.744 WARNING  livekit.agents     cloud turn detector failed (message='Invalid response status (401 Unauthorized)', status_code=401, retryable=False); falling back to local mini   
                                             model                                                                                                                                             
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:04.410 DEBUG    livekit.agents     received user preflight transcript {"user_transcript": "What does FHA section 223 F of housing act provide for?", "language": "en"}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:04.633 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "What does FHA section 223 F of housing act provide for?", "language": "en", "transcript_delay": 1.0138118267059326}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:04.636 DEBUG    livekit.agents     eot prediction  
                                         {"probability": 0.7253056168556213, "unlikely_threshold": 0.36, "endpointing_delay": 0.3, "language": "en", "trigger": "stt", "from_cache": true}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:04.639 DEBUG    livekit.agents     user turn committed  
                                         {"last_speaking_time": 1786848363.619498, "last_final_transcript_time": 1786848364.63628, "speech_start_time": 1786848358.330898, "delay_completed": 
false, "source": "stt", "end_of_turn_probability": 0.7253056168556213, "unlikely_threshold": 0.36}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:04.643 DEBUG    livekit.agents     using preemptive generation {"preemptive_lead_time": 0.22996997833251953}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:04.646 DEBUG    livekit.agents     conversation_item_added {"role": "user", "text": "What does FHA section 223 F of housing act provide for?"}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:05.943 DEBUG    livekit.agents     executing tool {"function": "query_info", "arguments": "{\"query\": \"FHA section 223 F of housing act\"}", "speech_id": "speech_c521790e881a"}
query: FHA section 223 F of housing act
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
   ? for shortcuts                                                                                                                                                                             DEBUG:llama_index.core.indices.utils:> Top 20 nodes:
> [Node d8308013-6aa4-4c16-8389-6f50d2669512] [Similarity score:                 0.634752] Section 223(a)(7) (Refinancing of FHA-Insured Mortgages). Section 223(a)(7) of the Housing Act pe...
> [Node 32cfb8da-d14d-427e-9bfa-308fb4f2a9cc] [Similarity score:                 0.586968] Sections 213 and 213(i) of the Housing Act provides for FHA insurance of mortgage loans on cooper...
> [Node 9381a6ec-23c0-4278-8e1b-7401d60e1001] [Similarity score:                 0.551707] Such projects may have been financed originally with conventional or FHA-insured mortgage loans.
...
> [Node 786cec8d-0e08-4cff-b00a-94e1b61e98dc] [Similarity score:                 0.499063] Section 236 of the Housing Act combines governmental mortgage insurance on multifamily housing pr...
> [Node 1f472f42-614b-45a8-86a1-138e21a89922] [Similarity score:                 0.483326] Mortgage Loans insured under the programs described in the related Offering Circular Supplement h...
> [Node 03d4c561-aa8c-4033-b8c7-fa3c28a89cf9] [Similarity score:                 0.388449] Each Ginnie Mae Project Loan Certificate will provide for the payment to the registered holder of...
> [Node f7927814-6be4-472b-9c96-51389a9ba8b5] [Similarity score:                 0.370135] ---

### 📄 Page 78

**Weighted Average Remaining Period from Issuance** — With respect to designa...
> [Node 3b62d0f1-a471-455b-ba2a-ed31950b2b5b] [Similarity score:                 0.361758] A corporation will not be treated as an instrumentality of the United States or any state or poli...
> [Node fcc56c3d-1f91-4004-beb4-19985b765d5e] [Similarity score:                 0.354325] **Prepayment Penalty End Date** — With respect to any multifamily Mortgage Loan, the date as of w...
> [Node 86415b5d-c93a-42d2-b185-0ecaea1150f3] [Similarity score:                 0.348899] If the prepayment rate on the Mortgage Loans decreases during a period of rising interest rates, ...
> [Node 59725d69-37a2-4ee7-a016-9cc9d0ac9dba] [Similarity score:                 0.347137] ---

### 📄 Page 20

### Section 538 Guarantee Program

The Section 538 Guaranteed Rural Rental Ho...
> [Node 6acc230f-5014-4423-8258-22269b50b418] [Similarity score:                 0.344156] If any of these representations and warranties are untrue with respect to any Trust MBS, the Spon...
> [Node 3a9de742-ca46-4570-8dcd-45e33c6b3792] [Similarity score:                 0.342905] 00 exceeds (b) the product of (i) the original principal amount of the Ginnie Mae Project Loan Ce...
> [Node fe0dd607-31e3-4533-b902-1ce1a38002db] [Similarity score:                 0.337367] Certain Trust MBS may be based on and backed by Mortgage Loans secured by a mark-to-market projec...
> [Node 02720481-a4c1-4377-8745-9d432d75621a] [Similarity score:                 0.334485] The original maturity of a Ginnie Mae Construction Loan Certificate is at least 200% of the const...
> [Node 8143abb0-2e8a-4a4c-b401-6a64d3fc47cc] [Similarity score:                 0.331362] The Trust Asset Depository or its nominees, as registered holder (on behalf of the Trustee) of th...
> [Node 2893b0c6-b1d0-40b8-9102-4f77eca8822c] [Similarity score:                 0.329946] Any transfer of a Mortgaged Property is subject to HUD review and approval under the terms of HUD...
> [Node 1f830d75-a35d-422a-a0f8-84fa1a7f283f] [Similarity score:                 0.327878] An election will be made to treat each Trust or certain assets of each Trust as one or more real ...
    08:16:06.202 DEBUG    llama_in…ces.utils > Top 20 nodes:                                                                                                                                   
                                             > [Node d8308013-6aa4-4c16-8389-6f50d2669512] [Similarity score:                 0.634752] Section 223(a)(7) (Refinancing of FHA-Insured          
                                             Mortgages). Section 223(a)(7) of the Housing Act pe...                                                                                            
                                             > [Node 32cfb8da-d14d-427e-9bfa-308fb4f2a9cc] [Similarity score:                 0.586968] Sections 213 and 213(i) of the Housing Act provides    
                                             for FHA insurance of mortgage loans on cooper...                                                                                                  
                                             > [Node 9381a6ec-23c0-4278-8e1b-7401d60e1001] [Similarity score:                 0.551707] Such projects may have been financed originally with   
                                             conventional or FHA-insured mortgage loans.                                                                                                       
                                             ...                                                                                                                                               
                                             > [Node 786cec8d-0e08-4cff-b00a-94e1b61e98dc] [Similarity score:                 0.499063] Section 236 of the Housing Act combines governmental   
                                             mortgage insurance on multifamily housing pr...                                                                                                   
                                             > [Node 1f472f42-614b-45a8-86a1-138e21a89922] [Similarity score:                 0.483326] Mortgage Loans insured under the programs described    
                                             in the related Offering Circular Supplement h...                                                                                                  
                                             > [Node 03d4c561-aa8c-4033-b8c7-fa3c28a89cf9] [Similarity score:                 0.388449] Each Ginnie Mae Project Loan Certificate will provide  
                                             for the payment to the registered holder of...                                                                                                    
                                             > [Node f7927814-6be4-472b-9c96-51389a9ba8b5] [Similarity score:                 0.370135] ---                                                    
                                                                                                                                                                                               
                                             ### 📄 Page 78                                                                                                                                    
                                                                                                                                                                                               
                                             **Weighted Average Remaining Period from Issuance** — With respect to designa...                                                                  
                                             > [Node 3b62d0f1-a471-455b-ba2a-ed31950b2b5b] [Similarity score:                 0.361758] A corporation will not be treated as an                
                                             instrumentality of the United States or any state or poli...                                                                                      
                                             > [Node fcc56c3d-1f91-4004-beb4-19985b765d5e] [Similarity score:                 0.354325] **Prepayment Penalty End Date** — With respect to any  
                                             multifamily Mortgage Loan, the date as of w...                                                                                                    
                                             > [Node 86415b5d-c93a-42d2-b185-0ecaea1150f3] [Similarity score:                 0.348899] If the prepayment rate on the Mortgage Loans           
                                             decreases during a period of rising interest rates, ...                                                                                           
                                             > [Node 59725d69-37a2-4ee7-a016-9cc9d0ac9dba] [Similarity score:                 0.347137] ---                                                    
                                                                                                                                                                                               
                                             ### 📄 Page 20                                                                                                                                    
                                                                                                                                                                                               
                                             ### Section 538 Guarantee Program                                                                                                                 
                                                                                                                                                                                               
                                             The Section 538 Guaranteed Rural Rental Ho...                                                                                                     
                                             > [Node 6acc230f-5014-4423-8258-22269b50b418] [Similarity score:                 0.344156] If any of these representations and warranties are     
                                             untrue with respect to any Trust MBS, the Spon...                                                                                                 
                                             > [Node 3a9de742-ca46-4570-8dcd-45e33c6b3792] [Similarity score:                 0.342905] 00 exceeds (b) the product of (i) the original         
                                             principal amount of the Ginnie Mae Project Loan Ce...                                                                                             
                                             > [Node fe0dd607-31e3-4533-b902-1ce1a38002db] [Similarity score:                 0.337367] Certain Trust MBS may be based on and backed by        
                                             Mortgage Loans secured by a mark-to-market projec...                                                                                              
                                             > [Node 02720481-a4c1-4377-8745-9d432d75621a] [Similarity score:                 0.334485] The original maturity of a Ginnie Mae Construction     
                                             Loan Certificate is at least 200% of the const...                                                                                                 
                                             > [Node 8143abb0-2e8a-4a4c-b401-6a64d3fc47cc] [Similarity score:                 0.331362] The Trust Asset Depository or its nominees, as         
                                             registered holder (on behalf of the Trustee) of th...                                                                                             
                                             > [Node 2893b0c6-b1d0-40b8-9102-4f77eca8822c] [Similarity score:                 0.329946] Any transfer of a Mortgaged Property is subject to     
                                             HUD review and approval under the terms of HUD...                                                                                                 
                                             > [Node 1f830d75-a35d-422a-a0f8-84fa1a7f283f] [Similarity score:                 0.327878] An election will be made to treat each Trust or        
                                             certain assets of each Trust as one or more real ...                                                                                              
                                             > [Node 16478aeb-8d79-4294-9011-918e57601e17] [Similarity score:                 0.326987] The Information Agent identified in the Offering       
                                             Circular Supplement will post the Class Factors,...                                                                                               
                                             > [Node b933dd07-7423-4f11-bcd2-6c6ec1062ccc] [Similarity score:                 0.323226] At any time following the final endorsement of a       
                                             Mortgage Loan and upon satisfaction of the prere...                                                                                               
Query result: Section 223(a)(7) (Refinancing of FHA-Insured Mortgages). Section 223(a)(7) of the Housing Act permits FHA to refinance existing insured mortgage loans under any section or 
title of the Housing Act. Such refinancing results in prepayment of the existing insured mortgage. The new, refinanced mortgage loan is limited to the original principal amount of the 
existing mortgage loan and the unexpired term of the existing mortgage loan plus 12 years.

Section 223(d)(Operating Loss Loans). Section 223(d) of the Housing Act provides for FHA insurance of separate loans that cover (1) operating losses during the first 2 years after completion 
or (2) up to 80% of the unreimbursed cash contributions by the project owner during any period of up to two years within the first 10 years after date of completion of the project. The 
project must be secured by an existing HUD-insured first mortgage loan.

Section 223(f) (Purchase or Refinancing of Existing Projects). Section 223(f) of the Housing Act provides for federal insurance of mortgage loans originated by FHA-approved lenders in 
connection with the purchase or refinancing of existing multifamily housing complexes, hospitals and nursing homes that do not require substantial rehabilitation. The principal objective of 
the Section 223(f) program is to permit the refinancing of mortgage loans to provide for a lower debt service or the purchase of existing properties in order to preserve an adequate supply of
affordable rental housing. Such projects may have been financed originally with conventional or FHA-insured mortgage loans.

Section 231 (Mortgage Insurance for Rental Housing for the Elderly). Section 231 of the Housing Act provides for insurance of mortgage loans to facilitate the construction and substantial 
rehabilitation of multifamily rental housing for elderly (62 or older) or disabled persons. The mortgage insurance may be used to finance the construction and substantial rehabilitation of 
detached, semi-detached, walk-up or elevator type rental housing designed specifically for elderly or disabled individuals consisting of 8 or more dwelling units. Section 231 was designed to 
increase the supply of rental housing specifically for the use and occupancy of elderly and/or disabled persons.

Section 232 (Mortgage Insurance for Nursing Homes, Immediate Care Facilities and Board and Care Homes).
Query result: Sections 213 and 213(i) of the Housing Act provides for FHA insurance of mortgage loans on cooperative housing projects. Sections 213 and 213(i) mortgage insurance enables 
nonprofit cooperative ownership housing corporations or trusts to develop or sponsor housing projects that will be operated as cooperatives. By using Sections 213 and 213(i) insurance, 
investors can construct or rehabilitate multifamily housing that will be sold to such nonprofit corporations or trusts.

Section 220 (Urban Renewal Mortgage Insurance). Section 220 of the Housing Act provides for federal insurance of mortgage loans on multifamily rental projects located in federally aided urban
renewal areas or in areas having a local redevelopment or urban renewal plan certified by FHA. The mortgage loans may finance the rehabilitation of existing salvable housing (including the 
refinancing of existing loans) or new construction in targeted areas. The purpose of Section 220 is to encourage quality rental housing in urban areas targeted for overall revitalization.

---

### 📄 Page 19

Section 221(d) (Housing for Moderate Income and Displaced Families). Sections 221(d)(3) and 221(d)(4) of the Housing Act provide for mortgage insurance to assist private industry in the 
construction or substantial rehabilitation of rental and cooperative housing for low- and moderate-income families and families that have been displaced as a result of urban renewal, 
governmental actions or disaster.

Section 223(a)(7) (Refinancing of FHA-Insured Mortgages). Section 223(a)(7) of the Housing Act permits FHA to refinance existing insured mortgage loans under any section or title of the 
Housing Act. Such refinancing results in prepayment of the existing insured mortgage. The new, refinanced mortgage loan is limited to the original principal amount of the existing mortgage 
loan and the unexpired term of the existing mortgage loan plus 12 years.

Section 223(d)(Operating Loss Loans). Section 223(d) of the Housing Act provides for FHA insurance of separate loans that cover (1) operating losses during the first 2 years after completion 
or (2) up to 80% of the unreimbursed cash contributions by the project owner during any period of up to two years within the first 10 years after date of completion of the project. The 
project must be secured by an existing HUD-insured first mortgage loan.
Query result: Such projects may have been financed originally with conventional or FHA-insured mortgage loans.

Section 231 (Mortgage Insurance for Rental Housing for the Elderly). Section 231 of the Housing Act provides for insurance of mortgage loans to facilitate the construction and substantial 
rehabilitation of multifamily rental housing for elderly (62 or older) or disabled persons. The mortgage insurance may be used to finance the construction and substantial rehabilitation of 
detached, semi-detached, walk-up or elevator type rental housing designed specifically for elderly or disabled individuals consisting of 8 or more dwelling units. Section 231 was designed to 
increase the supply of rental housing specifically for the use and occupancy of elderly and/or disabled persons.

Section 232 (Mortgage Insurance for Nursing Homes, Immediate Care Facilities and Board and Care Homes). Section 232 of the Housing Act provides for FHA insurance of private construction 
mortgage loans to finance new or rehabilitated nursing homes, intermediate care facilities, board and care homes, assisted living for the frail or elderly or allowable combinations thereof, 
including equipment to be used in their operation. Section 232 also provides for supplemental loans to finance the purchase and installation of fire safety equipment in these facilities.

Section 236 (Mortgage Insurance for Subsidized Rental Housing Projects). Section 236 of the Housing Act combines governmental mortgage insurance on multifamily housing projects with 
supplemental payments to reduce the project owners' monthly debt service payments. The supplemental payments are paid directly to the mortgagee of the project for the purpose of reducing the 
interest payment due from the project owner. The objective of these supplemental payments is to reduce rental payments required of low-income and elderly residents. To qualify for rental 
assistance under Section 236, tenants' annual income must be less than 80 percent of the median income of the area. Originations under Section 236 are no longer active, although refinancings 
under Section 223(a)(7) are authorized.

Section 241 (Supplemental Loans for Multifamily Projects). Sections 241, and 241(a) and 241(f) of the Housing Act provide for FHA insurance to finance property improvements, energy-conserving
improvements, supplemental increases or additions to any FHA-insured multifamily loan.
Query result: Section 236 of the Housing Act combines governmental mortgage insurance on multifamily housing projects with supplemental payments to reduce the project owners' monthly debt 
service payments. The supplemental payments are paid directly to the mortgagee of the project for the purpose of reducing the interest payment due from the project owner. The objective of 
these supplemental payments is to reduce rental payments required of low-income and elderly residents. To qualify for rental assistance under Section 236, tenants' annual income must be less 
than 80 percent of the median income of the area. Originations under Section 236 are no longer active, although refinancings under Section 223(a)(7) are authorized.

Section 241 (Supplemental Loans for Multifamily Projects). Sections 241, and 241(a) and 241(f) of the Housing Act provide for FHA insurance to finance property improvements, energy-conserving
improvements, supplemental increases or additions to any FHA-insured multifamily loan. The overall purpose of the Section 241 loan program is to provide a project with a means to remain 
competitive, to extend its economic life and to finance the replacement of obsolete equipment without the refinancing of the existing mortgage.

---

### 📄 Page 20

### Section 538 Guarantee Program

The Section 538 Guaranteed Rural Rental Housing Program is under the United States Department of Agriculture Rural Development. The authorizing statute is Title V of the Housing Act. Rural 
Development operates a broad range of programs that were formerly administered by the Farmers Home Administration to support affordable housing and community development in rural areas. 
Section 538 was established pursuant to Title V of the Housing Act. Section 538 is designed to increase the supply of affordable rural rental housing, through the use of loan guarantees that 
encourage partnerships between Rural Development, private lenders and public agencies. Mortgage loans are provided by Rural Development-approved multifamily lenders, including state and local
housing agencies. The Mortgage Loans guaranteed under the program will have the maturity and amortization features as Rural Development may approve.
Query result: Mortgage Loans insured under the programs described in the related Offering Circular Supplement have such maturities and amortization features as the FHA may approve. Generally 
the minimum mortgage loan term is at least ten years, and the maximum mortgage loan term does not exceed the lesser of 40 years and 75% of the estimated remaining economic life of the 
improvements on the Mortgaged Property.

Tenant eligibility for FHA-insured projects generally is not restricted by income, except for projects as to which rental subsidies are made available with respect to some of or all the units
therein or to specified tenants.

The principal characteristics of the Mortgage Loans underlying the Ginnie Mae Multifamily Certificates in a particular Trust, and the FHA Programs under which each Mortgage Loan is insured, 
are described below. The FHA Programs under which each Mortgage Loan in a Series is insured will be listed under "Terms Sheets – Certain Characteristics of the Ginnie Mae Multifamily 
Certificates and the Related Mortgage Loans Underlying the Trust Assets" in the related Offering Circular Supplement.

Section 207 (Mortgage Insurance for Multifamily Housing). Section 207 of the Housing Act provides for federal insurance of mortgage loans originated by FHA-approved lenders in connection with
the construction or substantial rehabilitation of multifamily housing projects, which includes manufactured home parks.

Section 213(i) (Cooperative Housing Projects). Sections 213 and 213(i) of the Housing Act provides for FHA insurance of mortgage loans on cooperative housing projects. Sections 213 and 213(i)
mortgage insurance enables nonprofit cooperative ownership housing corporations or trusts to develop or sponsor housing projects that will be operated as cooperatives. By using Sections 213 
and 213(i) insurance, investors can construct or rehabilitate multifamily housing that will be sold to such nonprofit corporations or trusts.

Section 220 (Urban Renewal Mortgage Insurance). Section 220 of the Housing Act provides for federal insurance of mortgage loans on multifamily rental projects located in federally aided urban
renewal areas or in areas having a local redevelopment or urban renewal plan certified by FHA. The mortgage loans may finance the rehabilitation of existing salvable housing (including the 
refinancing of existing loans) or new construction in targeted areas. The purpose of Section 220 is to encourage quality rental housing in urban areas targeted for overall revitalization.
Query result: Each Ginnie Mae Project Loan Certificate will provide for the payment to the registered holder of that Ginnie Mae Project Loan Certificate of monthly payments of principal and 
interest equal to the aggregate amount of

---

### 📄 Page 18

the scheduled monthly principal and interest payments on the Mortgage Loans underlying that Ginnie Mae Project Loan Certificate, less applicable servicing and guaranty fees. In addition, such
payments will include any prepayments and other unscheduled recoveries of principal of, and any Prepayment Penalties on, an underlying Mortgage Loan to the extent received by the Ginnie Mae 
Issuer during the month preceding the month of the payment.

The related Offering Circular Supplement will describe certain characteristics of the Ginnie Mae Project Loan Certificates and the underlying Mortgage Loans.

### FHA Insurance Programs

The FHA is an organizational unit within the U.S. Department of Housing and Urban Development. FHA was established to encourage improvement in housing standards and conditions, to provide an 
adequate home financing system by insuring housing mortgages and credit and to exert a stabilizing influence on the mortgage market.

FHA multifamily insurance programs generally are designed to assist private and public mortgagors in obtaining insured financing for the construction, purchase or rehabilitation of 
multifamily housing pursuant to the Housing Act or, with respect to a risk sharing pilot program, the Housing and Community Development Act of 1992, as amended. Mortgage Loans are provided by
FHA-approved institutions, which include mortgage banks, commercial banks, savings and loan associations, trust companies, insurance companies, pension funds, state and local housing finance 
agencies and certain other approved entities.

Mortgage Loans insured under the programs described in the related Offering Circular Supplement have such maturities and amortization features as the FHA may approve. Generally the minimum 
mortgage loan term is at least ten years, and the maximum mortgage loan term does not exceed the lesser of 40 years and 75% of the estimated remaining economic life of the improvements on the
Mortgaged Property.

Tenant eligibility for FHA-insured projects generally is not restricted by income, except for projects as to which rental subsidies are made available with respect to some of or all the units
therein or to specified tenants.

The principal characteristics of the Mortgage Loans underlying the Ginnie Mae Multifamily Certificates in a particular Trust, and the FHA Programs under which each Mortgage Loan is insured, 
are described below.
Query result: ---

### 📄 Page 78

**Weighted Average Remaining Period from Issuance** — With respect to designated Ginnie Mae Multifamily Certificates and a Cut-off Date, the average, expressed in months, of the remaining 
period from issuance, as of the Cut-off Date, of all the Mortgage Loans underlying the Ginnie Mae Multifamily Certificates, weighted on the basis of the respective current principal balances 
of those Mortgage Loans after giving effect to all payments of principal due on or before the Cut-off Date. For purposes of this definition "issuance" means the later of (a) one month prior 
to the first interest rate payment date after the most recent FHA endorsement and (b) the Cut-off Date of the related Ginnie Mae Multifamily Certificate, unless otherwise indicated in the 
Offering Circular Supplement.

**Weighted Average Remaining Prepayment Penalty Period** — With respect to designated Ginnie Mae Multifamily Certificates and a Cut-off Date, the average, expressed in months, of the 
remaining periods during which Prepayment Penalties will be payable, as of the Cut-off Date, of all the Mortgage Loans underlying the Ginnie Mae Multifamily Certificates, weighted on the 
basis of the respective current principal balances of those Mortgage Loans after giving effect to all payments of principal due on or before the related Cut-off Date.

---

### 📄 Page 79

## THIS PAGE INTENTIONALLY LEFT BLANK
Query result: A corporation will not be treated as an instrumentality of the United States or any state or political subdivision thereof if all of its activities are subject to tax and, with 
the exception of Freddie Mac, a majority of its board of directors is not selected by that governmental unit.

**Distribution Amount** — With respect to each Series (or, if the Series is segregated into Security Groups, each Security Group) and each Distribution Date, the sum of the Principal 
Distribution Amount (less principal, if any, payable to the Trustee as a Trustee Fee), the Interest Distribution Amount and the Accrual Amount(s), if any, for the Series (or Security Group).

---

### 📄 Page 59

**Distribution Date** — The date specified in the Trust Agreement or MX Trust Agreement, as applicable, relating to each Series (or Security Group) upon which distributions are required to be
made to Holders of Securities of such Series (or Security Group).

**Double REMIC Series** — A Series that provides for an Issuing REMIC and one or more Pooling REMICs.

**Effective Range** — With respect to any PAC, Scheduled or TAC Class or Component, the range of constant prepayment rates for which such Class or Component adheres to its schedule of 
Scheduled Principal Balances.

**Eligible Certificates** — Any Ginnie Mae Securities, as well as any Fannie Mae Securities or Freddie Mac Securities that are held in book-entry form.

ERISA ................................................. The Employee Retirement Income Security Act of 1974, as amended.

**Excess Class** — A Class that receives any principal and interest paid on the underlying Trust Assets in excess of the amount of the prescribed principal and interest required to be paid on
all Classes in the Series.

Fannie Mae.......................................... The Federal National Mortgage Association.

**Fannie Mae Securities** — Any securities previously issued and guaranteed by Fannie Mae that evidence beneficial ownership interests in Ginnie Mae Multifamily Certificates.

**Fedwire Book-Entry System** — The book-entry system for securities operated and maintained by the U.S. Federal Reserve Banks.

FHA .................................................... The Federal Housing Administration.

FHLB of San Francisco....................... The Federal Home Loan Bank of San Francisco.

**Final Data Statement** — With respect to each Series, the final list of Trust Assets to be included in the related Trust.
Query result: **Prepayment Penalty End Date** — With respect to any multifamily Mortgage Loan, the date as of which any such Mortgage Loan would no longer be subject to the payment of any 
Prepayment Penalties.

**Principal Distribution Amount** — With respect to each Series (or if the Series is segregated into Security Groups, each Security Group) and each Distribution Date, the sum of (I) with 
respect to each Ginnie Mae Multifamily Certificate (other than any Ginnie Mae Project Loan Certificate issued as a result of a conversion of the Ginnie Mae Construction Loan Certificate since
the preceding Distribution Date), the amount by which (a) the product of (i) the original principal amount of that Ginnie Mae Multifamily Certificate and (ii) the Certificate Factor, 
Corrected Certificate Factor or Calculated Certificate Factor, as applicable, for the preceding Distribution Date exceeds (b) the product of (i) the original principal amount of the Ginnie 
Mae Multifamily Certificate and (ii) the

---

### 📄 Page 68

Certificate Factor, Corrected Certificate Factor or Calculated Certificate Factor, as applicable, for the current Distribution Date; (II) with respect to each Ginnie Mae Project Loan 
Certificate issued as a result of a conversion of a Ginnie Mae Construction Loan Certificate since the preceding Distribution Date, the sum of (1) the amount by which (a) the product of (i) 
the original principal amount of the Ginnie Mae Project Loan Certificate and (ii) 1.00 exceeds (b) the product of (i) the original principal amount of the Ginnie Mae Project Loan Certificate 
and (ii) the Certificate Factor, Corrected Certificate Factor or Calculated Certificate Factor, as applicable, for the current Distribution Date and (2) any amounts received by the Trust from
the Ginnie Mae Issuer from the related P&I Custodial Account on the date of conversion of a Ginnie Mae Construction Loan Certificate to a Ginnie Mae Project Loan Certificate; (III) with 
respect to each Ginnie Mae Construction Loan Certificate that has been liquidated since the preceding Distribution Date, any proceeds received by the Trust from the Ginnie Mae Issuer with 
respect to any liquidation, exclusive of any amounts distributed pursuant to (I) above; (IV) with respect to each Underlying Certificate, Underlying Callable Security or Underlying SMBS 
Security,
Query result: If the prepayment rate on the Mortgage Loans decreases during a period of rising interest rates, investors may receive declining principal distributions when those investors 
otherwise may have been able to reinvest at higher interest rates than the Interest Rates of the applicable Classes of Securities.

### Payment Delay

Distributions of interest on the Securities on any Distribution Date will include interest accrued thereon through the Accounting Date, which for Fixed Rate Classes and Delay Classes is the 
last day of the month preceding the month in which such Distribution Date occurs. The effective yield to the Holders of such Securities will be lower than the yield otherwise produced by the 
applicable Interest Rate and purchase price because interest will not be distributed on such Securities that are Book-Entry Securities until the Distribution Date in the month following the 
month in which such interest accrues on the Trust Assets, and interest will not be distributed on Certificated Securities until the Business Day after such Distribution Date.

---

### 📄 Page 22

### Assumability of Mortgage Loans

Mortgage Loans do not contain "due-on-sale" clauses restricting sale or other transfer of the related Mortgaged Properties. Any transfer of a Mortgaged Property is subject to HUD review and 
approval under the terms of HUD's Regulatory Agreement with the owner, which is incorporated by reference into the mortgage.

### Weighted Average Life

The Weighted Average Life of a security refers to the average amount of time that will elapse from the date of its issuance until each dollar of principal of that security will be repaid to 
the investor. As a result, any projection of the Weighted Average Life of and yield on any Class of the Securities must include an assumption about the anticipated timing and amount of 
payments on those Securities, which will depend upon the rate of prepayments of the Mortgage Loans, including optional borrower prepayments and prepayments resulting from liquidation of 
defaulted Mortgage Loans. In general, prepayments of principal and defaults on the Mortgage Loans will shorten the Weighted Average Life and term to maturity of each related Class of 
Securities.
Query result: ---

### 📄 Page 20

### Section 538 Guarantee Program

The Section 538 Guaranteed Rural Rental Housing Program is under the United States Department of Agriculture Rural Development. The authorizing statute is Title V of the Housing Act. Rural 
Development operates a broad range of programs that were formerly administered by the Farmers Home Administration to support affordable housing and community development in rural areas. 
Section 538 was established pursuant to Title V of the Housing Act. Section 538 is designed to increase the supply of affordable rural rental housing, through the use of loan guarantees that 
encourage partnerships between Rural Development, private lenders and public agencies. Mortgage loans are provided by Rural Development-approved multifamily lenders, including state and local
housing agencies. The Mortgage Loans guaranteed under the program will have the maturity and amortization features as Rural Development may approve.

Tenant eligibility for Section 538 guaranteed projects is restricted to persons with income not in excess of 115% of the area median income

Under Section 515 of Title V of the Housing Act, Rural Development is authorized to make direct loans secured by multifamily properties with respect to which the tenants may include very 
low-, low- and moderate-income families, elderly persons and persons with handicaps and disabilities. Under Section 538, Rural Development subsequently may guarantee new loans, secured by the
same properties, made to revitalize the properties.

## UNDERLYING CERTIFICATES

If so specified in the related Offering Circular Supplement, the Trust Assets for a Series of Securities may include one or more Underlying Certificates. Any such Underlying Certificate will 
evidence a direct or indirect beneficial ownership interest in a separate pool of Ginnie Mae Multifamily Certificates and will have been issued and guaranteed as described in the Underlying 
Certificate Disclosure Document. Each Offering Circular Supplement will include a general description of the characteristics of each Underlying Certificate. In the event that any issue arises
under the trust agreement that governs the Underlying Trust, which requires a vote of the holders of Underlying Certificates, the Trustee will vote the Underlying Certificates in accordance 
with the Trust Agreement.

Investors in any Security representing an interest in one or more Underlying Certificates are urged to review, in particular, the related Underlying Certificate Disclosure Documents, which 
may be obtained from the Information Agent as described in the related Offering Circular Supplement.
Query result: If any of these representations and warranties are untrue with respect to any Trust MBS, the Sponsor, at its option, may (a) cure the breach, (b) substitute another Ginnie Mae 
Multifamily Certificate for the affected Trust MBS, or (c) with Ginnie Mae's consent, repurchase the affected Trust MBS from the Trust, in each case only to the extent permitted under the 
Trust Agreement and REMIC Provisions.

The Mortgage Loans underlying each Trust MBS will consist of Mortgage Loans secured by mortgages on real properties that are either residences housing five families or more or nursing 
facilities (collectively, "Mortgaged Properties").

---

### 📄 Page 16

Each multifamily Mortgage Loan underlying any Trust MBS (other than a Trust MBS issued under the Small Loan Program (as defined below under "The Ginnie Mae Project Loan Certificates")) bears 
interest at a Mortgage Rate that is 0.25% to 0.50% per annum greater than the related Certificate Rate, unless Ginnie Mae has given its prior consent to a spread in excess of 0.50% per annum.
Under the Small Loan Program, the Mortgage Rate on the Mortgage Loans backing the Trust MBS is required to be at least 0.50% greater than the Certificate Rate, but there is no maximum limit 
on this percentage. Certain Trust MBS may be based on and backed by Mortgage Loans secured by a mark-to-market project developed under HUD's Office of Multifamily Housing Assistance 
Restructuring Processing Procedures (the "RX Program"). The Mortgage Rate of Mortgage Loans underlying any such Trust MBS is required to be at least 0.50% greater than the Certificate Rate of
the related Trust MBS.

Each Ginnie Mae Issuer will perform the routine functions required for servicing of Trust MBS and related Mortgage Loans for which it is responsible, including mortgagor billings, receipt and
posting of payments, payment of property taxes and hazard insurance premiums, remittance, collections and customer service. Each Ginnie Mae Issuer will be obligated under its Certificate 
Guaranty Agreements with Ginnie Mae to service the Mortgage Loans in accordance with FHA requirements and with generally accepted practices in the mortgage lending industry.
Query result: 00 exceeds (b) the product of (i) the original principal amount of the Ginnie Mae Project Loan Certificate and (ii) the Certificate Factor, Corrected Certificate Factor or 
Calculated Certificate Factor, as applicable, for the current Distribution Date and (2) any amounts received by the Trust from the Ginnie Mae Issuer from the related P&I Custodial Account on 
the date of conversion of a Ginnie Mae Construction Loan Certificate to a Ginnie Mae Project Loan Certificate; (III) with respect to each Ginnie Mae Construction Loan Certificate that has 
been liquidated since the preceding Distribution Date, any proceeds received by the Trust from the Ginnie Mae Issuer with respect to any liquidation, exclusive of any amounts distributed 
pursuant to (I) above; (IV) with respect to each Underlying Certificate, Underlying Callable Security or Underlying SMBS Security, the amount by which (a) the product of (i) the original 
principal amount of that Underlying Certificate, Underlying Callable Security or Underlying SMBS Security, and (ii) the Underlying Certificate Factor (or Certificate Factor) for the preceding
Distribution Date exceeds (b) the product of (i) the original principal amount of the Underlying Certificate, Underlying Callable Security or Underlying SMBS Security, and (ii) the Underlying
Certificate Factor (or Certificate Factor) for the current Distribution Date; provided however, that the amount calculated pursuant to clause (IV) is subject to adjustment pursuant to the 
applicable Trust Agreement in the event that Underlying Certificate Factors (or Certificate Factors) are unavailable in respect of any Distribution Date. (For the first Distribution Date, the
product in clause (I)(a) and (IV)(a) above shall be the principal amount of the Ginnie Mae Multifamily Certificate as of the Closing Date.) The sum of the amounts so calculated for each 
Ginnie Mae Multifamily Certificate conveyed to a Trust for a Series (or, if the Series is segregated into Security Groups, for each Ginnie Mae Multifamily Certificate included in the related 
Trust Asset Group) is the Principal Distribution Amount for that Series (or Security Group(s)).

When preceded by a group or subgroup designation (e.g., the "Group 2 Principal Distribution Amount" or the "Subgroup 2A Principal Distribution Amount"), such amount for the specified Trust 
Asset Group or Subgroup.
Query result: Certain Trust MBS may be based on and backed by Mortgage Loans secured by a mark-to-market project developed under HUD's Office of Multifamily Housing Assistance Restructuring 
Processing Procedures (the "RX Program"). The Mortgage Rate of Mortgage Loans underlying any such Trust MBS is required to be at least 0.50% greater than the Certificate Rate of the related 
Trust MBS.

Each Ginnie Mae Issuer will perform the routine functions required for servicing of Trust MBS and related Mortgage Loans for which it is responsible, including mortgagor billings, receipt and
posting of payments, payment of property taxes and hazard insurance premiums, remittance, collections and customer service. Each Ginnie Mae Issuer will be obligated under its Certificate 
Guaranty Agreements with Ginnie Mae to service the Mortgage Loans in accordance with FHA requirements and with generally accepted practices in the mortgage lending industry. Each Ginnie Mae 
Issuer's responsibilities with respect to its Mortgage Loans will include:

(a) collection of all principal and interest payments, Prepayment Penalties, if any, and payments made by the borrower or borrowers toward escrows established for taxes and insurance 
premiums;

(b) maintenance of necessary hazard insurance policies;

(c) institution of all actions necessary to foreclose on, or take other appropriate action with respect to, loans in default; and

(d) collection of insurance and guaranty benefits.

The Trust Asset Depository or its nominees, as registered holder (on behalf of the Trustee) of the related Trust MBS, will have the right to proceed directly against Ginnie Mae under the 
terms of the related Trust MBS for any amounts that are not paid when due.

Ginnie Mae Multifamily Certificates are issued under the Ginnie Mae I Program.

### The Ginnie Mae Construction Loan Certificates

Each Ginnie Mae Construction Loan Certificate is based on and backed by a single Mortgage Loan secured by a multifamily project under construction and insured by FHA pursuant to an FHA 
Insurance Program described in the related Offering Circular Supplement. Ginnie Mae Construction Loan Certificates are generally issued monthly by the related Ginnie Mae Issuer as 
construction progresses on the related multifamily project and as advances are insured by FHA.
Query result: The original maturity of a Ginnie Mae Construction Loan Certificate is at least 200% of the construction period for the multifamily project anticipated by FHA. The stated 
maturity of a Ginnie Mae Construction Loan Certificate may be extended after issuance at the request of the related Ginnie Mae Issuer with the prior written approval of Ginnie Mae. Prior to 
approving any extension request, Ginnie Mae requires that the Contracted Security Purchaser, the entity bound under contract with the related Ginnie Mae issuer to purchase all of the Ginnie 
Mae Construction Loan Certificates relating to a particular project, consent to the extension of the maturity date. Upon the maturity of any Ginnie Mae Construction Loan Certificate, the 
related Ginnie Mae Issuer must retire the Ginnie Mae Construction Loan Certificate by the payment of cash to the holders provided that (i) no extension has been granted and (ii) the Ginnie 
Mae Construction Loan Certificate has failed to convert prior to that time. In addition if

---

### 📄 Page 17

the underlying Mortgage Loan is liquidated prior to maturity, the related Ginnie Mae Issuer must retire the Ginnie Mae Construction Loan Certificates by the payment of cash to the holders.

Each Ginnie Mae Construction Loan Certificate will provide for the payment to the holder of that Ginnie Mae Construction Loan Certificate of monthly payments of interest equal to a pro rata 
share of the interest payments on the underlying Mortgage Loan, less applicable servicing and guaranty fees. Holders of Ginnie Mae Construction Loan Certificates are not entitled to receive 
any payments of principal collected on the Mortgage Loan as long as the Ginnie Mae Construction Loan Certificates are outstanding. During such period, any prepayments and other recoveries of 
principal (other than proceeds from the liquidation of the related Mortgage Loan) or any Prepayment Penalties on the underlying Mortgage Loan received by the Ginnie Mae Issuer will be 
deposited into a non-interest bearing escrow account (the "P&I Custodial Account").
Query result: The Trust Asset Depository or its nominees, as registered holder (on behalf of the Trustee) of the related Trust MBS, will have the right to proceed directly against Ginnie Mae 
under the terms of the related Trust MBS for any amounts that are not paid when due.

Ginnie Mae Multifamily Certificates are issued under the Ginnie Mae I Program.

### The Ginnie Mae Construction Loan Certificates

Each Ginnie Mae Construction Loan Certificate is based on and backed by a single Mortgage Loan secured by a multifamily project under construction and insured by FHA pursuant to an FHA 
Insurance Program described in the related Offering Circular Supplement. Ginnie Mae Construction Loan Certificates are generally issued monthly by the related Ginnie Mae Issuer as 
construction progresses on the related multifamily project and as advances are insured by FHA. Prior to the issuance of Ginnie Mae Construction Loan Certificates, the Ginnie Mae Issuer must 
provide Ginnie Mae with supporting documentation regarding advances and disbursements on the Mortgage Loan and must satisfy the prerequisites for issuance as described in Chapter 32 of the 
Ginnie Mae Mortgage-Backed Securities Guide (the "MBS Guide"). Each Ginnie Mae Construction Loan Certificate upon conversion will be redeemed for a pro rata share of a Ginnie Mae Project Loan
Certificate.

The original maturity of a Ginnie Mae Construction Loan Certificate is at least 200% of the construction period for the multifamily project anticipated by FHA. The stated maturity of a Ginnie
Mae Construction Loan Certificate may be extended after issuance at the request of the related Ginnie Mae Issuer with the prior written approval of Ginnie Mae. Prior to approving any 
extension request, Ginnie Mae requires that the Contracted Security Purchaser, the entity bound under contract with the related Ginnie Mae issuer to purchase all of the Ginnie Mae 
Construction Loan Certificates relating to a particular project, consent to the extension of the maturity date. Upon the maturity of any Ginnie Mae Construction Loan Certificate, the related 
Ginnie Mae Issuer must retire the Ginnie Mae Construction Loan Certificate by the payment of cash to the holders provided that (i) no extension has been granted and (ii) the Ginnie Mae 
Construction Loan Certificate has failed to convert prior to that time.
Query result: Any transfer of a Mortgaged Property is subject to HUD review and approval under the terms of HUD's Regulatory Agreement with the owner, which is incorporated by reference into 
the mortgage.

### Weighted Average Life

The Weighted Average Life of a security refers to the average amount of time that will elapse from the date of its issuance until each dollar of principal of that security will be repaid to 
the investor. As a result, any projection of the Weighted Average Life of and yield on any Class of the Securities must include an assumption about the anticipated timing and amount of 
payments on those Securities, which will depend upon the rate of prepayments of the Mortgage Loans, including optional borrower prepayments and prepayments resulting from liquidation of 
defaulted Mortgage Loans. In general, prepayments of principal and defaults on the Mortgage Loans will shorten the Weighted Average Life and term to maturity of each related Class of 
Securities.

The Weighted Average Life of a Class is determined by (a) multiplying the amount of the net reduction, if any, of the Class Principal Balance (or Class Notional Balance) of such Class from 
one Distribution Date to the next Distribution Date by the number of years from the Closing Date to such next Distribution Date, (b) summing the results and (c) dividing the sum by the 
aggregate amount of the net reductions in Class Principal Balance (or Class Notional Balance) of such Class referred to in clause (a).

The Weighted Average Lives of the Securities will be influenced by, among other things, the rate at which principal is paid on the Mortgage Loans (or, in the case of a Security Group, on the 
Mortgage Loans underlying the related Trust Asset Group). In general, the Weighted Average Lives of the Securities will be shortened if the rate of prepayments of principal of the Mortgage 
Loans increases. However, the Weighted Average Lives will depend upon a variety of other factors, including the timing of changes in such rate of principal prepayments. Accordingly, no 
assurance can be given as to the Weighted Average Life of any Class. Further, to the extent the prices of the Securities represent discounts or premiums to their respective original principal
balances, variability in the Weighted Average Lives of such Classes could result in variability in the related yields to maturity.
Query result: An election will be made to treat each Trust or certain assets of each Trust as one or more real estate mortgage investment conduits for United States federal income tax 
purposes. See "Certain United States Federal Income Tax Consequences" in this Multifamily Base Offering Circular.

---

> **THE GOVERNMENT NATIONAL MORTGAGE ASSOCIATION GUARANTEES THE TIMELY PAYMENT OF PRINCIPAL AND INTEREST ON THE SECURITIES. THE GINNIE MAE GUARANTY IS BACKED BY THE FULL FAITH AND CREDIT OF 
THE UNITED STATES OF AMERICA. GINNIE MAE DOES NOT GUARANTEE THE PAYMENT OF PREPAYMENT PENALTIES. THE SECURITIES ARE EXEMPT FROM THE REGISTRATION REQUIREMENTS OF THE SECURITIES ACT OF 1933 AND
CONSTITUTE EXEMPTED SECURITIES UNDER THE SECURITIES EXCHANGE ACT OF 1934.**

---

*Offers of the Securities may be made through one or more different methods, including offerings through the Sponsor, as more fully described in the related Offering Circular Supplement. This
Multifamily Base Offering Circular may not be used to consummate sales of Securities unless you have received the related Offering Circular Supplement.*

**The date of this Multifamily Base Offering Circular is July 1, 2023.
Query result: The Information Agent identified in the Offering Circular Supplement will post the Class Factors, along with the Interest Rate for each Class, on ginniemae.gov.

For any Distribution Date, investors can calculate the amount of principal to be distributed on any Class (other than an Accrual Class or Partial Accrual Class) by multiplying the Original 
Class Principal Balance of that Class by the difference between its Class Factors for the preceding and current months. The amount of interest to be distributed on any Class (other than an 
Accrual Class or Partial Accrual Class) on each Distribution Date will equal 30 days' interest at the Interest Rate for that Class on its Class Principal Balance (or Class Notional Balance) 
as determined by its Class Factor for the preceding month. Based on the Class Factors and Interest Rates published each month, investors in an Accrual Class or Partial Accrual Class can 
calculate the total amount of principal and interest to be distributed to (or interest to be added to the Class Principal Balance of) that Class.

Prepayment Penalties received by the Trustee with respect to a Ginnie Mae Multifamily Certificate will be distributed, along with the related prepayment, on the Distribution Date, immediately
following the Trustee's actual receipt of the Prepayment Penalties, to Holders of one or more Classes of Securities in accordance with provisions set forth in the related Offering Circular 
Supplement. A distribution of Prepayment Penalties will not decrease the Class Principal Balance of any Class receiving such a distribution.

### Method of Distributions

Distributions of principal and interest (or, where applicable, of principal only or interest only) and related Prepayment Penalties, if any, on a Series (or, if the Series is segregated into 
Security Groups, on a Security Group) will be made on each Distribution Date for that Series (or Security Group) (or, with respect to Certificated Securities, the Business Day following the 
Distribution Date) to the Persons in whose names the Securities are registered on the related Record Date.
Query result: At any time following the final endorsement of a Mortgage Loan and upon satisfaction of the prerequisites for conversion outlined in Chapter 32 of the MBS Guide, Ginnie Mae 
Construction Loan Certificates shall be redeemed for Ginnie Mae Project Loan Certificates. The principal amount of the Ginnie Mae Project Loan Certificates may be less than or equal to the 
amount of advances that have been disbursed and insured on the Mortgage Loan underlying the related Ginnie Mae Construction Loan Certificates. Any difference between the principal balance of 
the Ginnie Mae Construction Loan Certificates and the principal balance of the Ginnie Mae Project Loan Certificates issued at conversion will be disbursed to the holders of the Ginnie Mae 
Construction Loan Certificates as principal upon conversion. To the extent the underlying Mortgage Loan begins to amortize prior to the conversion of the Ginnie Mae Construction Loan 
Certificates to Ginnie Mae Project Loan Certificate, the related Ginnie Mae Issuer will be required to pay, at its option, any such prepayments (i) on the date of the conversion of the Ginnie
Mae Construction Loan Certificates or (ii) on the Ginnie Mae Certificate Payment Date of the Ginnie Mae Project Loan Certificate following conversion.

The related Offering Circular Supplement will describe certain characteristics of the Ginnie Mae Construction Loan Certificates and the underlying Mortgage Loans.

Although the Trust may receive principal payments, unscheduled recoveries of principal and Prepayment Penalties, if any, in connection with the liquidation of a Mortgage Loan or following 
conversion of a Ginnie Mae Construction Loan Certificate prior to the Ginnie Mae Certificate Payment Date, the Trust will not distribute any such amounts until the Distribution Date 
immediately following receipt of such amounts. The Holders of the Securities will not be entitled to receive any interest on such amounts, and the related WACR will be reduced accordingly.

### The Ginnie Mae Project Loan Certificates

All Ginnie Mae Project Loan Certificates issued prior to August 1, 1997 will have an original term to maturity of not more than 40 years and will be based on and backed by one multifamily 
Mortgage Loan. Ginnie Mae Project Loan Certificates issued after August 1, 1997, may be based on or backed by more than one multi-family Mortgage Loan.
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:06.249 DEBUG    livekit.agents     tools execution completed {"speech_id": "speech_c521790e881a"}
   MacBook Pro Microphone   ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                                                                                                                                        
                                                                                                                                                                                               
    08:16:07.884 DEBUG    livekit.agents     aec warmup active, disabling interruptions for 3.00s  
   MacBook Pro Microphone          ▁ ▁ ▁ ▁ ▃ ▅ ▄ ▃ ▁ ▂ ▁ ▁ ▁ ▁                             end_of_turn: 1020ms · turn_completed_cb: 0.00ms · llm_ttft: 1249ms · tts_ttfb: 133ms  ─  e2e: 4264ms
                                                                                                                                                                                               
    08:16:10.890 DEBUG    livekit.agents     aec warmup expired, re-enabling interruptions  
   MacBook Pro Microphone          ▁ ▁ ▁ ▆ ▂ ▆ ▁ ▃ ▁ ▁ ▁ ▁ ▁ ▁                             end_of_turn: 1020ms · turn_completed_cb: 0.00ms · llm_ttft: 1249ms · tts_ttfb: 133ms  ─  e2e: 4264ms
                                                                                                                                                                                               
    08:16:23.004 DEBUG    livekit.agents     conversation_item_added  
                                         {"role": "assistant", "text": "FHA Section 223(f) allows for federal insurance of mortgage loans to purchase or refinance existing multifamily housing
that doesn't require substantial rehabilitation. It aims to provide lower debt service or support affordable rental housing."}
   MacBook Pro Microphone          ▁ ▁ ▁ ▃ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁ ▁                             end_of_turn: 1020ms · turn_completed_cb: 0.00ms · llm_ttft: 1249ms · tts_ttfb: 133ms  ─  e2e: 4264ms
                                                                                                                                                                                               
   ? for shortcuts                                                                                                                                                                             

The Conclusion

Conclusion

This was a very intense, deep hands-on session where the article explained how to build a complete Voice-Enabled RAG Agent from the ground up. We looked at how a user’s speech can be captured in real time, converted into text, passed through an LLM, enriched with relevant context retrieved from the Qdrant Vector Search Engine, and finally converted back into a natural voice response.

We also explored how LiveKit brings the real-time voice experience together, how LlamaIndex connects the agent with the knowledge layer, and how MLflow with OpenTelemetry gives us the traceability needed to understand what is happening inside the agent during every interaction.

The key takeaway is that a production-ready voice agent is not just about connecting speech-to-text and text-to-speech. The real value comes from combining voice interaction, semantic retrieval, grounded generation, and observability into one complete flow. This architecture gives us a strong foundation for building reliable voice assistants for domains such as mortgage, finance, healthcare, customer support, and other knowledge-intensive applications.

With this implementation in place, we now have a Voice RAG system that can listen, retrieve, reason, respond, and trace every interaction end to end.

Observable Voice RAG Agents: Tracing RAG Agents End-to-End with MLflow and Qdrant was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Architecture:

This architecture is basically a straight line from raw document to answer, with two branches for how the content gets embedded.

Source Document is where it starts — PDF filings, SEC 10-Ks, mortgage docs, whatever dense text-heavy file you’re working with.

Parse + Chunk breaks that document into smaller pieces. Nothing fancy here, just page and paragraph-level splitting. The idea is not to over-engineer this step, because the next step does most of the real work.

Metadata Enrichment is the important one. Each chunk gets tagged with things like entity, section, and reporting period. This is what lets you filter later instead of relying purely on vector similarity, which tends to blur together content that’s semantically similar but actually belongs to different brands, companies, or sections.

From there, each chunk goes through two embedding paths at the same time:

• Dense Embedding captures meaning — good for queries like “risk factors” even if the document says “principal risks” instead.
• Sparse Embedding captures exact keyword matches — good for things like specific terms, ticker symbols, or ingredient/product names that dense embeddings tend to under-weight.

Both vectors, along with the metadata, get stored together in Qdrant. That’s the key part: it’s not vectors in one place and metadata in another. They live on the same point, and Qdrant indexes the metadata right alongside the vector index, so filtering happens during the search itself, not after.

The Query Layer is where everything comes together — hybrid search (dense + sparse) combined with a metadata filter, fused using RRF, all in a single query.

And finally, that result goes to the Application Layer / User — whatever’s consuming it, a chatbot, a search UI, or an internal tool.

The short version: chunk it, tag it, embed it two ways, store it with its metadata, then query with both semantic search and hard filters at once instead of picking one or the other.

The Implementation:

import os

DATA_DIR = "data/10k_filings"
os.makedirs(DATA_DIR, exist_ok=True)

FILINGS = {
    "AAPL": os.path.join(DATA_DIR, "AAPL_10K_FY24.pdf"),
    "MSFT": os.path.join(DATA_DIR, "MSFT_10K_FY24.pdf"),
    "HON":  os.path.join(DATA_DIR, "HON_10K_FY24.pdf"),
    "GE":   os.path.join(DATA_DIR, "GE_10K_FY24.pdf"),
}

for ticker, path in FILINGS.items():
    exists = os.path.exists(path)
    print(f"{ticker}: {'found' if exists else 'MISSING'} at {path}")

The above code sets up a folder for filings and defines a dictionary mapping each company ticker to its 10-K PDF path. It then checks each file actually exists on disk before the rest of the pipeline touches it. You would place your downloaded 10-K PDFs, from SEC EDGAR or wherever you source them, into that folder ahead of running this.

import fitz
from typing import List, Dict

MAX_CHUNK_CHARS = 800


def parse_document(pdf_path: str, ticker: str) -> List[Dict]:
    doc = fitz.open(pdf_path)
    chunks = []
    for page_num, page in enumerate(doc):
        text = page.get_text()
        for para in text.split("\n\n"):
            para = para.strip()
            if len(para) <= 100:
                continue
            for i in range(0, len(para), MAX_CHUNK_CHARS):
                piece = para[i:i + MAX_CHUNK_CHARS].strip()
                if len(piece) > 100:
                    chunks.append({
                        "text": piece,
                        "page": page_num + 1,
                        "ticker": ticker,
                    })
    return chunks


raw_chunks = []
for ticker, path in FILINGS.items():
    if os.path.exists(path):
        raw_chunks.extend(parse_document(path, ticker))

print(f"Parsed {len(raw_chunks)} chunks across {len(FILINGS)} filings")

The above code opens each PDF page by page, splits the page text on blank lines to get paragraphs, and drops anything under 100 characters since that’s usually noise like headers or page numbers. Any paragraph longer than 800 characters gets sliced into smaller pieces so no single chunk is too large. Every chunk gets stamped with its page number and the ticker of the filing it came from, and the loop at the bottom runs this once per company so all four filings end up in one combined list.

import re
from collections import Counter

ITEM_PATTERNS = {
    "Item 1 - Business": r"item\s+1\.?\s+business",
    "Item 1A - Risk Factors": r"item\s+1a\.?\s+risk\s+factors",
    "Item 3 - Legal Proceedings": r"item\s+3\.?\s+legal\s+proceedings",
    "Item 7 - MD&A": r"item\s+7\.?\s+management.s\s+discussion",
    "Item 7A - Market Risk": r"item\s+7a\.?\s+quantitative\s+and\s+qualitative",
    "Item 8 - Financial Statements": r"item\s+8\.?\s+financial\s+statements",
    "Item 9A - Controls and Procedures": r"item\s+9a\.?\s+controls\s+and\s+procedures",
}


def classify_item(text: str) -> str:
    low = text.lower()
    for item, pattern in ITEM_PATTERNS.items():
        if re.search(pattern, low):
            return item
    return "General"


def enrich_chunk(chunk: Dict, fiscal_year: str) -> Dict:
    text = chunk["text"]
    item = classify_item(text)

    chunk["payload"] = {
        "ticker": chunk["ticker"],
        "document_section": item,
        "page": chunk["page"],
        "fiscal_year": fiscal_year,
    }
    return chunk


enriched_chunks = [enrich_chunk(c, "FY24") for c in raw_chunks]

print(f"Total chunks: {len(enriched_chunks)}")
print("\nItem distribution:")
print(Counter(c["payload"]["document_section"] for c in enriched_chunks))
print("\nChunks per ticker:")
print(Counter(c["payload"]["ticker"] for c in enriched_chunks))

The above code attaches structured metadata to every chunk. classify_item checks the chunk text against a set of regex patterns for the standard 10-K Item headers, Item 1A for risk factors, Item 7 for MD&A, and so on, and returns whichever one matches. Since 10-K numbering is mandated by SEC Regulation S-K, this is a reliable check regardless of which company filed it. The ticker doesn't need any detection logic here since it's already known from which file the chunk came from, it just gets carried into the payload alongside the section label, page number, and fiscal year. The print statements at the end just give you a sanity check on how chunks are distributed across items and across companies.

from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

texts = [c["text"] for c in enriched_chunks]
dense_vectors = embedding_model.encode(texts, show_progress_bar=True, batch_size=64)

VECTOR_SIZE = dense_vectors.shape[1]
print(f"Embedded {len(texts)} chunks into {VECTOR_SIZE}-dim dense vectors")

The above code loads a small local embedding model and runs every chunk’s text through it to produce dense vectors, 384 numbers per chunk that capture meaning rather than exact wording. This is what lets a query like “risk factors” also match a chunk that says “principal risks” instead. Since there are four filings’ worth of text here, this step will take noticeably longer to run than it would on a single document.

from fastembed import SparseTextEmbedding

sparse_model = SparseTextEmbedding(model_name="Qdrant/bm25")
sparse_vectors = list(sparse_model.embed(texts))

print(f"Generated {len(sparse_vectors)} sparse (BM25) vectors")

The above code generates a second type of vector for every chunk, this time using BM25, a statistical method that rewards exact keyword matches instead of semantic closeness. This matters a lot for 10-Ks specifically, since filings are full of exact tokens like accounting standard codes, litigation names, and regulatory citations that a dense embedding model tends to smooth over rather than treat as distinct.

from qdrant_client import QdrantClient
from qdrant_client.models import (
    VectorParams, Distance, SparseVectorParams, PointStruct, SparseVector
)

COLLECTION_NAME = "sec_10k_filings"

client = QdrantClient(url="http://localhost:6333", api_key="****")

if client.collection_exists(COLLECTION_NAME):
    client.delete_collection(COLLECTION_NAME)

client.create_collection(
    collection_name=COLLECTION_NAME,
    vectors_config={"dense": VectorParams(size=VECTOR_SIZE, distance=Distance.COSINE)},
    sparse_vectors_config={"sparse": SparseVectorParams()},
)

points = []
for idx, (chunk, dense_vec, sparse_vec) in enumerate(zip(enriched_chunks, dense_vectors, sparse_vectors)):
    points.append(
        PointStruct(
            id=idx,
            vector={
                "dense": dense_vec.tolist(),
                "sparse": SparseVector(indices=sparse_vec.indices.tolist(), values=sparse_vec.values.tolist()),
            },
            payload={"text": chunk["text"], **chunk["payload"]},
        )
    )

client.upsert(collection_name=COLLECTION_NAME, points=points)
print(f"Upserted {len(points)} points into '{COLLECTION_NAME}'")

The above code starts an in-memory Qdrant instance and creates a collection configured to hold two named vectors per point, one dense and one sparse. It then loops through every chunk and builds a PointStruct combining the chunk's dense vector, its sparse vector, and its full metadata payload, all attached to the same point ID. The final upsert call writes everything into Qdrant in one batch. Keeping the vectors and the metadata on the same point is what lets Qdrant build its payload index right alongside the similarity index, so filtering by ticker or section can happen during the search itself rather than as a separate step afterward.

def show_results(results, title):
    print(f"--- {title} ---")
    for p in results.points:
        print(f"score={p.score:.3f}  ticker={p.payload['ticker']:<6}  "
              f"section={p.payload['document_section']:<30} page={p.payload['page']}")
        print("   ", p.payload["text"][:140].replace("\n", " "), "...")
    print()


query_text = "supply chain disruption and litigation risk"
query_vector = embedding_model.encode(query_text).tolist()

unfiltered_results = client.query_points(
    collection_name=COLLECTION_NAME, query=query_vector, using="dense", limit=5,
)

show_results(unfiltered_results, "UNFILTERED (pure semantic search)")

The above code defines a small helper to print search results cleanly, then runs a single query with no metadata filter at all, just plain dense vector search across the whole collection. Run it and the top five results will typically mix tickers freely, a chunk from Apple’s filing sitting right next to one from GE, because cosine similarity has no built-in concept of “only Microsoft” or “only this section.” That’s the gap the next cell closes.

from qdrant_client.models import Filter, FieldCondition, MatchValue

filtered_results = client.query_points(
    collection_name=COLLECTION_NAME,
    query=query_vector,
    using="dense",
    query_filter=Filter(
        must=[
            FieldCondition(key="ticker", match=MatchValue(value="MSFT")),
            FieldCondition(key="document_section", match=MatchValue(value="Item 1A - Risk Factors")),
        ]
    ),
    limit=5,
)

show_results(filtered_results, "FILTERED (ticker = MSFT, section = Item 1A)")

The above code runs the exact same query as the previous cell, but this time layers a hard filter on top, restricting results to Microsoft’s filing and specifically its Item 1A risk factors section. Any chunk that fails either condition is excluded before it even gets ranked, regardless of how high its similarity score would have been. This is the core of the whole pattern, a semantic query with structural constraints enforced as a database-style rule rather than a soft nudge.

from qdrant_client.models import Prefetch, FusionQuery, Fusion

hybrid_query_text = "ASC 606 revenue recognition"
hybrid_query_vector = embedding_model.encode(hybrid_query_text).tolist()
hybrid_query_sparse = list(sparse_model.embed([hybrid_query_text]))[0]

hybrid_results = client.query_points(
    collection_name=COLLECTION_NAME,
    prefetch=[
        Prefetch(query=hybrid_query_vector, using="dense", limit=20),
        Prefetch(
            query=SparseVector(indices=hybrid_query_sparse.indices.tolist(), values=hybrid_query_sparse.values.tolist()),
            using="sparse",
            limit=20,
        ),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=5,
)

show_results(hybrid_results, "HYBRID (dense + sparse RRF fusion, query = 'ASC 606 revenue recognition')")

The above code runs a query for an accounting standard citation, exactly the kind of literal token a dense model tends to under-weight since it cares more about general topic than exact phrasing. It fetches candidates from both the dense and sparse legs separately, then fuses the two ranked lists together using Reciprocal Rank Fusion so a chunk that scores well on either signal has a chance to surface in the final top five.

FILTER_TEST_QUERIES = [
    {"query": "product liability and litigation exposure", "ticker": "GE", "label": "GE / Item 3"},
    {"query": "cybersecurity and data protection risk", "ticker": "AAPL", "label": "AAPL / Item 1A"},
    {"query": "interest rate and currency exposure", "ticker": "HON", "label": "HON / Item 7A"},
]

filter_test_results = []
for tc in FILTER_TEST_QUERIES:
    q_vec = embedding_model.encode(tc["query"]).tolist()
    q_filter = Filter(must=[FieldCondition(key="ticker", match=MatchValue(value=tc["ticker"]))])

    unfiltered = client.query_points(collection_name=COLLECTION_NAME, query=q_vec, using="dense", limit=5)
    filtered = client.query_points(collection_name=COLLECTION_NAME, query=q_vec, using="dense", query_filter=q_filter, limit=5)

    hits_unfiltered = sum(p.payload["ticker"] == tc["ticker"] for p in unfiltered.points)
    hits_filtered = sum(p.payload["ticker"] == tc["ticker"] for p in filtered.points)

    filter_test_results.append({
        "label": tc["label"],
        "unfiltered_rate": hits_unfiltered / 5,
        "filtered_rate": hits_filtered / 5,
    })

for r in filter_test_results:
    print(f"{r['label']:<16} unfiltered={r['unfiltered_rate']:.2f}  filtered={r['filtered_rate']:.2f}")

The above code runs three different company-scoped queries and, for each one, compares how often the unfiltered search happens to land on the right company versus how often the filtered search does, out of its top five results. The unfiltered rate will vary from query to query since it’s essentially getting lucky or unlucky depending on how much the filings overlap in language, while the filtered rate should sit at 1.0 every time, since a hard filter simply doesn’t allow a chunk from the wrong ticker through no matter how relevant it looks otherwise.

If you want, I can also do the charting cell that turns this comparison into the two bar charts, or extend the filter tests to combine ticker and Item section together in the same query.

The Conclusion

We have seen how combining dense embeddings, sparse BM25, and metadata as a hard filter closes a gap that plain vector search cannot close on its own, unfiltered semantic search kept mixing tickers and sections together since cosine similarity has no concept of “only this company” or “only this Item,” while layering a metadata filter on top pushed precision to a flat 100% every time, and the sparse leg separately rescued literal terms like accounting standard citations that dense embeddings tend to smooth over. This implementation can improve retrieval quality anywhere a corpus reuses similar language across distinct entities, filings across companies, loan documents across borrowers, annual reports across brands, and because Qdrant indexes the payload right alongside the vector index, this precision doesn’t come at the cost of a slower search, filter and retrieval happen together in a single call, making it a practical pattern to carry into a production RAG pipeline rather than just a notebook experiment.

Dense + Sparse + Filter: Building Precision Retrieval for Annual Reports and 10-Ks was originally published in Towards Dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is BM25 ?

BM25, or Best Matching 25, is a keyword-based ranking algorithm used in search engines. It ranks documents by considering:

• How frequently the query terms appear in a document
• How rare or important those terms are across the document collection
• The length of the document

For example, when searching for “vector database”, BM25 gives higher scores to documents containing both terms, especially when those words are uncommon and appear prominently. However, BM25 matches words statistically and does not understand their contextual meaning.

What is miniCOIL?

miniCOIL is Qdrant’s open-source sparse neural retrieval model. It extends BM25-style keyword matching by considering the contextual meaning of matching words.

For example, for the query “vectors in medicine”, BM25 may treat every occurrence of the word vector similarly. miniCOIL can understand that vector in “vector-borne diseases” is more relevant than vector in graphics or mathematics.

miniCOIL is inspired by the COIL approach and can be understood as:

BM25-style keyword retrieval with contextual understanding.

Because miniCOIL still relies on overlapping words, it is commonly combined with dense embeddings in hybrid search to retrieve documents that use different terminology.

The Architecture

For this experiment, I designed a controlled benchmarking architecture to compare two hybrid retrieval approaches in Qdrant. The first approach combines dense embeddings with BM25, while the second combines the same dense embeddings with miniCOIL. My intention was to keep every other part of the pipeline unchanged so that the final comparison would clearly show the impact of replacing BM25 with miniCOIL.

The experiment begins with the MedQuAD dataset, which contains medical questions, answers, and their corresponding contexts. During ingestion, I use only the contextual content as the searchable knowledge base. The questions and expected answers are maintained separately for validation. This separation is important because the benchmark should evaluate whether the retrieval system can find the correct medical context for a given question, rather than simply matching against the question-answer pairs already stored in the database.

The ingester processes each MedQuAD context and prepares it for storage in two separate Qdrant collections. For both collections, I generate dense vectors using the embeddinggemma-300m model. Using the same dense embedding model ensures that the semantic retrieval component remains identical across the two experiments. The only component that changes is the sparse retrieval method.

The first Qdrant collection uses a combination of embeddinggemma-300m dense vectors and BM25 sparse vectors. BM25 represents the traditional lexical side of hybrid retrieval. It rewards documents that contain query terms based on factors such as term frequency, document length, and inverse document frequency. This collection therefore combines semantic similarity from the dense vectors with conventional keyword matching from BM25.

The second collection uses the same embeddinggemma-300m dense vectors, but replaces BM25 with miniCOIL. Unlike traditional BM25, miniCOIL does not treat every occurrence of a matching word in exactly the same way. It generates contextualized sparse representations, allowing the retrieval system to consider the meaning of matching terms within the query and document. This makes the second collection an interesting alternative to the conventional dense-plus-BM25 setup.

Once both collections are populated with the same MedQuAD contexts, the benchmark component reads questions from the MedQuAD validation dataset. Each validation question is submitted independently to both collections. The first retrieval run searches the dense-plus-BM25 collection, while the second searches the dense-plus-miniCOIL collection. Since both systems receive exactly the same questions and search over exactly the same content, the comparison remains fair and reproducible.

For every query, the benchmark component records the ranked contexts returned by both retrieval configurations. These results are then compared with the expected context or answer associated with the validation question. The evaluation measures how often the correct context appears among the top retrieved results, how highly it is ranked, and how much time each retrieval approach requires.

In simple terms, the architecture creates two parallel retrieval pipelines. Both pipelines use the same dataset, the same dense embedding model, the same Qdrant environment, and the same validation questions. The only deliberate difference is the sparse retrieval mechanism: BM25 in the first pipeline and miniCOIL in the second. This controlled setup allows me to determine whether miniCOIL provides a measurable improvement over BM25 when combined with dense retrieval for medical question answering.

The Implementation

This script prepares the MedQuAD medical dataset for hybrid search in Qdrant using both dense embeddings and BM25 sparse vectors. It first downloads the dataset from Hugging Face and extracts only the medical context from each record. Every context is stored with a simple payload containing a document ID and the original text.

Next, the script calculates the average document length, which BM25 uses during scoring. It then generates dense semantic embeddings for all contexts using google/embeddinggemma-300m. These dense vectors capture the meaning of the medical text, while BM25 captures exact keyword relevance.

The script creates a Qdrant collection named medquad_dense_bm25 with two vector types: a dense vector using cosine similarity and a sparse BM25 vector using the IDF modifier. Each MedQuAD context is then stored as a Qdrant point containing the original text, its dense embedding, and its BM25 representation.

The documents are uploaded to Qdrant in batches of 16. To make ingestion faster, indexing is temporarily delayed while the data is inserted. Once all documents are uploaded, the script triggers indexing and waits until the Qdrant collection reaches the green state.

In simple terms, this code converts the MedQuAD contexts into a searchable Qdrant knowledge base that supports hybrid retrieval by combining semantic search from EmbeddingGemma with keyword search from BM25.

from datasets import load_dataset
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient, models

# --- config ---
HF_REPO_ID = "pavanmantha/MedQuAD_Context_GroupedQA"
SPLIT = "train"
MODEL_NAME = "google/embeddinggemma-300m"
BM25_MODEL = "Qdrant/bm25"
COLLECTION_NAME = "medquad_dense_bm25"
QDRANT_URL = "http://localhost:6333"
API_KEY = "API KEY"
BATCH_SIZE = 16
RECREATE = False


def load_contexts():
    """Pull the grouped dataset and return (contexts, payloads)."""
    ds = load_dataset(HF_REPO_ID, split=SPLIT)
    contexts, payloads = [], []
    for i, row in enumerate(ds):
        ctx = row["context"]
        contexts.append(ctx)
        payloads.append({
            "doc_id": f"doc_{i}",
            "text": ctx,
        })
    return contexts, payloads


def compute_avg_length(contexts):
    """Average word count across the corpus, part of BM25's scoring formula."""
    return sum(len(c.split()) for c in contexts) / len(contexts)


def embed_dense(contexts):
    """Precompute dense EmbeddingGemma vectors, same as the original dense-only pipeline."""
    print(f"Embedding {len(contexts)} contexts with {MODEL_NAME}...")
    model = SentenceTransformer(MODEL_NAME, device="mps")
    embeddings = model.encode_document(
        contexts, batch_size=BATCH_SIZE, show_progress_bar=True
    )
    vectors = [vec.tolist() for vec in embeddings]
    dim = len(vectors[0])
    print(f"Got {len(vectors)} dense vectors of dim {dim}")
    return vectors, dim


def create_collection(client, collection_name, dense_dim):
    if RECREATE and client.collection_exists(collection_name):
        client.delete_collection(collection_name)

    if not client.collection_exists(collection_name):
        client.create_collection(
            collection_name=collection_name,
            vectors_config={
                "dense": models.VectorParams(
                    size=dense_dim,
                    distance=models.Distance.COSINE,
                    hnsw_config=models.HnswConfigDiff(
                        m=64,
                        ef_construct=200,
                    ),
                ),
            },
            sparse_vectors_config={
                "bm25": models.SparseVectorParams(
                    modifier=models.Modifier.IDF,
                ),
            },
            optimizers_config=models.OptimizersConfigDiff(
                indexing_threshold=100000,
            ),
        )


def build_points(contexts, payloads, dense_vectors, avg_len):
    points = []
    for i, (ctx, payload, dvec) in enumerate(zip(contexts, payloads, dense_vectors)):
        points.append(
            models.PointStruct(
                id=i,
                payload=payload,
                vector={
                    "dense": dvec,
                    "bm25": models.Document(
                        text=ctx,
                        model=BM25_MODEL,
                        options={"avg_len": avg_len},
                    ),
                },
            )
        )
    return points


def upsert_points(client, collection_name, points):
    print("1. Upserting points (dense + bm25)...")
    for start in range(0, len(points), BATCH_SIZE):
        batch = points[start:start + BATCH_SIZE]
        client.upsert(collection_name=collection_name, points=batch)
        print(f"  upserted {start + len(batch)}/{len(points)}")


def trigger_indexing(client, collection_name):
    print("2. All points upserted. Triggering indexing...")
    client.update_collection(
        collection_name=collection_name,
        optimizers_config=models.OptimizersConfigDiff(
            indexing_threshold=1000,
        ),
    )

    import time
    while True:
        info = client.get_collection(collection_name)
        if info.status.value == "green":
            print("Index ready — collection is green")
            break
        print(f"  status: {info.status.value} — waiting...")
        time.sleep(5)
    print("3. indexing done")


def main():
    contexts, payloads = load_contexts()
    avg_len = compute_avg_length(contexts)
    dense_vectors, dense_dim = embed_dense(contexts)

    client = QdrantClient(url=QDRANT_URL, api_key=API_KEY)
    create_collection(client, COLLECTION_NAME, dense_dim)

    points = build_points(contexts, payloads, dense_vectors, avg_len)
    upsert_points(client, COLLECTION_NAME, points)
    trigger_indexing(client, COLLECTION_NAME)


if __name__ == "__main__":
    main()

This script builds the second retrieval pipeline in the experiment, where dense semantic search is combined with miniCOIL instead of BM25. It downloads the MedQuAD dataset from Hugging Face, extracts the medical context from each record, and prepares the original text and document ID as payload data for Qdrant.

The dense representation is generated using google/embeddinggemma-300m. To avoid recomputing the same embeddings every time the script runs, the vectors are saved in a local .cache folder using a pickle file. On later runs, the script checks whether the cached vectors match the current dataset size and loads them directly. This makes repeated ingestion much faster.

Since both EmbeddingGemma and miniCOIL may consume significant memory, the script releases the dense embedding model after the vectors are generated. It clears Python memory and the Apple MPS cache before miniCOIL is used. This prevents memory conflicts when running both models in the same process on a Mac.

The script calculates the average document length because miniCOIL uses a BM25-based scoring structure. It then creates a Qdrant collection called medquad_dense_minicoil with two named vector spaces. The dense vector stores the EmbeddingGemma representation and uses cosine similarity, while the minicoil vector stores the contextual sparse representation generated using Qdrant/minicoil-v1. The miniCOIL vector is also configured with the required IDF modifier.

Each MedQuAD context is stored as a Qdrant point containing its payload, dense embedding, and miniCOIL representation. The documents are uploaded in batches of 16. Index construction is initially delayed to make ingestion faster, and after all the records are inserted, the indexing threshold is lowered so Qdrant can build the final search index.

The script then waits until the collection reaches the green state, confirming that indexing has finished successfully. In simple terms, this code creates a searchable MedQuAD collection that combines semantic understanding from EmbeddingGemma with context-aware lexical matching from miniCOIL. This collection forms the miniCOIL side of the comparison against the dense-plus-BM25 pipeline.

import os

os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")  # avoid HF tokenizer fork after MPS init

import gc
import pickle
from pathlib import Path

from datasets import load_dataset
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient, models

# --- config ---
HF_REPO_ID = "pavanmantha/MedQuAD_Context_GroupedQA"
SPLIT = "train"
MODEL_NAME = "google/embeddinggemma-300m"
MINICOIL_MODEL = "Qdrant/minicoil-v1"
COLLECTION_NAME = "medquad_dense_minicoil"
QDRANT_URL = "http://localhost:6333"
API_KEY = "API key"
BATCH_SIZE = 16
RECREATE = False

CACHE_DIR = Path(__file__).resolve().parent / ".cache"
CACHE_DIR.mkdir(exist_ok=True)
print(f"Cache dir: {CACHE_DIR}")


def load_contexts():
    """Pull the grouped dataset and return (contexts, payloads)."""
    ds = load_dataset(HF_REPO_ID, split=SPLIT)
    contexts, payloads = [], []
    for i, row in enumerate(ds):
        ctx = row["context"]
        contexts.append(ctx)
        payloads.append({
            "doc_id": f"doc_{i}",
            "text": ctx,
        })
    return contexts, payloads


def compute_avg_length(contexts):
    """Average word count across the corpus, used by miniCOIL's BM25-based scoring."""
    return sum(len(c.split()) for c in contexts) / len(contexts)


def embed_dense(contexts):
    """Precompute dense EmbeddingGemma vectors, cached to disk, model released
    from MPS before returning so it isn't still alive when minicoil's ONNX
    model loads during upsert."""
    cache_path = CACHE_DIR / "dense.pkl"
    if cache_path.exists():
        with open(cache_path, "rb") as f:
            cached_len, vectors = pickle.load(f)
        if cached_len == len(contexts):
            print(f"Loaded {cached_len} cached dense vectors from {cache_path}")
            return vectors, len(vectors[0])
        print(f"Cache is for {cached_len} contexts, need {len(contexts)} — recomputing")

    print(f"Embedding {len(contexts)} contexts with {MODEL_NAME}...")
    model = SentenceTransformer(MODEL_NAME, device="mps")
    embeddings = model.encode_document(
        contexts, batch_size=BATCH_SIZE, show_progress_bar=True
    )
    vectors = [vec.tolist() for vec in embeddings]
    dim = len(vectors[0])
    print(f"Got {len(vectors)} dense vectors of dim {dim}")

    # Release before minicoil's ONNX model ever loads in this same process
    del model
    gc.collect()
    try:
        import torch
        torch.mps.empty_cache()
    except Exception:
        pass

    with open(cache_path, "wb") as f:
        pickle.dump((len(vectors), vectors), f)
    print(f"Cached dense vectors to {cache_path}")

    return vectors, dim


def create_collection(client, collection_name, dense_dim):
    if RECREATE and client.collection_exists(collection_name):
        client.delete_collection(collection_name)

    if not client.collection_exists(collection_name):
        client.create_collection(
            collection_name=collection_name,
            vectors_config={
                "dense": models.VectorParams(
                    size=dense_dim,
                    distance=models.Distance.COSINE,
                    hnsw_config=models.HnswConfigDiff(
                        m=64,
                        ef_construct=200,
                    ),
                ),
            },
            sparse_vectors_config={
                "minicoil": models.SparseVectorParams(
                    modifier=models.Modifier.IDF,
                ),
            },
            optimizers_config=models.OptimizersConfigDiff(
                indexing_threshold=100000,
            ),
        )


def build_points(contexts, payloads, dense_vectors, avg_len):
    points = []
    for i, (ctx, payload, dvec) in enumerate(zip(contexts, payloads, dense_vectors)):
        points.append(
            models.PointStruct(
                id=i,
                payload=payload,
                vector={
                    "dense": dvec,
                    "minicoil": models.Document(
                        text=ctx,
                        model=MINICOIL_MODEL,
                        options={"avg_len": avg_len},
                    ),
                },
            )
        )
    return points


def upsert_points(client, collection_name, points):
    print("1. Upserting points (dense + minicoil)...")
    for start in range(0, len(points), BATCH_SIZE):
        batch = points[start:start + BATCH_SIZE]
        client.upsert(collection_name=collection_name, points=batch)
        print(f"  upserted {start + len(batch)}/{len(points)}")


def trigger_indexing(client, collection_name):
    print("2. All points upserted. Triggering indexing...")
    client.update_collection(
        collection_name=collection_name,
        optimizers_config=models.OptimizersConfigDiff(
            indexing_threshold=1000,
        ),
    )

    import time
    while True:
        info = client.get_collection(collection_name)
        if info.status.value == "green":
            print("Index ready — collection is green")
            break
        print(f"  status: {info.status.value} — waiting...")
        time.sleep(5)
    print("3. indexing done")


def main():
    contexts, payloads = load_contexts()
    avg_len = compute_avg_length(contexts)
    dense_vectors, dense_dim = embed_dense(contexts)

    client = QdrantClient(url=QDRANT_URL, api_key=API_KEY)
    create_collection(client, COLLECTION_NAME, dense_dim)

    points = build_points(contexts, payloads, dense_vectors, avg_len)
    upsert_points(client, COLLECTION_NAME, points)
    trigger_indexing(client, COLLECTION_NAME)


if __name__ == "__main__":
    main()

The Results

These benchmark results make the comparison much clearer. Both configurations successfully evaluated all 100 questions without errors. The dense-plus-BM25 pipeline achieved a contextual precision of 0.879 and contextual recall of 0.840, while the dense-plus-miniCOIL pipeline improved these scores to 0.898 and 0.875, respectively.

In practical terms, miniCOIL improved contextual precision by approximately 1.93 percentage points and contextual recall by 3.51 percentage points. The larger improvement in recall suggests that miniCOIL was more successful at retrieving the complete set of medical information required to answer the validation questions. Its contextualized term representations appear to provide an advantage over traditional BM25 when matching specialised medical queries with relevant contexts.

Answer relevancy remained almost identical across both pipelines. BM25 achieved 0.3997, while miniCOIL achieved 0.3990. This difference is negligible and indicates that changing the sparse retrieval model improved retrieval quality but did not improve how the generation model used the retrieved information. The answer-generation stage remains a separate area that requires further optimisation.

{
  "bm25": {
    "n_scored": 100,
    "n_errors": 0,
    "avg_contextual_precision": 0.8785067433768732,
    "avg_contextual_recall": 0.8398373220591189,
    "avg_answer_relevancy": 0.39968247900511716
  },
  "minicoil": {
    "n_scored": 100,
    "n_errors": 0,
    "avg_contextual_precision": 0.8978049792139078,
    "avg_contextual_recall": 0.8749693639693639,
    "avg_answer_relevancy": 0.3989726123549653
  }
}

The Conclusion

In this article, we learned what BM25 is, how miniCOIL extends lexical retrieval with contextual understanding, and how both approaches can be combined with dense embeddings to build hybrid search pipelines in Qdrant. We designed an experimental architecture using the MedQuAD dataset, EmbeddingGemma for dense representations, and two separate Qdrant collections to compare dense retrieval combined with BM25 and miniCOIL.

We also implemented the complete workflow, from loading and preparing the dataset to generating vectors, ingesting them into Qdrant, executing validation queries, and evaluating the retrieved results. By keeping the dataset, dense embedding model, validation questions, and Qdrant configuration consistent, the experiment created a fair environment for studying the effect of changing only the sparse retrieval model.

The benchmark produced strong retrieval results, with an average contextual precision of 0.879 and contextual recall of 0.840. These scores show that the system was generally successful in retrieving and ranking the medical contexts required to answer the validation questions. However, the lower answer relevancy score also highlighted an important lesson: retrieving the correct context does not automatically guarantee a correct final answer. The generation model must still understand, follow, and accurately use the retrieved evidence.

Overall, BM25 remains a reliable and efficient choice when exact keyword matching is important, while miniCOIL introduces contextual awareness into sparse retrieval and can better distinguish the meaning of matching terms. The right choice depends on the dataset, query patterns, latency requirements, and domain. Rather than selecting a retrieval technique based only on its popularity, this experiment shows why it is important to benchmark each configuration using real domain data and measurable evaluation metrics.

Hybrid Retrieval Under the Microscope: BM25 vs MiniCOIL on MedQuAD was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.