The Power of Contextual Datasets: Quantifying Memory Retrieval Performance

8 min readNov 10, 2025

Posted by Sandeep Kumar Nayak, AI Engineer of Gaudiy Inc.

Introduction

In recent years, with the growing adoption of LLM-based conversational AI, “contextual understanding of conversation” has become increasingly important. However, improving this ability takes more than new models. True contextual understanding depends on the quality of the dataset that trains and evaluates it. Real conversations are rarely self-contained — they rely on shared history, emotional tone, and subtle references. A strong dataset captures these nuances, enabling models to retrieve the right memories and respond meaningfully. In short, advancing contextual AI starts with data that reflects how people actually communicate. In this article, we introduce our approach to improving memory retrieval through contextual queries at Gaudiy by leveraging a carefully designed dataset. This dataset enables us to quantify performance on contextual memory query tasks and drive meaningful improvements in product quality.

About Gaudiy and Its Product

At Gaudiy, we’re integrating AI-based interactive chat agents into our products to enable meaningful, sustained conversations with users. Our goal is to make these AI-driven interactions feel deeply personal, responsive, and memorable.

The Gaudiy AI team consists of researchers and engineers working across natural language processing, retrieval systems, and human-AI interaction design. Our mission is to build emotionally intelligent AI that adapts to users in real time — leveraging memory and personalization to create rich, dynamic conversations. We combine academic research with product-driven experimentation to bring innovations from lab to life. One of our current focus areas involves experimental conversational features that allow users to engage with AI agents representing specific characters in collaboration with rights holders.

However, user messages are often vague or context-light (e.g., “Nice!” or “Really?”). In such cases, the AI risks retrieving irrelevant or generic content unless it understands the message in the context of recent dialogue. This presents a fundamental challenge in AI conversations: handling ambiguous user inputs accurately.

To tackle this, one of our key areas of improvement is the chat agent’s ability to understand and respond appropriately based on the context of prior conversations. A crucial part of this personalized experience is the AI’s capacity to retain and retrieve long-term memories — shaping ongoing dialogues into richer, more coherent, and contextually relevant interactions.

Why Contextual Queries Matter — and Why the Right Dataset Is Key

Understanding context lies at the heart of delivering meaningful conversational AI experiences. In some setups, such as when the entire prior conversation is stored directly in the prompt, there may be no need for separate memory retrieval steps.

However, to make systems scalable and cost-efficient, most agents rely on short-term context windows. For deeper personalization and long-term coherence, we must store and retrieve relevant long-term memories separately. Identifying which memories are contextually relevant — and retrieving them appropriately — requires an extra step: memory querying. To evaluate how well a system can perform this step, the role of the dataset becomes crucial.

We aim for our dataset to support evaluation across key dimensions that matter for real-world deployment:

More appropriate responses: Contextual understanding allows the AI to interpret queries in light of recent conversation history — improving clarity, reducing miscommunication, and increasing user engagement.
Memory retrieval accuracy: A well-structured dataset helps benchmark how effectively AI agents retrieve relevant long-term memories.
Quantitative Comparison across models: With ground truth and structured dialogue, our dataset enables model benchmarking using rigorous metrics such as MAP@10 .
Model selection trade-offs: Our dataset lets us balance performance with cost and latency, guiding embedding model choices based on quantified results.

Illustrative Example: Why Context & Memory Matters

Consider this snippet from a conversation between AI assistant and a user.

User: Have you read any new books recently?
AI: Yes, I recently started reading Harry Potter.
User: That's great. Which volume are you reading?
AI: The first one. It's really interesting!
User: Nice!

This ensures that the AI retrieves relevant details — such as prior mentions of Harry Potter — and can respond with something like: “Yes! I’m excited to continue the story tonight.” rather than a generic affirmation.

Long Term Memory History

Now consider a case where long-term memory comes into play

User: I’m feeling a bit anxious about tomorrow.

Here, the phrase “your big audition” was not mentioned in the current chat. It was something the user mentioned weeks ago:

Long-Term Memory in Database: "User is preparing for a regional theater
audition on June 10th. It's important to them."

Retrieving this information allows the AI to reference it meaningfully, even though it’s outside the recent dialogue window. Without this retrieval, the AI would respond generically. With long-term memory, it might say:

“You’ve worked so hard for this audition — I believe in you.

What kind of dataset helps here?

It includes multiple turns of chat history.
It contains annotations or ground-truth for the intended memory reference. ie It simulates long-term memory entries linked to prior user sessions.
It contains evaluation annotations about which memory entries are contextually relevant given ambiguous prompts.

This showcases the power of memory queries — not just for interpreting immediate context, but for maintaining emotional continuity and personalization over time.

In short, building a high-quality dataset isn’t just about data collection — it’s about unlocking measurable progress in contextual AI by enabling rich, context-aware evaluation

Building on Previous Work

In a prior internal experiment, one of our AI engineers explored contextual query rewriting techniques[1] — including single-shot, multi-shot, and hypothetical reply approaches. This exploration demonstrated how prompt engineering can significantly influence the relevance of memory retrieval, particularly by reducing embedding distances between user queries and relevant memories.

Cosine similarity remains a useful metric for evaluating retrieval effectiveness. This was complemented by qualitative checks — assessing whether the retrieved memory intuitively matched the user query within context. While this method supports rapid iteration and insight into model behavior, it is difficult to ensure consistent, comparable evaluation across different strategies.

Get Gaudiy Lab’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To address this, our current work introduces a dataset specifically designed for grounded and measurable evaluation of contextual memory retrieval. Unlike existing public datasets, it targets ambiguous or open-ended Japanese conversational queries, providing ground-truth expectations for what should be retrieved. This allows us to objectively compare retrieval approaches — for example, using metrics such as Mean Average Precision (MAP) — and conduct comprehensive, reproducible evaluations of model performance.

Our Solution

To address these gaps, we developed a private Japanese conversation dataset with the following features:

Rich conversation history across multiple turns.
Simulated long-term memory per user.
Personalization layers including personality traits

This dataset enables quantitative evaluation of how well embedding models perform when handling contextual memory retrieval tasks.

Methodology

1. Data Preparation

We created a context-enriched Japanese dialogue dataset involving:

- Multi-turn conversations.
- Ground-truth memory references.
- Annotated character personas.

Each user query in the dataset is paired with:

- A non-contextual version (standalone user input).
- A contextual version (query + prior conversation history).

This setup lets us measure retrieval quality under real-world, ambiguous input scenarios by directly comparing context-free and context-enriched queries.

2. Performance Metric

To evaluate retrieval, we used:
- Mean Average Precision @10 (MAP@10): Measures how well the top 10 retrieved memories align with ground-truth relevant memories.

Results

1. Contextual Queries Improve Model Performance

Our analysis revealed compelling evidence that our dataset plays a crucial role in demonstrating the need for context-aware memory retrieval. Unlike public datasets, which often lack conversational continuity, our dataset includes rich contextual traces — making it possible to observe the discrepancies in memory retrieval performance when using different embedding models.

Figure 1 : MAP@10 for contextual and non-contextual queries

We found that all models showed a clear improvement in precision when using contextual queries. Furthermore, some embedding models demonstrated significantly greater gains than others. Therefore, in Figure 2, we compare the relative performance improvements to identify which models achieved the highest increase.

Figure 2 : MAP@10 improvement across models of various sizes.

As from both the figures, we see, across all models, adding context to the queries (contextual queries) improved retrieval performance significantly:
Average improvement in MAP: +20%
Top model improvement: Nearly +30%

Insight 1: Even weaker models with poor non-contextual performance showed consistent gains — highlighting how contextual understanding universally boosts performance.

2. Model Selection via Dataset Evaluation

As from the Figure 1, open source models see huge jumps in MAP when context is included — bringing them close to top-tier models like:

OpenAI/text-embedding-3-large
Google_gekkko_001

Context Reduces the Dependence on Model Scale:

The performance advantage of expensive, large-scale models diminishes when contextual understanding is introduced at the query level.
This implies better ROI for real-world applications using smaller models with proper contextualization.

Insight 2
Use a smaller model
Pre-process user queries with contextual enrichment
Achieve similar accuracy with lower cost and latency

Conclusion

Through our newly introduced context-rich dataset, we have been able to quantitatively measure the performance of different text embedding models in our use case. The results not only validate the effectiveness of context-aware retrieval but also reveal each model’s capability to leverage conversational context — something that public datasets often fail to capture due to their lack of dialogue continuity.

While it has enabled evaluation of memory retrieval, we look ahead to also include the absence of end-to-end output scoring (e.g., user satisfaction) and unmeasured latency-cost tradeoffs — both of which we plan to address.

By enabling quantifiable evaluation on our dataset, we gain a surgical view into memory-retrieval methods and embedding models. Optimizing both in tandem yields systems that produce context-rich, personalized responses without inflating infrastructure cost — raising the bar for conversational AI beyond Gaudiy’s use case.

Looking ahead, we aim to integrate user-level evaluation metrics and investigate lower-cost, high-context embedding models to further improve intelligence and scalability. To ensure these results stand up to scrutiny, we are preparing to submit our findings for peer review. We’ll share updates as that process unfolds — stay tuned as we continue to push the boundaries of context-aware AI.