What I Found When I Went Looking

How I built an auditable deep research evaluation seed set from public human conversations

In the first article in this series, Real Traffic Is the Benchmark You Have Not Built Yet, I set out a framework for turning observed demand into durable evaluations through Selection, Curation, and Compression. This case study applies the first two stages to OpenResearcher under a cold-start constraint.

I started with a practical problem: OpenResearcher is open about its model, training trajectories, benchmarks, and search environment, but I could not find a public first-party dataset showing what people actually ask its demo [1][7]. Without that distribution, I could either use synthetic users or look for an external source of real human demand proxy.

I chose the second path. From 72,371 usable LMSYS and Chatbot Arena conversations, I built a five-stage selection funnel, independently audited accepted and rejected cases, corrected classification errors, and retained an 84-case seed set whose records I had individually reviewed. I then produced 336 WIDTH requirements and 167 DEPTH criteria grounded in 170 verified sources.

What I built

Figure 1 The implemented path from public conversations to a validated seed set and reference checklists

What the 84 cases contain

The final set spans three broad domains. Academic and scientific questions are the largest group. Business and market questions account for more than a third of the set, while self-directed learning forms a smaller third category. The information-seeking taxonomy includes 47 Learn tasks, 30 Investigate tasks, and seven Lookup boundary cases.

Figure 2 Aggregate composition by source, domain, and information-seeking activity

Of the 84 cases, 81 are answerable as written and three require clarification; I retain that result in prose because it is important operationally but too imbalanced to need its own chart panel. These counts describe the composition of the curated seed set, not the prevalence of research intents in LMSYS and not OpenResearcher usage.

Selection from 72371 conversations to 598 candidates

I first filtered LMSYS Chat 1M to English-language conversations and combined it with the English-by-construction Chatbot Arena release [2][3]. I retained first turns between 4 and 200 words, removing extremely short prompts and long pasted documents or code. The resulting 72,371 rows form the denominator for every funnel rate below.

I did not design the funnel perfectly in advance. After each stage, I read what remained, found a repeatable false-positive class, and tightened the next decision boundary. Inspection was part of the method, and the pipeline improved as those errors became visible.

What the taxonomy changed

My first operational taxonomy borrowed directly from Kuhlthau’s Information Search Process [6]. It was conceptually useful but too permissive for classification. Manual review showed that 46 to 52 percent of the two largest research-oriented subcategories were single-answer lookups: riddles, isolated concept explanations, or rewrites wrapped in academic language.

I replaced that ad-hoc split with Marchionini’s Lookup, Learn, and Investigate taxonomy [4]. The distinction focuses on the structure of the information need. Lookup has one bounded satisfying answer. Learn requires synthesis across subtopics. Investigate explores evidence toward a formal outcome. All four known failure cases from the earlier taxonomy resolved correctly under the new rule. Separately, I retained the domain-and-skill classification; the final set’s Academic/scientific, Business/market, and Self-directed skill-building categories are shown in the composition above.

A later deduplication check exposed a sharper version of the same problem. A classic riddle wrapped in instructions to act as a researcher was classified as Investigate because the model followed the roleplay vocabulary instead of the underlying task. A keyword scan bounded the issue to two distinct prompts and seven rows. I patched the rubric to judge answer-checkability instead of persona language and regression-tested the change on nine contrasting cases.

Figure 3 The revised taxonomy distinguishes answer structure rather than surface research language

Figure 4 Intersection of information-need structure and domain classification in the curated set

Independent audit changed the artifact

Regression cases were useful during development, but they were not an independently sampled ground truth. I therefore drew 151 non-overlapping cases across three audit rounds from both the 674-row accepted pool and the 1,470-row rejected pool. Small and previously weak strata were oversampled for diagnostic power, so the raw audit proportions were never presented as population proportions.

Each case was read fresh and judged on taxonomy correctness, answerability, whether it required synthesis of external evidence, and whether it could become a realistic deep-research evaluation task. Weighting each accepted stratum’s observed pass rate by its actual population share produced a 67.2 percent precision estimate for the unaudited accepted remainder. The rejected audit was small at 2.2 percent of that pool, but it revealed a 24.2 percent miss rate within the reviewed sample.

Human adjudication produced 26 concrete corrections: 18 accepted cases were overturned and 8 rejected cases were restored. Those corrections reduced the 598 candidates to 567. I did not treat 567 as the evaluation set, because most of those rows still depended on extrapolated quality. Instead, I retained only the records read in both the independent audit and my adjudication pass: 87 high-certainty cases.

Why certainty mattered more than nominal coverage

This choice traded breadth for defensibility. A larger candidate pool can support later coverage work, but an evaluation seed set silently inherits the errors of every unreviewed inclusion. For the first iteration, I preferred a smaller set in which every case could be traced to an actual human decision. The 67.2 percent estimate remains useful for understanding the wider candidate pool; it is not used to imply that unreviewed rows are verified.

Figure 5 Curation reduced nominal breadth in exchange for individually verified cases

Validation

Overlap with OpenResearcher

I compared the 87 cases with 6,102 unique OpenResearcher training questions and 11,639 texts from its benchmark history using exact matching and bge-large-en-v1.5 similarity. The first pass reported no match above 0.79. But the first implementation did not deserve confidence.

Two benchmark splits had encrypted answer fields that had been compared as ciphertext. The documented fixed-key recipe also failed to reproduce the benchmark’s own worked example because the real key depended on a per-row canary dropped by a redistributed copy. I traced the affected splits to their canonical sources, decrypted them with their actual canaries, and verified the procedure against the published example. The corrected result remained zero matches, zero near-duplicates, and a maximum similarity of 0.79.

Privacy

These were real conversations, so privacy was a hard gate. I manually read all 87 records and ran automated scans for emails, phone numbers, URLs, and social handles. The pattern scan found no hits, but manual review found three records with meaningful re-identification risk: a rare location combined with a distinctive venture, a highly specific company case, and a uniquely searchable thesis title. I removed all three, leaving 84 cases.

Get Sahil Chutani’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Internal diversity

I measured all pairwise similarities among the 84 retained prompts instead of assuming diversity from their labels. Mean cosine similarity was 0.516 and the maximum was 0.788, below the project’s approximate 0.85 near-duplicate anchor. HDBSCAN found two loose clusters covering 14 records; 70 were not dense enough to join a cluster. An independent topic classifier found no History, Politics, or Religion cases [5]. Because those topics existed in the broader candidate pool, I documented the gap.

Figure 6 Overlap, privacy, and internal-diversity checks answer different validation questions

Reference checklists for open ended answers

A single canonical answer is a poor gold standard for broad research questions. I therefore separated reference expectations into WIDTH and DEPTH. WIDTH captures requirement-level coverage: which issues, comparisons, evidence types, or decisions a strong response must address. DEPTH captures reasoning: how the response should connect evidence, qualify claims, reconcile tradeoffs, or adapt conclusions to the question.

Each WIDTH requirement contains a broad, source-grounded set of satisfying paths and the conditions under which each path is valid. An open-set rule allows an unlisted approach when the response provides equivalent substantive support. Judges can therefore credit substantive equivalence without requiring citation identity or treating my initial enumeration as exhaustive.

Before generating checklists, I separated 81 ANSWERABLE cases from 3 NEEDS CLARIFICATION cases. The distinction took three rubric iterations. Early versions confused breadth with ambiguity and flagged most cases for clarification. The final procedure forces the classifier to name the missing information, test it against explicit exclusions, and only then decide. Clarification cases will be evaluated on whether the agent recognizes the blocking ambiguity, not against a fixed comprehensive-answer checklist.

The completed reference set contains 336 WIDTH requirements and 167 DEPTH criteria grounded in 170 distinct verified sources, with no orphaned or unused references.

Figure 7 WIDTH and DEPTH provide distinct reference dimensions for answerable cases

Mass generation flattened the checklists

Field completeness initially hid a serious generation defect. In a mass-generated batch affecting 71 of 81 cases, all four WIDTH requirements within a case shared byte-identical satisfying paths. Across the affected batch, the conditions field collapsed to one or two generic templates. The topics sounded appropriate, but four supposedly independent checks had become one check wearing four labels.

I repaired the large batch and then found the same defect in ten earlier cases that had been assumed clean. After two passes, all 336 WIDTH items had unique satisfying-path text and unique conditions text. I also revised the weights to reflect substantive importance rather than item position. The defect affected scoring: without the repair, a response could appear to satisfy several dimensions by repeating one generic approach.

Figure 8 Content-level review exposed and repaired repeated checklist logic

What the work establishes

This project produced a reproducible cold-start Selection and Curation pipeline and an 84-case seed set in which I verified every case. The WIDTH and DEPTH protocol is now ready to score OpenResearcher outputs. The work also records what the privacy, overlap, near-duplication, diversity, audit, and checklist checks found.

These claims have clear limits. LMSYS is an external proxy. I also froze the criteria before running the agent, which means this stage cannot report model performance. Formal SME validation comes later. For now, I reduced that risk through source grounding, my review of every retained case, alternative satisfying paths, explicit conditions, open-set handling, and repeated content checks. The result is auditable, but my review is not a substitute for subject-matter expertise across every domain.

What I learned

1. Oversampling early is useful when later stages target identifiable false-positive classes.
2. Rejected cases need human sampling because model agreement can preserve systematic recall failures.
3. A taxonomy helps only when it classifies the underlying information need rather than surface vocabulary.
4. Privacy review benefits from combining pattern detection with human re-identification judgment.
5. Benchmark contamination checks are only as credible as their source and transformation verification.
6. Generated rubrics require semantic quality control; populated fields and valid schemas are not enough.

Dataset construction is evaluation work. Decisions about what enters the suite, what gets excluded, and what counts as a satisfying answer shape every score that follows.

What comes next

Next, I will run all 84 cases through OpenResearcher. Two independent judges will score the 81 answerable cases for WIDTH, DEPTH, and citation correctness; the other three will test whether the agent asks for needed clarification. Before targeted SME review, I will deduplicate disagreements, open-set matches, and representative samples of pass-pass and fail-fail decisions. Compression comes later, once the response panel is varied enough to show which cases are redundant.

Those results will show whether this usage-inspired cold-start set reveals behavior that OpenResearcher’s existing benchmarks miss.

Figure 9 Planned evaluation and review sequence after checklist freezing

References

[1] Li, Z. et al. (2026). OpenResearcher A Fully Open Pipeline for Long Horizon Deep Research Trajectory Synthesis. arXiv 2603.20278. https://arxiv.org/abs/2603.20278

[2] Zheng, L. et al. (2024). LMSYS Chat 1M A Large Scale Real World LLM Conversation Dataset. ICLR 2024. https://openreview.net/forum?id=BOfDKxfwt0

[3] Zheng, L. et al. (2023). Judging LLM as a Judge with MT Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. https://openreview.net/forum?id=uccHPGDlao

[4] Marchionini, G. (2006). Exploratory Search From Finding to Understanding. Communications of the ACM 49 4 41 to 46.

[5] Wettig, A. et al. (2025). Organize the Web Constructing Domains Enhances Pre Training Data Curation. https://huggingface.co/WebOrganizer/TopicClassifier-NoURL

[6] Kuhlthau, C. C. (1991). Inside the Search Process Information Seeking from the User Perspective. Journal of the American Society for Information Science 42 5 361 to 371. https://wp.comminfo.rutgers.edu/ckuhlthau/information-search-process/

[7] OpenResearcher public interactive demo on Hugging Face Spaces. Accessed August 2026. https://huggingface.co/spaces/OpenResearcher/OpenResearcher