When the Same Tool Produces Different Realities: A Lesson from the Field
We talk a lot about data quality in public health. We invest in cleaning protocols, dashboard refinements, and analytical rigour. What we talk about less is where some of the most consequential quality problems actually begin: not in the dataset, but in the tool that produced it, and in the assumptions we made when we designed it.
This is something I encountered directly during my master’s research.
The study
I led a comparative study on knowledge, perceptions, and use of herbal medicine during pregnancy across urban and rural communities in Nigeria. The design was straightforward: one questionnaire, two settings. The instrument had been carefully developed. It was well-structured, reviewed, and on paper, sound.
What happened in practice was more complicated.
Two settings, one tool
In the urban setting, respondents engaged with the questions largely as intended. There was familiarity with clinical language. When something was unclear, people asked for clarification. You could observe an active, if sometimes imperfect, attempt to interpret the questions the way they were designed to be interpreted.
In the rural setting, the dynamic was different. Not wrong, but different in ways that mattered enormously for what the data would eventually say.
Some questions did not carry the same meaning, even after translation. A “yes” could reflect awareness of a practice rather than actual engagement with it. In certain cases, responses appeared shaped less by lived experience and more by what seemed socially acceptable to say, particularly in the presence of an interviewer perceived as an outsider to the community.
Nothing about the tool had changed. Everything about the context had.
What this reveals
This is not simply a story about translation gaps or rural literacy, though both are real considerations. It points to something more structural: that a survey instrument is not a neutral object. It carries embedded assumptions about how respondents process questions, what language they are comfortable with, and how they relate to the person asking.
Get Iyobosa Edenamigho’s stories in your inbox
Join Medium for free to get updates from this writer.
Background is one factor. A respondent with clinical familiarity and a respondent without it will not experience the same question the same way, even if both can read it fluently. Context is another. Meaning is not fixed. It is negotiated within a setting, shaped by what is culturally legible, socially safe, and locally relevant. And the interaction between respondent and interviewer is a third. A professionally trained interviewer does not automatically have the cultural proximity required to read what sits beneath a response, to hear what is implied rather than what is said. That gap is not a training failure. It is, in many cases, a design failure.
So when we later label issues in our datasets as bias or inconsistency, the honest question is whether the data failed us, or whether we underestimated the setting it came from.
Where does this leave us?
It leaves us with a responsibility that sits earlier in the research process than most of us tend to look.
Questions should be framed with range in mind. Clinical language does not carry equal weight across educational and cultural backgrounds, and instruments that assume it does will produce systematically uneven data long before anyone opens a cleaning script. Design reviews should include voices that represent the populations the tool will actually encounter, not only methodological experts who share the same professional vocabulary as the researchers.
And perhaps most fundamentally, we need to stop designing tools that place the full burden of interpretation on the interviewer. When a respondent cannot safely or clearly express nuance through a direct question, the instrument itself should create that space. Indirect questioning, graduated response options, and vignette-based items are not methodological luxuries. In contexts where what is said and what is meant are routinely different, they are measurement necessities.
The distinction that stays with me
It is easy to focus on what happens after data is collected: the cleaning, the modelling, the visualization. Those things matter. But a significant portion of the work lives upstream, in decisions that can seem small at the design stage: how a question is framed, what we assume about comprehension, what context we fail to account for.
We do not only design surveys. We design how people experience answering questions. And that experience is shaped by who they are, where they are, and who is asking.
Same tool. Different realities. The gap between those two things is not noise. It is data we chose not to capture.
If you have worked across different community settings, you have likely encountered a version of this. When has a survey instrument behaved differently from what you expected? I would be glad to hear your experience in the comments.
Iyobosa Edenamigho is an M&E professional and PhD student with a focus on health data, health systems, and health policy in sub-Saharan Africa.
