DEV Community

Cover image for RAG observability: the five things worth logging (and the ten that aren't)
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

RAG observability: the five things worth logging (and the ten that aren't)

#rag #observability #production

Most RAG logging setups capture the full prompt and full response and nothing else, which makes debugging a bad answer nearly impossible six weeks later.

Log this

  • The retrieved chunk IDs and their similarity scores, not just the final context text.
  • Whether a confidence gate or relevance check fired, and its verdict.
  • Latency per stage (embed, search, rerank, generate), not just total time.
  • User feedback (thumbs up/down) linked to the exact chunk IDs used.
  • Query rewrites, if you decompose or rewrite the original question.

Skip logging every raw embedding vector

Vectors are expensive to store and rarely useful after the fact — you can always re-embed the original chunk text if you need to inspect it. Log the text and the score, not the numbers.

See testing LLM applications, a practical guide for how this logging feeds an evaluation loop.

About Pranjul Rathour

Pranjul Rathour pitching with a microphone in front of a projector slide at CSJMU, Kanpur
Pitching at CSJMU, Kanpur

Pranjul Rathour speaking from the podium at VSICS, Kanpur
At the VSICS podium, Kanpur

Pranjul Rathour holding a trophy and a certificate of merit after a win
Trophy and certificate after a win

Pranjul Rathour presenting evaluation criteria — feasibility, innovation, practicality, problem solving — on a projector screen
Walking a room through evaluation criteria

Pranjul Rathour presenting BrandHive on a projector screen
Presenting BrandHive

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)