<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Idan Felz on Medium]]></title>
        <description><![CDATA[Stories by Idan Felz on Medium]]></description>
        <link>https://medium.com/@idanfelz?source=rss-186158c435a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-ZJMdBtyRzCYXeiErNw9ZQ.jpeg</url>
            <title>Stories by Idan Felz on Medium</title>
            <link>https://medium.com/@idanfelz?source=rss-186158c435a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:15:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@idanfelz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Logging Isn’t Enough for LLM Systems (and How Observability Fixes It)]]></title>
            <link>https://medium.com/elementor-engineers/why-logging-isnt-enough-for-llm-systems-and-how-observability-fixes-it-018e528e9f89?source=rss-186158c435a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/018e528e9f89</guid>
            <category><![CDATA[langfuse]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Idan Felz]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 10:35:33 GMT</pubDate>
            <atom:updated>2025-12-11T10:56:49.280Z</atom:updated>
            <content:encoded><![CDATA[<p>As our LLM features grew, so did our blind spots.</p><p>We had multiple flows - some simple, some chained, some influencing each other - but we only had standard application logs to understand them. Every prompt tweak required a redeploy. Experiments were hard to compare. Debugging felt like guesswork.</p><p>Logs told us <em>what</em> happened, but they rarely told us <em>why</em>. Once our app scaled, that wasn’t enough.</p><h3>How We Started</h3><p>Our LLM stack started as a handful of small features inside our business suite for web creators. Some were user-facing, like the AI assistant in our proposals editor or smart suggestions. <br>Others were internal pipelines that summarized project data or classified meeting recordings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vl8e9bfnw_wyDS0jymVW0A.png" /><figcaption>our proposal assistant side-by-side with our proposal editor</figcaption></figure><p>Early on, our tooling was minimal:</p><ol><li><strong>Logs</strong> to inspect inputs and outputs</li><li><strong>Hardcoded Prompts</strong> stored in the codebase, requiring a full redeploy for even the smallest edit.</li></ol><p>This setup worked during the early POC phase. A simple single-prompt flow that generated a reply or produced a quick analysis didn’t need complex tooling. <br><strong>But</strong> as the product evolved, these features became deeper and more interconnected. What started as a basic assistant or report generator turned into multi-step pipelines and agent involving retrieval, reasoning, classification, and refinement.</p><p>That’s when the friction began. Logs told us <strong>what happened</strong>, but not <strong>why</strong>. We couldn’t easily reproduce user reports, see which prompt version was used, or understand how one step’s output shaped the next. And we had no reliable way to evaluate whether results were improving or if changes in prompts or models were silently degrading quality.<br>As our LLM features multiplied, those visibility gaps only became more painful.</p><h3>Why Traditional Logs Fail LLM Apps</h3><ul><li><strong>LLMs are nondeterministic:</strong> Identical inputs can produce different outputs. Standard logs capture the event, but not the subtle shifts in model behavior or sampling.</li><li><strong>Pipelines are multi-step, logs are flat:</strong> Prompt construction, embeddings, RAG retrieval, tool calls, and retries happen in a hierarchy. Logs show isolated events, making it hard to see how step A influenced step C.</li><li><strong>Lack of Context:</strong> Debugging an LLM requires knowing the prompt version, the specific chunks retrieved from the vector DB, intermediate outputs, and metadata that logs don’t naturally capture.</li><li><strong>No Structure or Quality Metrics:</strong> LLM systems need trace hierarchies, timelines, diffs, RAG insights, evaluations, and drift detection — things logs can’t provide no matter how much you log.</li></ul><h3>What We Tried (And Why It Didn’t Work)</h3><p>Realizing that our logs could only take us so far, we began exploring better ways to understand and debug the internals of our LLM workflows.</p><p><strong>First, we manually mapped our flows</strong>. We documented everything happening inside each pipeline-prompt construction, embeddings, RAG retrieval, chain logic, retries, and fallbacks. While this exercise was useful, it quickly highlighted just how many critical decisions were completely invisible in our standard logs.</p><p><strong>Next, we attempted to enhance our logging.</strong> We tried brute-forcing the problem by logging <em>more</em>: prompt versions, chunk metadata, similarity scores, tool call details, and retry logic. The result? Noisy and inconsistent logs that still didn’t provide a coherent story. We had more data, but not more insight.</p><p><strong>We also explored OpenTelemetry.</strong> Distributed tracing seemed promising at first, but we found that OTel was built for microservices, not AI pipelines. It doesn’t inherently understand concepts like prompts, RAG, or model reasoning. We found ourselves trying to build missing pieces manually, which felt like reinventing the wheel.</p><p><strong>Finally, we evaluated LLM-focused observability tools.</strong> We realized we needed something purpose-built for AI workflows-a tool that could give us structure, context, and visibility without bloating our codebase. That’s when we found <strong>Langfuse</strong>.</p><h3>What Langfuse Brings to the Table</h3><p><a href="https://langfuse.com/">Langfuse</a> isn’t just a logging tool — it’s an LLM-first observability layer built for the unique challenges of AI systems. Instead of relying on scattered logs and manual instrumentation, it provides a structured, visual understanding of how LLM-powered workflows actually behave under the hood.</p><p>Below are some of its core<strong> capabilities:</strong></p><h4>1. End-to-End Tracing for LLM Workflows</h4><p>Langfuse converts each LLM request into a visual trace composed of spans, steps, and contextual metadata. This makes it possible to see:</p><ul><li>a hierarchical flow of embeddings → RAG retrieval → LLM calls → tool calls → post-processing</li><li>inputs, outputs, latency, and token usage for each step</li><li>retries, fallbacks, and intermediate states</li><li>the full execution path needed to reproduce behavior</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2aCOYF_T6ugZkTHGqnMcqg.png" /></figure><p>This level of tracing helps turn debugging into a structured, efficient process rather than guesswork.</p><h4>2. Prompt Management &amp; Versioning</h4><p>Prompts tend to evolve rapidly and get scattered across files, configs, or services. Langfuse offers:</p><ul><li>built-in prompt versioning</li><li>diff views to compare changes between prompt iterations</li><li>linking prompt versions to real traces</li><li>safer, more confident prompt experimentation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1xbtQBZM2sKwXxkJTqEDQ.png" /></figure><p>ֿWith proper versioning, subtle regressions become visible instead of slipping into production unnoticed.</p><h4>3. Evaluation Tools for Quality Monitoring</h4><p>LLM quality often degrades gradually, not through obvious failures. Langfuse includes evaluation features designed to monitor quality over time, such as:</p><ul><li>datasets for storing and replaying real interactions</li><li>model or prompt benchmarking</li><li>automated or human-in-the-loop scoring</li><li>drift detection for early warning signals</li><li>optional integration into CI/CD workflows</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFJb2DonN43Qty_owi3i0Q.png" /></figure><p>These evaluations help turn subjective model behavior into measurable trends.</p><h4>4. Rich Metadata, Cost, and Token Insights</h4><p>LLM pipelines produce large amounts of metadata that are valuable for optimization but tedious to maintain manually. Langfuse automatically captures token usage, cost per step, user/session identifiers, and latency breakdowns. Surfacing this information visually makes performance bottlenecks and cost anomalies far easier to identify.</p><h3>How We Work With Langfuse</h3><p>Integrating Langfuse was straightforward. We started by instrumenting our core flows — the ones where debugging was most painful, and expanded from there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIInR6iDtECJBezcVIJQtw.png" /><figcaption>generation — a special span unique to LLM calls that adds essential metadata</figcaption></figure><p><strong>1. Tracing Key Flows</strong> Each critical step (RAG retrieval, LLM call, post-processing) is wrapped in a span. When a weird bug happens, we don’t grep logs; we open a trace. We can immediately answer: <em>What data did the model see? Did RAG return irrelevant chunks? Did a fallback trigger?</em></p><p><strong>2. Iterating on Prompts</strong> Prompt updates no longer require code deployments. We manage prompts in the UI, compare versions, and roll back if needed. This has democratized prompt engineering within the team.</p><p><strong>3. Evaluations (Work in Progress)</strong> We are currently experimenting with Langfuse’s evaluation tools to monitor quality. While our workflow here is still evolving, early results are promising for spotting drift and validating whether a prompt change actually improved the output.</p><p>Since introducing Langfuse into our workflows, the impact has been clear. It has helped us:</p><ul><li>trace user reports more reliably and understand what actually happened inside a flow</li><li>catch unexpected model or agent behaviors that were previously invisible</li><li>monitor performance and cost across our LLM pipelines</li><li>iterate on prompts faster and with more confidence</li><li>spot regressions or quality drops before they became real issues</li></ul><p>As our LLM usage continued to grow, the limits of traditional logging became impossible to ignore. We needed visibility into how our flows behaved, not just what they returned.</p><p>Introducing Langfuse gave us that missing layer — turning complex, opaque pipelines into transparent, traceable processes we can actually debug, evaluate, and improve. It hasn’t solved everything yet, but it’s already reshaped how we understand and build our LLM features, and it’s quickly becoming an essential part of how we work with AI at scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=018e528e9f89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/elementor-engineers/why-logging-isnt-enough-for-llm-systems-and-how-observability-fixes-it-018e528e9f89">Why Logging Isn’t Enough for LLM Systems (and How Observability Fixes It)</a> was originally published in <a href="https://medium.com/elementor-engineers">Elementor Engineers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>