<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stanley Dakwoji on Medium]]></title>
        <description><![CDATA[Stories by Stanley Dakwoji on Medium]]></description>
        <link>https://medium.com/@sdakwoji?source=rss-7a8293a29b16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*eTYUJSG_7qRvvy2b</url>
            <title>Stories by Stanley Dakwoji on Medium</title>
            <link>https://medium.com/@sdakwoji?source=rss-7a8293a29b16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:23:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sdakwoji/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Log Chaos to Real-Time Anomaly Detection with Elastic Streams]]></title>
            <link>https://medium.com/@sdakwoji/from-log-chaos-to-real-time-anomaly-detection-with-elastic-streams-068ac23380cf?source=rss-7a8293a29b16------2</link>
            <guid isPermaLink="false">https://medium.com/p/068ac23380cf</guid>
            <dc:creator><![CDATA[Stanley Dakwoji]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 21:01:08 GMT</pubDate>
            <atom:updated>2026-09-19T22:17:41.151Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*athOZz2S6ANA0vr10zN8wg.png" /></figure><h3>How to use Elasticsearch Streams to ingest, partition, and alert on a noisy service — and the production lessons that don’t make it into the docs</h3><p>Picture a busy backend tier — I’ll use a database (DB) access layer as the running example — that emits a firehose of logs: tens of millions of lines a day, of which a small but important slice are <em>errors</em>: query timeouts, connection failures, pool exhaustion, and the occasional brand-new failure nobody has seen before.</p><p>The goal sounds simple: <strong>know, within minutes, when errors spike — and know which host and which kind of error.</strong> Doing that well at scale is where <strong>Elastic Streams</strong> earns its place.</p><blockquote><strong>This is not a database thing.</strong> The DB tier is just my example. The exact same pattern fits anything that emits lots of events you need to triage by type — <strong>API gateways, microservices, payment or auth services, message brokers, IoT device fleets, CDN/edge logs.</strong> Wherever you have “high volume, several categories, and a need to spot spikes per category,” Streams fits. Read <em>db</em> below as &quot;your service.&quot;</blockquote><p>This is a practical walk-through — <strong>ingesting incoming data, organizing data at rest, partitioning into child streams, wiring up ML datafeeds, and creating alert rules</strong> — plus the hard-won lessons that only show up in production.</p><blockquote>A note on the code: the <em>//</em> lines in the snippets are <strong>explanations for you, the reader</strong> — JSON doesn&#39;t actually allow comments, so strip them before running.</blockquote><h3>Metrics tell you that. Logs tell you what.</h3><p>Most teams start with metrics dashboards — error rate, latency, connection counts, CPU. Those are essential, but they have a ceiling: <strong>a metrics dashboard tells you <em>that</em> something is wrong, never <em>what</em>.</strong></p><p>You watch the error-rate line spike at 2 a.m. and connections climb. Great — now what? “Errors are up 400%” doesn’t tell you <em>which</em> error. Was it a query timeout? A connection pool exhausted? A single unreachable node? A brand-new failure nobody has seen? The dashboard can’t say, because that detail was never captured as a metric — <strong>it lives in the log files.</strong></p><p>The exact error — its code, its category, the host it hit, the message — is written in the log lines the service produces. So the real job isn’t just “chart the error rate.” It’s <strong>get the actual errors out of the log files, structured and searchable, per host and per type, fast enough to act on.</strong> That is the entire point of the pipeline below.</p><p>Two layers, and you need both:</p><ul><li><strong>Metrics</strong> = the smoke alarm. It tells you there’s smoke.</li><li><strong>Logs</strong> = walking into the room and seeing <em>what is actually on fire.</em></li></ul><p>Elastic Streams is how I turn a raw, noisy log firehose into that second layer — without it collapsing under its own volume.</p><h3>Why Elastic Streams?</h3><p>Elastic Streams gives you a <strong>hierarchy of data streams</strong> with a parent/child structure, backed by logsdb index mode. In one sentence: it lets you land everything in one logical place, then automatically fan the data out into well-typed sub-streams — without a pile of brittle, hand-maintained indices.</p><p>What it buys you:</p><ul><li><strong>One entry point, many destinations.</strong> Everything enters at the top of the hierarchy; routing rules sort it into child streams. No shipper needs to know the final index name.</li><li>logsdb<strong> efficiency at rest.</strong> Columnar storage + best-compression means logs cost far less to keep, so you retain more history for investigations.</li><li><strong>Clean separation by event type.</strong> Each category gets its own child stream, so dashboards, queries, and ML jobs can target exactly what they care about.</li><li><strong>A built-in catch-all.</strong> New, unrecognized events don’t vanish — they stay in the parent stream, still searchable and alertable.</li></ul><p>The mental model (swap the names for your domain — payments, api-gateway, iot, whatever):</p><pre>logs.ecs                        (top of the hierarchy)<br>└── logs.ecs.db                 (PARENT — the catch-all for this service)<br>    ├── logs.ecs.db.query-timeout        (child — one error category)<br>    ├── logs.ecs.db.connection-failed    (child)<br>    ├── logs.ecs.db.pool-exhausted       (child)<br>    └── logs.ecs.db.unavailable          (child)</pre><h3>1. Ingesting incoming data</h3><p>Raw logs land first in a normal, high-volume daily index (think db-errors-YYYY.MM.DD). You <em>don&#39;t</em> want every log line in the Streams hierarchy — only the errors. So the raw index carries a default_pipeline: a rule that runs on every document as it&#39;s indexed.</p><p>That pipeline does one job — <strong>reroute error documents into the Streams hierarchy</strong>:</p><pre>PUT _ingest/pipeline/db-errors-to-logs-ecs<br>{<br>  &quot;processors&quot;: [<br>    {<br>      &quot;reroute&quot;: {<br>        // Send this document into the Streams hierarchy instead of the raw index...<br>        &quot;destination&quot;: &quot;logs.ecs&quot;,<br>        // ...but ONLY if it&#39;s an error. Normal logs are untouched and stay put.<br>        // (doc_type is just a field your shipper/pipeline sets; use whatever marks &quot;this is an error&quot;.)<br>        &quot;if&quot;: &quot;ctx.doc_type == \&quot;error\&quot;&quot;<br>      }<br>    }<br>  ]<br>}</pre><p>Attach it to the index (or, better, the index template) as the default_pipeline:</p><pre>&quot;settings&quot;: {<br>  &quot;index&quot;: {<br>    // Run the rule above on EVERY doc that lands in this index, automatically.<br>    &quot;default_pipeline&quot;: &quot;db-errors-to-logs-ecs&quot;<br>  }<br>}</pre><p>Now every error document is automatically lifted out of the raw firehose and dropped into the top of the Streams hierarchy, while normal logs stay put. <strong>The shipper never needs to know about </strong>logs.ecs<strong> at all</strong> — the cluster handles routing.</p><blockquote><strong>Key idea:</strong> <em>default_pipeline</em> is a switch that says &quot;run this rule on everything coming in.&quot; The rule it points to happens to be a forwarding rule. That indirection is what makes the whole thing resilient — the shipper just writes to one place.</blockquote><h3>2. Data at rest</h3><p>Inside the Streams hierarchy, each stream is a <strong>data stream</strong> in logsdb mode:</p><ul><li>logsdb<strong> index mode</strong> — columnar, best_compression, index-sorted by host.name + @timestamp. Events compress beautifully and queries stay fast.</li><li><strong>Data Stream Lifecycle (DSL)</strong> handles rollover and retention automatically — no manual ILM babysitting per child stream.</li><li><strong>One append-only surface per category</strong>, so a query like “timeout errors in the last 24h” hits exactly the right backing indices.</li></ul><p>An important property to internalize: <strong>writes go in at the top of the hierarchy, not into a child stream directly</strong> — data flows in via the parent and cascades down. Great for consistency, but it has real implications for backfills (more in the lessons).</p><h3>3. Partitioning into child streams</h3><p>This is where Streams shines. A <strong>reroute pipeline</strong> on the parent inspects each document and sends it to the right child based on a classification field (I use an error.category keyword set upstream):</p><pre>PUT _ingest/pipeline/logs.ecs.db@stream.reroutes<br>{<br>  &quot;processors&quot;: [<br>    // Each rule: &quot;if the category is X, send it to child stream X.&quot;<br>    { &quot;reroute&quot;: { &quot;destination&quot;: &quot;logs.ecs.db.query-timeout&quot;,<br>                   &quot;if&quot;: &quot;ctx.error.category == &#39;query_timeout&#39;&quot; } },<br>    { &quot;reroute&quot;: { &quot;destination&quot;: &quot;logs.ecs.db.connection-failed&quot;,<br>                   &quot;if&quot;: &quot;ctx.error.category == &#39;connection_failed&#39;&quot; } },<br>    { &quot;reroute&quot;: { &quot;destination&quot;: &quot;logs.ecs.db.pool-exhausted&quot;,<br>                   &quot;if&quot;: &quot;ctx.error.category == &#39;pool_exhausted&#39;&quot; } },<br>    { &quot;reroute&quot;: { &quot;destination&quot;: &quot;logs.ecs.db.unavailable&quot;,<br>                   &quot;if&quot;: &quot;ctx.error.category == &#39;unavailable&#39;&quot; } }<br>    // NOTE: there is deliberately no &quot;else&quot;. Anything that matches none of these<br>    // stays in the parent (logs.ecs.db) = the catch-all, so a new error type isn&#39;t<br>    // silently dropped (as long as it still reaches ingestion).<br>  ]<br>}</pre><p>Two design choices worth calling out:</p><p><strong>The catch-all is free.</strong> Notice there’s no else. If a document&#39;s error.category matches none of the rules, <strong>it simply stays in the parent stream.</strong> That&#39;s the safety net: a brand-new error type nobody wrote a rule for still lands somewhere searchable and alertable, instead of being dropped or misfiled. (I verified this with a pipeline _simulate — an unknown category stays in the parent, exactly as intended.)</p><p><strong>Classify by a stable field, not free text.</strong> Route on a low-cardinality error.category keyword, and separately keep an error.code keyword for finer analysis. Free-text messages are terrible for routing and aggregation — a mapped keyword is the right tool.</p><blockquote><strong>Mapping gotcha:</strong> if a field is stored in <em>_source</em> but not mapped (or mapped as <em>text</em> instead of <em>keyword</em>), it looks present in Discover but you can&#39;t aggregate or filter on it. Always confirm your classifier fields are <em>keyword</em> via <em>_field_caps</em>. I once had a status field come out as <em>text</em> on one day&#39;s index and <em>keyword</em> on others — a cluster-wide query failed with a type conflict until the mapping was made consistent.</blockquote><h3>4. Setting up ML datafeeds (anomaly detection)</h3><p>With clean, partitioned data, an anomaly-detection job is straightforward. Mine watches for <strong>unusually high event counts, per host and per type</strong>, using two detectors:</p><pre>PUT _ml/anomaly_detectors/db_error_spikes<br>{<br>  &quot;analysis_config&quot;: {<br>    &quot;bucket_span&quot;: &quot;15m&quot;,                       // analyze counts in 15-minute windows<br>    &quot;detectors&quot;: [<br>      // Detector 0: spike in a category, per host (100% coverage — includes the parent catch-all)<br>      { &quot;function&quot;: &quot;high_count&quot;, &quot;by_field_name&quot;: &quot;stream.name&quot;,<br>        &quot;partition_field_name&quot;: &quot;host.name&quot; },<br>      // Detector 1: spike in a specific error code, per host (finer-grained)<br>      { &quot;function&quot;: &quot;high_count&quot;, &quot;by_field_name&quot;: &quot;error.code&quot;,<br>        &quot;partition_field_name&quot;: &quot;host.name&quot; }<br>    ],<br>    // Influencers = &quot;which entities to blame&quot; when something is anomalous<br>    &quot;influencers&quot;: [&quot;host.name&quot;, &quot;stream.name&quot;, &quot;error.code&quot;]<br>  },<br>  &quot;data_description&quot;: { &quot;time_field&quot;: &quot;@timestamp&quot; }<br>}</pre><p>Then a <strong>datafeed</strong> pulls from the whole hierarchy — parent <em>and</em> children — so the catch-all is monitored too:</p><pre>PUT _ml/datafeeds/datafeed-db_error_spikes<br>{<br>  &quot;job_id&quot;: &quot;db_error_spikes&quot;,<br>  // The trailing * covers the parent AND every child stream in one shot.<br>  &quot;indices&quot;: [&quot;logs.ecs.db*&quot;],<br>  &quot;query&quot;: { &quot;match_all&quot;: {} },<br>  // query_delay = how long to WAIT after a bucket closes before analyzing it,<br>  // so late-arriving data has time to land. MUST be longer than your shipping lag. (See Lesson 1.)<br>  &quot;query_delay&quot;: &quot;120s&quot;<br>}</pre><p>Detector 0 (by stream.name) covers every category with 100% coverage — including the parent catch-all, which shows up as its own stream.name bucket. Detector 1 (by error.code) adds finer, type-level detection.</p><p><strong>The knob that matters most here is </strong>query_delay — and it&#39;s the one people misread. Read on.</p><h3>5. Visualizing the results: anomaly score, actual vs. typical</h3><p>The ML job doesn’t just flag anomalies — it writes them to a <strong>results index</strong> you can query and chart like any other data. Every detected anomaly is a document (result_type: record) carrying the fields that make a dashboard genuinely useful:</p><ul><li>record_score — 0–100, how unusual this was (this is what your alerts threshold on).</li><li>actual — what actually happened in that bucket (e.g. 420 errors).</li><li>typical — what the model <em>expected</em> (e.g. ~30). The gap between actual and typical is the whole story.</li><li>partition_field_value<strong> / </strong>by_field_value — which host, and which stream / error type.</li><li>timestamp — when it happened.</li></ul><p>Two ways to see it:</p><p><strong>a) The built-in viewers.</strong> Kibana’s <strong>Anomaly Explorer</strong> (swim lanes per host and stream) and <strong>Single Metric Viewer</strong> (which plots actual against the shaded typical band) come free with the job — great for eyeballing an incident.</p><p><strong>b) A custom dashboard.</strong> Point a data view at the results index and build Lens panels: a table of <em>top anomalies (score, actual, typical, host)</em>, a max-score-over-time line, a per-host heatmap.</p><blockquote><strong>Point the data view at the read alias, not the raw index.</strong> Every ML job exposes a filtered alias <em>.ml-anomalies-&lt;job_id&gt;</em> that always tracks the job&#39;s current results index and filters to just that job. Use it — a data view pinned to the physical <em>…-000001</em> index goes stale the moment the results index rolls over. (Learned that one the hard way.)</blockquote><p>Here’s an ES|QL panel that surfaces the worst offenders with actual vs. typical side by side:</p><pre>FROM .ml-anomalies-db_error_spikes          -- the job&#39;s read alias<br>| WHERE result_type == &quot;record&quot;             -- one row per detected anomaly<br>| KEEP timestamp, record_score, partition_field_value, by_field_value, actual, typical<br>| RENAME partition_field_value AS host, by_field_value AS stream<br>| SORT record_score DESC                     -- most anomalous first<br>| LIMIT 20</pre><p>That single query answers the on-call question at a glance: <em>how bad (score), on which host, in which stream — and how far </em>actual<em> ran above what the model expected (</em>typical<em>).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*WpTXCtJEajG6-KFPCUzb5Q.png" /></figure><p><em>A “top anomalies” table on the results index: the score is color-coded, and </em>actual<em> sits right next to the model&#39;s </em>typical<em> — so how far things ran off is obvious at a glance (420 errors vs an expected ~31).</em></p><h3>6. Creating alert rules</h3><p>I run three complementary rules, all emailing the same on-call list:</p><ol><li><strong>Volume / outage alert</strong> (an Elasticsearch-query rule): <em>fire if event count &gt; N in a rolling window.</em> The dependable outage detector — a hard threshold that never drifts.</li><li><strong>Forwarding-broken alert</strong>: <em>fire if error documents pile up in the raw index unforwarded.</em> When routing is healthy, errors reroute <em>out</em> of the raw index within moments; if they start accumulating there, the pipeline has broken. This catches the failure mode below in minutes instead of hours.</li><li><strong>ML anomaly alert</strong>: fires on the anomaly-detection job for subtler pattern shifts.</li></ol><p>And for ad-hoc investigations, ES|QL over the partitioned data is a joy:</p><pre>FROM logs.ecs.db*                                -- query across parent + all children<br>| WHERE event.type == &quot;health_check&quot;             -- narrow to the events you care about<br>| STATS up   = COUNT(*) WHERE health.status == &quot;UP&quot;,     -- conditional counts in one pass<br>        down = COUNT(*) WHERE health.status == &quot;DOWN&quot;,<br>        latency_avg = ROUND(AVG(health.latency_ms), 1)<br>    BY host.name                                  -- one row per host<br>| SORT down DESC                                  -- worst hosts first</pre><h3>The lessons the docs don’t tell you</h3><p>This is the part I most wanted to write down, because every one of these cost real time.</p><h3>Lesson 1 — query_delay must be longer than your shipping interval</h3><p>query_delay is <strong>not</strong> how often the job runs. It&#39;s <strong>how long the job waits after a time bucket closes before analyzing it</strong> — a grace period for late-arriving data.</p><p>Here’s the trap. If your agent flushes logs every <strong>15 minutes</strong>, but the ML job waits only <strong>1 minute</strong> before analyzing a bucket, it looks at an almost-empty index, counts ~zero, scores it “normal,” and moves on. The real data lands 14 minutes later — but the job has already finished with that bucket and never looks back.</p><p>That’s exactly how a real outage got <strong>missed</strong>: the error-count dashboard showed a clear 200+/15-min spike, but the ML job had scored those buckets at ~0 because the data hadn’t arrived yet when it looked.</p><p><strong>The rule:</strong> query_delay must be <em>longer than</em> the time it takes data to land. Shipping every 15 min → wait ~20 min. Once the flush was reduced to 60 seconds, I dropped query_delay back to ~2 minutes for near-real-time detection.</p><blockquote>Same math applies to alert windows: a rolling 30-minute count query will <strong>miss a spike that lands 40 minutes late.</strong> Late data will defeat a time-window detector whenever the window is shorter than the delay — so the durable fix is at the shipper (flush more often), not downstream.</blockquote><h3>Lesson 2 — Don’t rely on ML alone for spikes on a noisy platform</h3><p>Anomaly detection learns what “normal” looks like. On a platform that spikes <em>often</em>, the model learns to treat those spikes <strong>as</strong> normal — after training over a stretch full of them, a big burst stops looking unusual and the job scores it low. I saw exactly this: a clear outage scored close to 0.</p><p>That’s why a <strong>hard-threshold volume alert</strong> is the real outage catcher, and the ML job is a supplement for subtle shifts. Use both; don’t expect ML to be your smoke detector.</p><h3>Lesson 3 — Your index template can be silently overwritten</h3><p>The forwarding default_pipeline kept <strong>disappearing at midnight.</strong> Cause: an external provisioning job re-applied the index template every night <em>without</em> the pipeline line, so each new daily index came up with routing disabled — and monitoring went blind until someone noticed.</p><p>Two-part fix: (1) the <em>real</em> cure is adding the pipeline line to whatever config manages the template at its source; (2) as a resilient workaround, create a <strong>second index template with higher priority</strong> carrying the pipeline. Elasticsearch always applies the highest-priority matching template, so yours wins even when the nightly job rewrites the lower-priority one. It survived subsequent rollovers cleanly.</p><h3>Lesson 4 — You can’t backfill into child streams with _reindex</h3><p>When I tried to backfill a gap, direct writes to the child streams were <strong>rejected</strong> — the docs landed in the <em>failure store</em>, and _reindex happily reported &quot;created&quot; while nothing reached the target stream. What actually worked was re-feeding the data through live ingestion at the top of the hierarchy. <strong>Test any backfill on a few docs first</strong> — don&#39;t assume you can reindex your way out of a gap.</p><h3>Lesson 5 — “No data” ≠ “broken”</h3><p>On an <em>error</em> stream, an empty chart usually just means <strong>no errors happened</strong> — which is good. The trick is distinguishing “quiet” from “the pipeline died.” I did it with the <em>forwarding-broken</em> alert: it watches the raw index for errors <strong>piling up unforwarded</strong>. No errors → nothing piles up → no false alarm. Broken routing → pileup → alert. That signal doesn’t false-alarm on quiet periods the way “the stream went empty” does — a genuinely error-free hour won’t trip it.</p><h3>Build it yourself (reproduce the whole thing)</h3><p>You can get this behavior two ways: with Elastic’s managed <strong>Streams</strong> feature (Kibana-driven), or with <strong>plain data streams + ingest pipelines</strong>, which is fully reproducible in Dev Tools and is what I’ll show here. Adapt the names to taste — db is just a placeholder for your service.</p><p><strong>Step 1 — Classify at ingest.</strong> Your events need a low-cardinality error.category keyword to route on. If it isn&#39;t already set, derive it from the message. Crucially, leave <em>unmatched</em> docs without a category so they fall through to the catch-all:</p><pre>PUT _ingest/pipeline/db-classify<br>{<br>  &quot;processors&quot;: [<br>    // Tag the doc with a category based on what its message contains.<br>    { &quot;set&quot;: { &quot;field&quot;: &quot;error.category&quot;, &quot;value&quot;: &quot;query_timeout&quot;,<br>               &quot;if&quot;: &quot;ctx.message != null &amp;&amp; ctx.message.contains(&#39;timeout&#39;)&quot; } },<br>    { &quot;set&quot;: { &quot;field&quot;: &quot;error.category&quot;, &quot;value&quot;: &quot;connection_failed&quot;,<br>               &quot;if&quot;: &quot;ctx.message != null &amp;&amp; ctx.message.contains(&#39;connection refused&#39;)&quot; } }<br>    // ...one per known category. Anything unmatched has NO error.category → parent catch-all.<br>  ]<br>}</pre><p><strong>Step 2 — Create the parent + child data streams</strong> via index templates. Parent:</p><pre>PUT _index_template/logs.ecs.db<br>{<br>  &quot;index_patterns&quot;: [&quot;logs.ecs.db&quot;],   // this template owns the parent stream<br>  &quot;data_stream&quot;: {},                    // make it a data stream, not a plain index<br>  &quot;priority&quot;: 500,<br>  &quot;template&quot;: {<br>    &quot;settings&quot;: { &quot;index&quot;: {<br>      &quot;mode&quot;: &quot;logsdb&quot;,                                  // efficient columnar storage<br>      &quot;default_pipeline&quot;: &quot;logs.ecs.db@reroutes&quot;         // run the fan-out rules on ingest<br>    } },<br>    &quot;mappings&quot;: { &quot;properties&quot;: {<br>      &quot;@timestamp&quot;:     { &quot;type&quot;: &quot;date&quot; },<br>      &quot;host.name&quot;:      { &quot;type&quot;: &quot;keyword&quot; },<br>      &quot;error.category&quot;: { &quot;type&quot;: &quot;keyword&quot; },   // routing key — MUST be keyword<br>      &quot;error.code&quot;:     { &quot;type&quot;: &quot;keyword&quot; },   // fine-grained key — MUST be keyword<br>      &quot;message&quot;:        { &quot;type&quot;: &quot;match_only_text&quot; }<br>    } }<br>  }<br>}</pre><p>Create one template per child (logs.ecs.db.query-timeout, …connection-failed, …) with the same mappings. (Tip: put the shared mappings in a <em>component template</em> and compose them, so they never drift apart.)</p><p><strong>Step 3 — The reroute pipeline</strong> (parent → children) is the one from Section 3, named logs.ecs.db@reroutes.</p><p><strong>Step 4 — Send a few sample events</strong> to the parent and watch them sort themselves:</p><pre>POST logs.ecs.db/_bulk<br>{&quot;create&quot;:{}}<br>{&quot;@timestamp&quot;:&quot;2026-01-01T00:00:00Z&quot;,&quot;host.name&quot;:&quot;db-1&quot;,&quot;error.category&quot;:&quot;query_timeout&quot;,&quot;error.code&quot;:&quot;QueryTimeout&quot;,&quot;message&quot;:&quot;query timeout after 30s&quot;}<br>{&quot;create&quot;:{}}<br>{&quot;@timestamp&quot;:&quot;2026-01-01T00:00:05Z&quot;,&quot;host.name&quot;:&quot;db-2&quot;,&quot;error.category&quot;:&quot;connection_failed&quot;,&quot;error.code&quot;:&quot;ConnRefused&quot;,&quot;message&quot;:&quot;connection refused&quot;}<br>{&quot;create&quot;:{}}<br>{&quot;@timestamp&quot;:&quot;2026-01-01T00:00:10Z&quot;,&quot;host.name&quot;:&quot;db-1&quot;,&quot;message&quot;:&quot;something nobody has classified yet&quot;}  // no category → catch-all</pre><p><strong>Step 5 — Verify routing:</strong></p><pre>GET logs.ecs.db.query-timeout/_count       # → 1<br>GET logs.ecs.db.connection-failed/_count   # → 1<br>GET logs.ecs.db/_count                      # → 1  (the unclassified one stayed in the parent — catch-all works)</pre><p><strong>Step 6 — Add the ML job and datafeed</strong> from Section 4, point a results dashboard at .ml-anomalies-db_error_spikes (Section 5), then wire the alert. Here&#39;s the full <strong>volume-alert</strong> rule (Kibana alerting API) so you don&#39;t have to reverse-engineer it:</p><pre>POST kbn:/api/alerting/rule<br>{<br>  &quot;name&quot;: &quot;DB error volume spike&quot;,<br>  &quot;rule_type_id&quot;: &quot;.es-query&quot;,          // &quot;fire when a query&#39;s count crosses a threshold&quot;<br>  &quot;consumer&quot;: &quot;stackAlerts&quot;,<br>  &quot;schedule&quot;: { &quot;interval&quot;: &quot;5m&quot; },     // re-check every 5 minutes<br>  &quot;notify_when&quot;: &quot;onActionGroupChange&quot;, // email once when it trips (not every cycle)<br>  &quot;params&quot;: {<br>    &quot;searchType&quot;: &quot;esQuery&quot;,<br>    &quot;index&quot;: [&quot;logs.ecs.db*&quot;],          // watch parent + all children<br>    &quot;timeField&quot;: &quot;@timestamp&quot;,<br>    &quot;esQuery&quot;: &quot;{\&quot;query\&quot;:{\&quot;match_all\&quot;:{}}}&quot;,<br>    &quot;size&quot;: 100,<br>    &quot;timeWindowSize&quot;: 30,               // count events over a rolling 30-minute window<br>    &quot;timeWindowUnit&quot;: &quot;m&quot;,<br>    &quot;thresholdComparator&quot;: &quot;&gt;&quot;,         // fire if...<br>    &quot;threshold&quot;: [200]                  // ...more than 200 events in that window<br>  },<br>  &quot;actions&quot;: [<br>    { &quot;group&quot;: &quot;query matched&quot;, &quot;id&quot;: &quot;&lt;your-email-connector-id&gt;&quot;,<br>      &quot;params&quot;: { &quot;to&quot;: [&quot;oncall@example.com&quot;],<br>                  &quot;subject&quot;: &quot;DB error spike: {{context.value}} in 30m&quot;,   // {{context.value}} = the count<br>                  &quot;message&quot;: &quot;Error volume crossed threshold — check the DB error dashboard.&quot; } }<br>  ]<br>}</pre><p>Swap in your email connector ID and you’ve got a working end-to-end skeleton — from raw log line to on-call email — to build on.</p><h3>Takeaways</h3><p>Elastic Streams turned an unmanageable log firehose into a <strong>clean, typed, self-organizing</strong> pipeline — and remember, this works for <strong>any</strong> high-volume event source, not just databases:</p><ul><li><strong>Ingest once</strong> at the top; let reroute rules do the sorting.</li><li><strong>Partition by a stable keyword</strong>, and lean on the <strong>parent as a catch-all</strong> so a new error type isn’t silently dropped.</li><li><strong>Store efficiently</strong> with logsdb + DSL.</li><li><strong>Detect</strong> with ML datafeeds — but tune query_delay to your shipping latency.</li><li><strong>Alert</strong> with hard thresholds for outages, a pileup check for pipeline health, and ML for the subtle stuff.</li></ul><p>And the meta-lesson: most of the pain wasn’t Elasticsearch — it was <strong>latency and lifecycle</strong>. Get your data landing promptly and your templates managed at the source, and Streams does the rest.</p><p><em>If you’re building something similar and hit these same walls, I’m happy to compare notes.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=068ac23380cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability Became Part of the Deployment Plan Instead of an Afterthought]]></title>
            <link>https://medium.com/@sdakwoji/observability-became-part-of-the-deployment-plan-instead-of-an-afterthought-1128aad6c12e?source=rss-7a8293a29b16------2</link>
            <guid isPermaLink="false">https://medium.com/p/1128aad6c12e</guid>
            <dc:creator><![CDATA[Stanley Dakwoji]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 22:36:28 GMT</pubDate>
            <atom:updated>2026-07-08T22:40:54.847Z</atom:updated>
            <content:encoded><![CDATA[<p>How one production incident changed the way I think about instrumentation, distributed tracing, and software deployments.</p><p>⸻</p><p>There was a time when I viewed observability as something you added after an application was built.</p><p>The application team would finish development, QA would validate the business logic, the service would be deployed, and then the SRE or platform team would come in to enable metrics, logs, traces, dashboards, and alerts.</p><p>Observability was the final step.</p><p>Or so I thought.</p><p>A production incident involving OpenTelemetry completely changed that perspective.</p><p>Today, I no longer see observability as something that sits beside an application.</p><p>I see it as part of the application itself.</p><p>The Goal</p><p>Our team was introducing distributed tracing to improve visibility across our services.</p><p>Like many modern systems, our application communicated with several downstream APIs. When something failed, we could identify the failure, but understanding where it failed required jumping between logs from multiple services.</p><p>Distributed tracing promised to solve that problem.</p><p>The plan was straightforward:</p><ul><li>Instrument the application.</li><li>* Automatically create spans for outbound HTTP requests.</li><li>* Propagate trace context to downstream services.</li><li>* View the complete request journey in our observability platform.</li></ul><p>On paper, it was a simple rollout.</p><p>Automatic Instrumentation</p><p>Rather than manually modifying every HTTP call, we relied on an HTTP interceptor.</p><p>Every outbound request would automatically receive trace context before being sent.</p><p>That meant every request leaving the application carried enough information for downstream services to continue the same distributed trace.</p><p>No developer had to remember to add tracing headers.</p><p>No business logic needed to change.</p><p>Everything happened automatically.</p><p>Or at least, that was the expectation.</p><p>Something Was Wrong</p><p>Not long after enabling the instrumentation, we started seeing requests fail.</p><p>The strange part was that nothing else had changed.</p><p>The same endpoint.</p><p>The same payload.</p><p>The same authentication.</p><p>The same users.</p><p>Yet requests that had always succeeded were suddenly being rejected.</p><p>Whenever we disabled the interceptor, the requests worked again.</p><p>Enable it…</p><p>The failures returned.</p><p>At that point, it was obvious the business logic wasn’t responsible.</p><p>Something in the instrumentation layer had changed the behavior of the request.</p><p>Following the Evidence</p><p>Our first instinct was to investigate the obvious suspects.</p><p>Was the API unavailable?</p><p>No.</p><p>Were authentication tokens expired?</p><p>No.</p><p>Did the payload change?</p><p>No.</p><p>Was there a networking problem?</p><p>No.</p><p>Eventually we stopped looking at the request body and started examining the request itself.</p><p>Every HTTP request consists of two important parts:</p><ul><li>The payload</li><li>* The headers</li></ul><p>The payloads were identical.</p><p>The headers were not.</p><p>The Difference</p><p>Without instrumentation, the request contained the usual headers.</p><ul><li>Authorization</li><li>* Content-Type</li><li>* Accept</li></ul><p>With instrumentation enabled, additional headers appeared.</p><ul><li>traceparent</li><li>* tracestate</li><li>* baggage (when applicable)</li></ul><p>These headers were being injected automatically by the HTTP interceptor as part of OpenTelemetry’s implementation of the W3C Trace Context standard.</p><p>Nothing unusual.</p><p>In fact, this is exactly how distributed tracing is supposed to work.</p><p>Understanding Trace Context</p><p>Distributed tracing only works if every service handling a request can identify that it belongs to the same transaction.</p><p>That’s the purpose of trace context.</p><p>Instead of every service creating isolated logs, trace context allows multiple services to contribute to one complete trace.</p><p>The most important header is traceparent.</p><p>It contains identifiers that allow downstream services to continue an existing trace rather than creating a new one.</p><p>When everything supports the standard, the result is a beautiful end-to-end view of a request moving through an entire distributed system.</p><p>It’s one of the biggest advantages of OpenTelemetry.</p><p>The Real Problem</p><p>Here’s the interesting part.</p><p>OpenTelemetry wasn’t broken.</p><p>The interceptor wasn’t broken.</p><p>The trace headers were completely valid.</p><p>The problem existed elsewhere.</p><p>Somewhere between our application and the destination service was infrastructure that wasn’t handling the additional tracing headers correctly.</p><p>Depending on an organization’s architecture, that could be:</p><ul><li>An API Gateway</li><li>* A Web Application Firewall (WAF)</li><li>* A Reverse Proxy</li><li>* A Load Balancer</li><li>* A Legacy Service</li><li>* A Security Appliance</li></ul><p>From the application’s perspective, nothing had changed except a few additional HTTP headers.</p><p>From the infrastructure’s perspective, however, those additional headers changed how the request was processed.</p><p>The request never reached the business logic.</p><p>It failed before the application even had a chance to handle it.</p><p>The Lesson That Stayed With Me</p><p>We could have disabled tracing and moved on.</p><p>Instead, we asked a different question.</p><p>Why did we assume observability could simply be switched on after deployment?</p><p>Instrumentation isn’t passive.</p><p>It injects headers.</p><p>It creates spans.</p><p>It propagates context.</p><p>It interacts with gateways, proxies, service meshes, authentication systems, security controls, and downstream applications.</p><p>In other words…</p><p>It changes production traffic.</p><p>That realization completely changed how I think about observability.</p><p>A New Deployment Mindset</p><p>Since that incident, our approach has been different.</p><p>Observability is no longer the final task on a deployment checklist.</p><p>It is planned alongside the application itself.</p><p>Before enabling instrumentation, we validate that:</p><ul><li>Downstream services understand propagated trace context.</li><li>* API gateways and proxies can safely process tracing headers.</li><li>* Security controls don’t reject or modify trace information.</li><li>* Automatic instrumentation behaves exactly as expected.</li><li>* Rollouts happen gradually instead of all at once.</li></ul><p>Instrumentation receives the same level of planning, testing, and validation as any other production feature.</p><p>Because that’s exactly what it is.</p><p>What Every Engineering Team Should Remember</p><p>Modern observability platforms are incredibly powerful.</p><p>Metrics help us understand health.</p><p>Logs help us investigate incidents.</p><p>Distributed traces show us the complete path of a request across dozens of services.</p><p>But those capabilities don’t come for free.</p><p>They become part of the application’s execution path.</p><p>Treating observability as something you “add later” increases the likelihood of discovering compatibility issues in production instead of during deployment.</p><p>Whether you’re using OpenTelemetry or another observability platform, instrumentation deserves architectural consideration — not just operational consideration.</p><p>Final Thoughts</p><p>As engineers, we often celebrate new dashboards, cleaner traces, and better monitoring.</p><p>Those things matter.</p><p>But the biggest lesson from this experience wasn’t about traces.</p><p>It was about engineering discipline.</p><p>Observability isn’t an accessory.</p><p>It isn’t something that lives outside your application.</p><p>It’s part of how your application behaves.</p><p>That one incident fundamentally changed how I approach production rollouts.</p><p>Today, when I review a deployment plan, observability is discussed alongside functionality, security, scalability, and reliability — not after them.</p><p>Because the moment instrumentation starts modifying requests, creating spans, and propagating context, it stops being “just monitoring.”</p><p>It becomes part of the system you’re deploying.</p><p>And that’s why, for me, observability became part of the deployment plan instead of an afterthought.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*1JADLlaZGBdZcLf_lTV4FQ@2x.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1128aad6c12e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Replaced console.log with Pino — and What We Learned Instrumenting Mobile at the Same Time]]></title>
            <link>https://medium.com/@sdakwoji/why-we-replaced-console-log-with-pino-and-what-we-learned-instrumenting-mobile-at-the-same-time-3b852fbeaace?source=rss-7a8293a29b16------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b852fbeaace</guid>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Stanley Dakwoji]]></dc:creator>
            <pubDate>Sat, 13 Jun 2026 23:23:59 GMT</pubDate>
            <atom:updated>2026-06-13T23:28:43.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9N3MpgIr9xdeFG1uqFNG5w.png" /></figure><p>Recently I worked on an observability initiative that spanned two very different environments:</p><ul><li>A Flutter mobile application that needed OpenTelemetry instrumentation from real devices.</li><li>A Next.js admin application where distributed traces were already flowing into our observability platform, but logs were not.</li></ul><p>This article focuses on the server-side change that had the biggest operational impact: replacing console.log with structured logging. I&#39;ll also touch on how mobile instrumentation fit into the same effort, because the languages, runtimes, and constraints were completely different.</p><h3>Part 1 — The Real Problem: Traces Without Logs</h3><p>We had a familiar observability gap.</p><p>SurfaceRuntimeTracingStructured LoggingBackend servicesTypeScript✅✅Admin APITypeScript / Next.js✅❌Mobile appDart / FlutterIn progressIn progress</p><p>When an API becomes slow or starts failing, traces tell you where time was spent.</p><p>Logs tell you what the application decided:</p><ul><li>Request outcome</li><li>Status codes</li><li>Safe contextual information</li><li>Error details</li></ul><p>Without both, mean time to resolution (MTTR) increases.</p><p>console.log felt like logging.</p><p>Operationally, it wasn’t.</p><h3>Part 2 — Why console.log Failed Us</h3><p>Our API routes looked like many Node.js applications:</p><pre>console.log(&quot;something happened&quot;);<br>console.error(&quot;failed&quot;, error);</pre><p>The problems became obvious during incident response:</p><h3>Unstructured Output</h3><p>Easy to print.</p><p>Difficult to search, index, aggregate, and analyze consistently.</p><h3>No Trace Correlation</h3><p>There was no trace_id attached to log events, making it difficult to jump between traces and logs during investigations.</p><h3>Security Risks</h3><p>Dumping entire error objects often exposes more than intended:</p><ul><li>Request headers</li><li>Configuration data</li><li>Authentication tokens</li><li>Third-party client internals</li></ul><h3>Inconsistent Operational Experience</h3><p>Other services already emitted structured logs with a predictable schema.</p><p>The admin application did not.</p><p>Engineers were effectively working with two different logging models.</p><p>We didn’t need more logs.</p><p>We needed better logs.</p><h3>Part 3 — Why Pino</h3><p>We chose Pino because it aligned well with our requirements:</p><ul><li>High-performance structured JSON logging</li><li>Strong OpenTelemetry ecosystem support</li><li>Built-in redaction capabilities</li><li>Straightforward integration with modern Node.js applications</li></ul><p>Other logging libraries can work perfectly well.</p><p>For our environment, Pino provided the shortest path to:</p><ol><li>Structured logs</li><li>Trace correlation</li><li>Consistent operational visibility</li></ol><h3>Part 4 — What We Implemented</h3><h3>Central Logger</h3><p>A server-only logger became the single entry point for application logging.</p><p>Key characteristics:</p><ul><li>JSON output only</li><li>Consistent event structure</li><li>Trace and span identifiers injected from the active OpenTelemetry context</li><li>Production-safe log levels</li><li>Built-in redaction</li></ul><h3>Safe Request Metadata</h3><p>Instead of logging entire request objects, we created allow-listed request metadata.</p><p>Examples included:</p><ul><li>HTTP method</li><li>Route path</li><li>Response status</li><li>Request identifiers</li></ul><p>Sensitive or unnecessary information was excluded.</p><p>Query strings were removed from logged paths.</p><h3>Header Allow-Listing</h3><p>Only explicitly approved header names were logged.</p><p>Everything else was redacted.</p><p>This significantly reduced the risk of accidental credential exposure.</p><h3>Error Handling</h3><p>Structured error logging used a controlled error shape rather than serializing arbitrary objects.</p><p>This prevented large third-party client objects from leaking configuration details into logs.</p><h3>Route Helpers</h3><p>API routes shared common logging helpers.</p><p>Successful requests emitted a standard event:</p><pre>API request handled</pre><p>Failures emitted a structured error event:</p><pre>API_ERROR</pre><p>During incidents, consistency matters more than creativity.</p><h3>Part 5 — Bootstrap Order Matters</h3><p>One lesson stood out:</p><p>Instrumentation load order is not a detail.</p><p>It’s a requirement.</p><p>Our startup sequence looked roughly like this:</p><pre>Load environment<br>→ Start OpenTelemetry SDK<br>→ Initialize logger<br>→ Start application</pre><p>If the logger initialized before telemetry instrumentation, automatic correlation could fail silently.</p><p>Fortunately, we discovered this during implementation rather than after deployment.</p><h3>Production Bundling Matters Too</h3><p>Modern frameworks optimize aggressively.</p><p>In production builds, instrumentation libraries and logging libraries may need special handling to ensure runtime integrations remain active.</p><p>Development success does not guarantee production success.</p><p>Treat observability verification as a release gate.</p><h3>Part 6 — What We Deliberately Did Not Change</h3><p>We kept scope intentionally small.</p><p>Out of scope:</p><ul><li>Browser-side React logging</li><li>Development-only console output</li><li>Alternative log collection architectures</li></ul><p>The goal was simple:</p><p>Improve observability for server-side API routes.</p><p>Nothing more.</p><h3>Part 7 — Mobile Instrumentation: Same Goal, Different Rules</h3><p>At the same time, we were instrumenting a Flutter application using OpenTelemetry.</p><p>Although both efforts served the same observability goals, the implementation patterns were completely different.</p><p>ServerMobileRuns in controlled infrastructureRuns on user devicesStructured logs emitted to stdoutLogs exported from the applicationServer-side request handlingClient-side interactions and network callsInfrastructure-managed telemetryIn-app telemetry SDK</p><p>For mobile, we focused on:</p><ul><li>Distributed tracing</li><li>HTTP client instrumentation</li><li>Navigation and screen events</li><li>Key user journey tracking</li><li>Structured application logs</li></ul><p>Metrics were intentionally deferred until there was a clear operational need.</p><p>In many mobile environments, traces and logs provide most of the actionable signal early on.</p><h3>Part 8 — How We Verified Success</h3><p>Verification was intentionally simple.</p><h3>Build Validation</h3><ul><li>Build succeeds</li><li>Type checking succeeds</li><li>No runtime startup errors</li></ul><h3>Smoke Testing</h3><ul><li>API routes return expected responses</li><li>Structured log events appear as expected</li></ul><h3>Correlation Testing</h3><p>The most important test:</p><ol><li>Trigger an API request.</li><li>Capture the trace identifier from a log event.</li><li>Open the corresponding trace in the observability platform.</li></ol><p>If that workflow works consistently, engineers can move seamlessly between traces and logs during incident response.</p><h3>Lessons Learned</h3><h3>Traces Without Logs Are Incomplete</h3><p>Tracing explains where time was spent.</p><p>Logs explain what happened.</p><p>You need both.</p><h3>Structured Logging Is an Operational Tool</h3><p>Logging is not simply writing text to stdout.</p><p>It’s creating reliable, searchable operational data.</p><h3>Redaction Is Part of the Design</h3><p>Security controls belong in the logging strategy from day one.</p><p>Not as a follow-up task.</p><h3>Correlation Changes Incident Response</h3><p>Before correlation, engineers often search logs by timestamp and guesswork.</p><p>After correlation, a trace identifier becomes a direct bridge between telemetry signals.</p><h3>Different Runtimes Need Different Patterns</h3><p>The goal may be shared.</p><p>The implementation rarely is.</p><p>Server-side applications, mobile applications, and backend services each require observability approaches tailored to their runtime constraints.</p><h3>Closing</h3><p>The most impactful server-side change in this project was straightforward to describe but easy to implement incorrectly:</p><p>Stop relying on console.log for production API routes and adopt structured logging with trace correlation.</p><p>At the same time, we extended observability into a mobile application so client and server telemetry could be viewed together.</p><p>Observability is rarely about installing a single library.</p><p>It’s about producing the right signals, in the right format, with enough context to help engineers understand what happened when systems inevitably fail</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b852fbeaace" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>