<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vineet Gandhi on Medium]]></title>
        <description><![CDATA[Stories by Vineet Gandhi on Medium]]></description>
        <link>https://medium.com/@vgvineet4?source=rss-b5e3d9fc4c4b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*iP72vcybEN79uPFj.jpg</url>
            <title>Stories by Vineet Gandhi on Medium</title>
            <link>https://medium.com/@vgvineet4?source=rss-b5e3d9fc4c4b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:34:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vgvineet4/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Timestamp based ordering of Mircoservices logs in ElasticSeach, Filebeat]]></title>
            <link>https://medium.com/@vgvineet4/timestamp-based-ordering-of-mircoservices-logs-in-elasticseach-filebeat-2b2517b9a7e9?source=rss-b5e3d9fc4c4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b2517b9a7e9</guid>
            <category><![CDATA[filebeat]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[timestamp]]></category>
            <dc:creator><![CDATA[Vineet Gandhi]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 09:32:06 GMT</pubDate>
            <atom:updated>2020-07-06T09:32:06.663Z</atom:updated>
            <content:encoded><![CDATA[<h3>Logging architecture</h3><ul><li>Logs generated from each microservice is shipped by filebeat to elasticsearch which can be searched in kibana.</li><li>Logs generated are in json format and each log has an associated time field which can be in any format based on the encoder (<a href="https://github.com/logstash/logstash-logback-encoder">logstash-logback-encoder</a> or <a href="http://logback.qos.ch/manual/appenders.html">default logback timestamp</a> field) used.</li><li>Most of the logs have correlationId associated which helps in request tracing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FIVv3vJ_11nAyW4BKENrQ.png" /></figure><h3>Problems</h3><ul><li>If we search by say correlationId in Kibana, logs are not ordered by the time they actually got generated at individual microservice rather the logs are ordered as per @timestamp field which gets generated when filebeat actually harvest the log file.</li><li>In case the microservice has <strong>@timestamp</strong> field in json log its getting overridden by filebeat <strong>@timestamp</strong> field.</li></ul><p>Let me try explaining by giving an example.</p><pre>#log from first microservice</pre><pre>{<br> &quot;timestamp&quot;: &quot;1594022068798&quot;,<br> &quot;level&quot;: &quot;INFO&quot;,<br> &quot;thread&quot;: &quot;**********-c9da2787-378a-4b4a-a00a-064327fe5b6e-*****-22&quot;,<br> &quot;mdc&quot;: {<br>  &quot;kafka_correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br>  &quot;correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br> },<br> &quot;logger&quot;: &quot;com.***.***.kafka.streams.****&quot;,<br> &quot;message&quot;: &quot;message body&quot;,<br> &quot;context&quot;: &quot;default&quot;<br>}</pre><pre># log from second microservice</pre><pre>{<br> &quot;<a href="http://twitter.com/timestamp">@timestamp</a>&quot;: &quot;2020-07-06T07:54:35.364Z&quot;,<br> &quot;source_host&quot;: &quot;****-5cf587bd5d-wnmwk&quot;,<br> &quot;file&quot;: &quot;***.java&quot;,<br> &quot;method&quot;: &quot;*****&quot;,<br> &quot;level&quot;: &quot;WARN&quot;,<br> &quot;line_number&quot;: &quot;1064&quot;,<br> &quot;thread_name&quot;: &quot;pool-2-thread-2&quot;,<br> &quot;mdc&quot;: {<br>  &quot;kafka_correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br>  &quot;correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br> },<br> &quot;<a href="http://twitter.com/version">@version</a>&quot;: 1,<br> &quot;logger_name&quot;: &quot;org.apache.kafka.clients.NetworkClient&quot;,<br> &quot;message&quot;: &quot;message body&quot;,<br> &quot;class&quot;: &quot;org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater&quot;,<br> &quot;mdc&quot;: {}<br>}</pre><p>First microservice generates the log with <strong>timestamp</strong> field and second microservice generates log with <strong>@timstamp </strong>field<strong>.</strong></p><p>Filebeat harvester picks the respective microservices file log independent of which log is written first and also adds <strong>@ timestamp </strong>when the file is read by filebeat.</p><pre>#log from first microservice</pre><pre>{<br> &quot;@timestamp&quot; : &quot;2020-07-06T07:55:02.211Z&quot;,<br> &quot;timestamp&quot;: &quot;1594022068798&quot;,<br> &quot;level&quot;: &quot;INFO&quot;,<br> &quot;thread&quot;: &quot;**********-c9da2787-378a-4b4a-a00a-064327fe5b6e-*****-22&quot;,<br> &quot;mdc&quot;: {<br>  &quot;kafka_correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br>  &quot;correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br> },<br> &quot;logger&quot;: &quot;com.***.***.kafka.streams.****&quot;,<br> &quot;message&quot;: &quot;message body&quot;,<br> &quot;context&quot;: &quot;default&quot;<br>}</pre><pre># log from second microservice</pre><pre>{<br> &quot;<a href="http://twitter.com/timestamp">@timestamp</a>&quot;: &quot;2020-07-06T07:54:47.961Z&quot;,<br> &quot;source_host&quot;: &quot;****-5cf587bd5d-wnmwk&quot;,<br> &quot;file&quot;: &quot;***.java&quot;,<br> &quot;method&quot;: &quot;*****&quot;,<br> &quot;level&quot;: &quot;WARN&quot;,<br> &quot;line_number&quot;: &quot;1064&quot;,<br> &quot;thread_name&quot;: &quot;pool-2-thread-2&quot;,<br> &quot;mdc&quot;: {<br>  &quot;kafka_correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br>  &quot;correlationId&quot;: &quot;e9c60d05-b53b-11ea-912e-61e12362b00e&quot;,<br> },<br> &quot;<a href="http://twitter.com/version">@version</a>&quot;: 1,<br> &quot;logger_name&quot;: &quot;org.apache.kafka.clients.NetworkClient&quot;,<br> &quot;message&quot;: &quot;message body&quot;,<br> &quot;class&quot;: &quot;org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater&quot;,<br> &quot;mdc&quot;: {}<br>}</pre><p>From the above logs statements @timestamp is added in both log statements and also if searched by <strong>correlationId </strong>then log from second microservice will be shown first in kibana.</p><h3>Solution</h3><ul><li>Made sure to convert all different timestamp fields to a common field <strong>@timestamp</strong>.</li><li>Elastic search provides <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-processors.html">ingest data processor</a> which are more like pre processors before data is actually indexed.</li></ul><pre>PUT _ingest/pipeline/preprocessDate<br>{<br>  &quot;description&quot; : &quot;map log timestamp to <a href="http://twitter.com/timestamp">@timestamp</a>&quot;,<br> &quot;processors&quot; : [<br>    {<br>      &quot;date&quot; : {<br>        &quot;field&quot; : &quot;timestamp&quot;,<br>        &quot;target_field&quot; : &quot;<a href="http://twitter.com/timestamp">@timestamp</a>&quot;,<br>        &quot;formats&quot; : [&quot;UNIX_MS&quot;],<br>        &quot;timezone&quot; : &quot;UTC&quot;,<br>        &quot;on_failure&quot; : [<br>          {<br>            &quot;set&quot; : {<br>              &quot;field&quot; : &quot;pipelineError&quot;,<br>             &quot;value&quot; : &quot;{{ _ingest.on_failure_message }}&quot;<br>            }<br>          }<br>        ]<br>      }<br>    }<br>  ]<br>  <br>}</pre><p>The above code maps the timestamp to @timestamp and in case no timestamp is present it adds a field <em>pipelineError</em> with error message.</p><ul><li>Use ingestNodeProcessor created in elastic search in filebeat configuration file filebeat.yml.</li></ul><pre>output.elasticsearch:  <br>enabled: true  <br>hosts: [ &quot;${ELASTIC_SEARCH_HOST}&quot; ]  <br>protocol: &quot;https&quot;  <br>index: &quot;filebeat-%{+yyyy.MM.dd}&quot;  <br>pipeline: preprocessDate</pre><ul><li>To solve the second problem of keeping actual <strong>@timstamp</strong> value generated in log added below configuration in filebeat.yml to pick value from the decoded json log.</li></ul><pre>json.overwrite_keys: true </pre><p>The above changes solves the problem for me to keep the logs ordered by <strong>@timestamp</strong> field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b2517b9a7e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>