<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dmitry Kolomiets on Medium]]></title>
        <description><![CDATA[Stories by Dmitry Kolomiets on Medium]]></description>
        <link>https://medium.com/@dmitrykolomiets?source=rss-3e42fa96c5e9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xTPcrht47G7R7od_.</url>
            <title>Stories by Dmitry Kolomiets on Medium</title>
            <link>https://medium.com/@dmitrykolomiets?source=rss-3e42fa96c5e9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:59:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dmitrykolomiets/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Jaeger Quick Start v0.2]]></title>
            <link>https://medium.com/better-programming/jaeger-quick-start-v0-2-adce4d1bfe64?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/adce4d1bfe64</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tracing]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Mon, 30 May 2022 15:42:39 GMT</pubDate>
            <atom:updated>2022-05-31T07:54:42.757Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c"><em>Musings on Tracing</em></a></h4><h4>Service Performance Monitoring support</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*buoNNYMSsHH1yXTnKOPPFQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@aoddeh?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ahmad Odeh</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Summary</h3><p>We live in exciting times for distributed tracing and the OpenTelemetry community. Things are moving fast and it may be difficult to keep up with the latest changes.</p><p>This is a state of play post to share the latest updates for Jaeger Quick Start and a hint at the ecosystem changes that are likely to happen soon.</p><p>A quick recap — Jaeger Quick Start helps you to deploy a highly available Jaeger — open source, end-to-end distributed tracing backend on the Amazon Web Services (AWS) Cloud. You can find more in the previous post:</p><p><a href="https://betterprogramming.pub/introducing-jaeger-quick-start-deploying-on-aws-f0ee5d398e8a">Introducing Jaeger Quick Start — Deploying on AWS</a></p><h3>Deployment architectures</h3><h4>In-memory storage backend</h4><p>There are multiple ways you can export your traces to Jaeger. The most basic one is to use OpenTelemetry SDK and Jaeger Exporter (see language-specific packages, e.g. <a href="https://github.com/open-telemetry/opentelemetry-java/tree/main/exporters/jaeger">Java</a>, <a href="https://github.com/open-telemetry/opentelemetry-python/tree/main/exporter/opentelemetry-exporter-jaeger">Python</a>, <a href="https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-exporter-jaeger">JavaScript</a>, <a href="https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Exporter.Jaeger/README.md">.NET</a>, etc.) to export the traces directly from the application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Z37JmngRrOclYwH6xnUrqg.png" /><figcaption>The most basic Jaeger deployment (in-memory storage backend)</figcaption></figure><p>Jaeger Quick Start deployed with the default configuration, creates the simplest <a href="https://www.jaegertracing.io/docs/1.34/deployment/#all-in-one">Jaeger All-In-One</a> instance that keeps all the traces in memory. It is still a fully-featured Jaeger though — you can collect the traces, use search, see the dependencies between services, etc.</p><p>Albeit, you will lose all the data if the Jaeger instance dies or is rebooted. This mode is perfect for evaluations, test environments, and proof-of-concept work.</p><p>The following ports are exposed for integrations:</p><ul><li>14250 (gRPC) <br>Accepts trace spans in protobuf format</li><li>14268 (HTTP) <br>Accepts trace spans in Thrift format</li><li>80 / 443 (HTTP)<br>/api/* endpoints and Jaeger UI at /</li><li>[optional] 14269 (HTTP) <br>Admin port: health check at / and metrics at /metrics</li></ul><h4>Elasticsearch storage backend</h4><p>You may want a more substantial setup for your production environment, preferably a highly available architecture with the trace data persisted in a durable storage and Jaeger instances distributed across multiple <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-availability-zones">availability zones</a> for resiliency.</p><p>Jaeger Quick Start supports this deployment scenario by using multiple <a href="https://www.jaegertracing.io/docs/1.34/deployment/#collector">Collector</a> and <a href="https://www.jaegertracing.io/docs/1.34/deployment/#query-service--ui">Query service</a> Fargate tasks behind a load balancer and auto-scaling groups. This provides resiliency and elasticity in case of trace traffic spikes. <a href="https://aws.amazon.com/opensearch-service">Amazon OpenSearch Service</a> is used as durable storage — again, distributed across multiple availability zones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*CVdPaIjSTtHdfXV6nEnd1Q.png" /><figcaption>Jaeger with Elasticsearch storage backend</figcaption></figure><p>The following ports are exposed for integrations:</p><ul><li>14250 (gRPC) <br>Accepts trace spans in protobuf format</li><li>14268 (HTTP) <br>Accepts trace spans in Thrift format</li><li>80 / 443 (HTTP)<br>/api/* endpoints and Jaeger UI at /</li><li>[optional] 14269 (HTTP) <br>Collector admin port: health check at / and metrics at /metrics</li><li>[optional] 16687 (HTTP) <br>Query service admin port: health check at / and metrics at /metrics</li></ul><h4>OpenTelemetry Collector in front of Jaeger</h4><p>You may have noticed that the previous two options presented do not expose a standard <a href="https://opentelemetry.io/docs/reference/specification/protocol/">OTLP</a> port — you have to either use <a href="https://www.jaegertracing.io/docs/1.34/architecture/#jaeger-client-libraries">Jaeger Client libraries</a> (in process of being retired), <a href="https://www.jaegertracing.io/docs/1.34/architecture/#agent">Jaeger Agent</a>, or <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerexporter">Jaeger OpenTelemetry Collector exporter</a> to send the traces to Jaeger.</p><p>This may not be feasible in some scenarios — maybe you already have a large application that relies on OTLP that you can’t modify or you can’t easily install Jaeger OpenTelemetry Collector exporter as, for example, the case with <a href="http://AWS Distro for OpenTelemetry">AWS Distro for OpenTelemetry</a> (ADOT)—Jaeger exporter is not included into <a href="https://github.com/aws-observability/aws-otel-collector">ADOT Collector</a>.</p><p>To overcome this, Jaeger Quick Start provides an option to deploy a highly available OpenTelemetry Collector in front of Jaeger Collector — pre-configured to accept traces using OTLP and export them to Jaeger. Collector instances are distributed across availability zones and backed by an auto-scaling group following <a href="https://docs.aws.amazon.com/whitepapers/latest/real-time-communication-on-aws/high-availability-and-scalability-on-aws.html">high availability and scalability principles</a>.</p><p>This is a turn-key solution that ensures your application can continue to use OTLP to export the traces to Jaeger. As we’ll see in a moment, this integration also enables additional interesting options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ASTjOxqGrobFiQ11qiQIw.png" /><figcaption>Jaeger with a front OpenTelemetry Collector</figcaption></figure><blockquote>OpenTelemetry collector can be deployed in front of Jaeger regardless of the storage configured — both In-memory and Elasticsearch options are supported.</blockquote><p>The following ports are exposed for integrations (in addition to Jaeger-specific ports we already covered above):</p><ul><li>4317 (gRPC / HTTP) <br>OTLP receiver</li><li>[optional] 1777 (HTTP) <br><a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/pprofextension">pprof extension for OpenTelemetry Collector</a></li><li>[optional] 55679 (HTTP) <br><a href="https://github.com/open-telemetry/opentelemetry-collector/tree/main/extension/zpagesextension">zPages extension for OpenTelemetry Collector</a></li></ul><h4>OpenTelemetry Collector and Jaeger operational metrics</h4><p>Jaeger Collector, Jaeger Query service and OpenTelemetry Collector expose a number of operational metrics you may want to collect and expose in operational dashboards to monitor and alert on.</p><p>To simplify this task of collecting the metrics across all instances, Jaeger Quick Start provides an option to scrape and export metrics to a centralized <a href="https://aws.amazon.com/prometheus/">Amazon Managed Service for Prometheus</a> workspace. You can either request creation of a new workspace or use an existing one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q0OmOxshHHLazHuc8mE8CQ.png" /><figcaption>Jaeger with optional integration with Amazon Managed Service for Prometheus</figcaption></figure><p>This setup is essential for production-quality environments where operational aspects of the telemetry platform are crucial.</p><h3>Service Performance Monitoring (SPM)</h3><h4>SPM with Jaeger history</h4><p>The deployment options summarized in the previous section give you lots of flexibility in architecting a distributed tracing platform with Jaeger. There is another option I’d like to cover though to spur your interest and demonstrate how powerful the distributed tracing can be.</p><p>There was an interesting idea that folks from <a href="https://logz.io/">logz.io</a> proposed <a href="https://github.com/jaegertracing/jaeger/issues/2954">back in April 2021</a> and announced its availability in December 2021:</p><p><a href="https://logz.io/blog/monitoring-microservices-opentelemetry-jaeger/">From Distributed Tracing to APM: Taking OpenTelemetry &amp; Jaeger Up a Level | Logz.io</a></p><p>Here is the high-level thinking behind this feature:</p><ul><li>OpenTelemetry Collector is a focal point of all the traces from different services</li><li>Traces contain lots of information, such as error codes and latencies among other things</li><li>What if we configure OpenTelemetry Collector to use the incoming traces and calculate <a href="https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/">R.E.D. metrics</a> for each service (rate, errors, duration)?</li></ul><p>In fact, this is exactly what the Jaeger community did by introducing <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanmetricsprocessor">Span Metrics Processor</a> for OpenTelemetry Collector! By utilizing the OpenTelemetry stack, the processor generates metrics that can be used by Jager UI to present R.E.D. metrics across all your services. This feature is documented as <a href="https://www.jaegertracing.io/docs/1.34/spm/">Service Performance Monitoring</a> in Jaeger documentation and is part of the <a href="https://logz.io/platform/distributed-tracing/#backend">commercial offering from Logz.io</a>.</p><blockquote>Service Performance Monitoring is still an experimental feature, as indicated in Jaeger documentation. There may be further changes in both the OpenTelemetry and Jaeger sides of the configuration.</blockquote><h4>SPM in Jaeger Quick Start</h4><p>As we covered in the previous section, Jaeger Quick Start supports the deployment of all the necessary components for the SPM:</p><ul><li>Jaeger Collector / Query Services + UI</li><li>OpenTelemetry Collector</li><li>Amazon Managed Service for Prometheus workspace</li></ul><p>Enabling SPM is therefore a matter of proper configuration of all these components (configure Otel Collector to use Span Metrics Processor and expose the metrics to Prometheus, configure Jaeger to pull the metrics from Prometheus, etc.)</p><p>Jaeger Quick Start automates these steps and provides SPM support via a simple on/off configuration option. It enables a low-friction way to evaluate SPM in your own environment with real traces from the services:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1pOPK19n_LQ0P0RRga0jzg.png" /><figcaption>Jaeger with Service Performance Monitoring support</figcaption></figure><h4>SPM in Jaeger Quick Start — demo</h4><p>Jaeger Quik Start comes with a .taskcat.yml file that contains a number of tests crafted to deploy different Jaeger architectures we covered in this post. This file relies on <a href="https://github.com/aws-ia/taskcat">TaskCat</a> to automate the deployments which I’m going to use below. You can also refer to the <a href="https://github.com/kolomiets/quickstart-jaeger#deployment">manual deployment instructions</a> to deploy the Quick Start using AWS Console.</p><p>Use the following command to deploy Jaeger Quick Start with SPM in your AWS environment:</p><p>taskcat test run -n -t inmemory-with-spm</p><p>Once deployed, you can get Jaeger Query endpoint from the output parameters of the stack:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2YIaeyDENZgZk3aQKA40TQ.png" /><figcaption>JaegerURL output parameter of the Quick Start CloudFormation stack</figcaption></figure><p>Jaeger Quick Start also provides a simple span-generator application that can be used to generate sample traces and export them to OpenTelemetry Collector that sits in front of Jaeger (remember that <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanmetricsprocessor">Span Metrics Processor</a> that captures R.E.D. metrics is running in the context of OpenTelemetry Collector). You can start the generator application by using the following command (replace {otel-collector-domain} with the domain name from the stack output parameter we discussed above):</p><p>OTEL_COLLECTOR_HOST={otel-collector-domain} docker-compose up</p><p>If you now navigate to the Jaeger Query URL, you will see the traces generated by the application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFITbKNzR_qYaBm9GeoKyQ.png" /><figcaption>Jaeger trace details view</figcaption></figure><p>This is all standard distributing tracing functionality though. What is new is the Monitor tab that presents calculated SPM metrics across all services and operations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ygPlSzJ08dvlecNBdJZVg.png" /><figcaption>Jaeger SPM Monitor tab</figcaption></figure><p>Feel free to play with this application and review the metrics captured. To get more detailed information about the feature, refer to the official <a href="https://www.jaegertracing.io/docs/1.34/spm/">Jaeger documentation on the topic</a>.</p><h3>Next steps</h3><p>Jaeger Quick Start v0.2 is a big milestone as it enables a number of common deployment scenarios and showcases a major experimental feature from a joint OpenTelemetry and Jaeger communities (SPM). The project is still in its infancy though and this section highlights the obvious next steps.</p><h4>OTLP support in Jaeger</h4><blockquote><strong>31.05.2022 update</strong>: <a href="https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c">OTLP support in Jaeger is released</a></blockquote><p>You may have noticed that Jaeger does not support the OTLP protocol for trace submission. In some environments, it may be a limiting factor and Jaeger Quick Start works around this by deploying the OpenTelemetry Collector in front of Jaeger to accept OTLP traces and provide additional features such as SPM.</p><p>Jaeger team recognized the limitation and is working on addressing it:</p><p><a href="https://github.com/jaegertracing/jaeger/issues/3625">Accept OpenTelemetry trace data via OTLP · Issue #3625 · jaegertracing/jaeger</a></p><p>At the time of writing, the OTLP support is <a href="https://github.com/jaegertracing/documentation/pull/578">slated for 1.35 Jaeger release</a>. Once the feature is available, I’ll adjust the Quick Start to make use of the direct OTLP integration.</p><h4>Apache Kafka support</h4><p>Another very common Jaeger architecture involves an Apache Kafka stream between Jaeger Collector and the long-term durable storage such as Elasticsearch. This is often useful for high-volume ingestion when an intermediate buffer is needed between the collector and the storage. Refer to the <a href="https://www.jaegertracing.io/docs/1.34/performance-tuning/#consider-using-apache-kafka-as-intermediate-buffer">Jaeger documentation</a> for additional details on this pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lF0Tsw2W3JKcj7WwLLQBMA.png" /><figcaption>Jaeger with Elasticsearch storage backend and Kafka stream</figcaption></figure><p>At the moment, Jaeger Quick Start does not support Apache Kafka streaming architecture. There is an open issue on GitHub to track this work. Please visit the GitHub page to see it.</p><h3>Conclusion</h3><p>Do try SPM in your environment — this is an interesting experimental project that builds intelligence on top of distributed tracing. SPM shows the potential of tools and processes built with raw trace data — the tools and processes that can quickly become an important part of the commercial offerings in the distributed trace space, as <a href="https://logz.io/">Logz.io</a> recently proved.</p><h3>Resources</h3><ul><li><a href="https://github.com/kolomiets/quickstart-jaeger">kolomiets/quickstart-jaeger</a> — Jaeger Quick Start project</li><li><a href="https://kolomiets.github.io/quickstart-jaeger/">Jager Quick Start Deployment guide</a></li><li><a href="https://www.jaegertracing.io/docs/1.34/spm/">Jaeger Service Performance Monitoring (SPM)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adce4d1bfe64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/jaeger-quick-start-v0-2-adce4d1bfe64">Jaeger Quick Start v0.2</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modest Architectures]]></title>
            <link>https://medium.com/better-programming/designing-and-building-modest-architectures-2a0a9978234b?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a0a9978234b</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Thu, 19 May 2022 16:46:32 GMT</pubDate>
            <atom:updated>2022-12-13T08:18:50.810Z</atom:updated>
            <content:encoded><![CDATA[<h4>A playful methodology for solid results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9J6wnL8tU9Ma31_rZU-7g.png" /><figcaption>Photo by <a href="https://unsplash.com/@cristofer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Cristofer Maximilian</a> on <a href="https://unsplash.com/s/photos/modest-clothes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>This is a short post about principles I apply to everything I build — whether it is a class interface, web API, <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">Infrastructure as Code</a> (IaC) template, or architectural design.</p><p>I didn’t plan for this; I was thinking about principles for building good IaC templates, but I quickly realized that these are the general principles I constantly use. I put together a list, took the first letters, and used <a href="https://wordfinder.yourdictionary.com/">WordFinder</a> to get a word. I was amused by how good the word was:</p><blockquote><strong>Modest</strong> — not wanting to talk about your abilities or achievements and to say that you are good at something, even when you are — used to show approval.<strong><br><br></strong><em> — </em><a href="https://www.ldoceonline.com/dictionary/modest"><em>Longman Dictionary of Contemporary English</em></a></blockquote><p>The principles are:</p><ul><li>Meaningful defaults</li><li>Orthogonality</li><li>Documentation</li><li>Extensibility</li><li>Security</li><li>Testability</li></ul><p>So here we are, one more mnemonic acronym. There is always a place for a new one.</p><h3>The Principles</h3><p>Let&#39;s start by covering the principles. Just for your amusement, I will provide examples from the IaC domain to illustrate that they are quite general.</p><h4>Meaningful defaults</h4><blockquote>Simple things should be simple, complex things should be possible.<br><a href="https://en.wikiquote.org/wiki/Alan_Kay">Alan Kay</a></blockquote><p>Meaningful defaults help you minimize the friction required to use your artefact. As with API design, consider the <a href="https://www.w3schools.com/python/ref_list_sort.asp">sort</a> method in Python:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e31cad56c3aeb5c505abcf55e9bb856d/href">https://medium.com/media/e31cad56c3aeb5c505abcf55e9bb856d/href</a></iframe><p>Note that by default, sort returns the items in ascending order. You can request descending order by providing a non-default parameter, as shown below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3596961c19f7d926c97db6e07d6117f4/href">https://medium.com/media/3596961c19f7d926c97db6e07d6117f4/href</a></iframe><p>Similarly, in the IaC world — imagine you have a large CloudFormation stack with tens of parameters. For the sake of argument, <a href="https://kolomiets.github.io/quickstart-jaeger/#_parameter_reference">Jaeger Quick Start has more than 50</a>. Still, all parameters have default values, and you do not have to change them if you want to try the Quick Start out. This frictionless experience builds up confidence and spurs further experimentation.</p><h4>Orthogonality</h4><p>Orthogonality is an important principle in software design. As <a href="https://en.wikipedia.org/wiki/Orthogonality_(programming)">Wikipedia defines it</a>, an orthogonal operation changes just one thing without affecting others. A traditional example is <a href="https://en.wikipedia.org/wiki/Pure_function">pure functions</a> — pure functions do not have side effects, and therefore, it is easier to compose multiple functions and reason about them.</p><p>An IaC example may be a bit more difficult to grasp. Let’s say we have a large infrastructure deployment that provides the following options:</p><ul><li>DNS: Disabled / Private Hosted Zone / Public Hosted Zone</li><li>Storage backend: In-Memory Storage / Elasticsearch</li><li>Export metrics: Disabled / Enabled</li></ul><p>These are orthogonal options — you can mix and match them as you want, and you will always get a valid combination:</p><ul><li>No DNS configuration + In-Memory Storage + Disabled metrics</li><li>Public DNS + Elasticsearch + Enabled metrics</li></ul><p>Approaching IaC design with orthogonality in mind usually results in solutions that are easier to test and support, as all features are independent.</p><h4>Documentation</h4><p>Documentation is one of the most neglected areas of software engineering. Good documentation is very difficult, and (almost) nobody likes to write it. It is not clear what is more challenging — creating sophisticated code or writing meaningful documentation for it.</p><p>To be clear, by documentation, I don’t mean comments like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a6c9170337ead0e8d97787d1183f2da/href">https://medium.com/media/4a6c9170337ead0e8d97787d1183f2da/href</a></iframe><p>Trivial things don’t need an explanation. It’s more about more involved snippets that you can’t easily comprehend without knowing some background information:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48ed60918cdb8b92c5af0056d4e229da/href">https://medium.com/media/48ed60918cdb8b92c5af0056d4e229da/href</a></iframe><p>Knowing what you need to document comes with practice, like everything else. You need to write (and read!) a lot of documentation to know what good looks like. If in doubt, explain your code to an imaginary friend (or a <a href="https://en.wikipedia.org/wiki/Rubber_duck_debugging">duck</a>) and think about the things that may raise their eyebrows (not sure if ducks have the eyebrows, though).</p><p>In the IaC domain, documentation plays a central role as well. How many times have you seen templates with undocumented parameters so you had to guess the correct values you can use? How many times have you guessed incorrectly?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t9GEbmVLHvbX2dG3xP09Ig.png" /><figcaption>CloudFormation template with no documentation for parameters</figcaption></figure><p>Would it be better to have parameters logically grouped, with meaningful descriptions and links to web pages where clients can get further information?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*klMoBQgiYQh1H1b1NQgI-w.png" /><figcaption>A more polished version with parameters grouped and documented</figcaption></figure><p>Remember the last time you complained about a sluggish mobile application, unintuitive user interface, or crappy error message. That is how your fellow teammates may feel working with your IaC template.</p><p>Do better; they deserve it!</p><h4>Extensibility</h4><p>Whether you design a class that will be used by your teammates or build a large-scale microservices architecture — extensibility is something you probably need to think about. Likely, you haven’t envisioned all the possible usage scenarios, so creating an open solution would benefit you long term.</p><blockquote>Open here is for O in <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> — another popular mnemonic and something you may want to remember for your next interview. It would be amusing (for both parties) if you talk about MODEST principles with your interviewer as well.</blockquote><p>An example from the IaC domain: Imagine that you build infrastructure for a product and deploy resources that other teams will use— VPC, security groups, IAM roles, etc. Later, a different team wants to deploy a microservice in the VPC. They would probably need a VPC ID, a list of subnets, and maybe a security group to attach to their service. What would be the process for the team to retrieve this information?</p><p>They can open AWS Console, get the identifiers/ARNs and hardcode them in service configuration files. If you have multiple environments (Dev/Test/Prod) — ask them to repeat the process for each environment (and run).</p><p>Surely we can do better. What if, as part of your IaC implementation, we provide a set of SSM parameters with all the necessary identifiers? Then a service team can import the bits they need using the well-defined parameter names. Treat these parameters as your IaC “public interface” — once published, you have to maintain them and ensure backward compatibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cp_hC40g2QoGCwJkxKhOIw.png" /></figure><h4>Security</h4><p>Security is a catch-all principle. If you don’t know where to start, start with security.</p><p>Seriously though, you need to think about security, regardless of what you are building. From static code analysis, container runtime security tools, network scanners, log analysis, anomaly detection, and IaC security tools — the spectrum is huge. <a href="https://www.thoughtworks.com/radar">Thoughtworks Technology Radar</a> is a good place to start.</p><h4>Testability</h4><p>I think we’ve made very good progress over the years — even the legendary <a href="https://stackoverflow.com/questions/67299/is-unit-testing-worth-the-effort">Is Unit Testing worth the effort</a> StackOverflow question is closed for comments.</p><p>Let’s postulate the following for the sake of the final MODEST principle: testing is good and automated testing is even better.</p><p>Designing for testability means that your artefact can be efficiently tested either during development or by your clients during integration. Think about load/integration tests the clients may want to conduct against the class/API/IaC/architecture you developed.</p><p>Continuing our IaC examples, you can do a lot to make your IaC code testable. Start with unit tests for your IaC provider and consider actual deployment tests for different combinations of parameters and target regions. This is especially valuable if you need to support non-standard regions, such as China or GovCloud. Here is an example of a <a href="https://github.com/aws-ia/taskcat">TaskCat</a> test file (from <a href="https://github.com/kolomiets/quickstart-jaeger">quickstart-jaeger repository</a>):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a861b39abea93645fd185a461651f122/href">https://medium.com/media/a861b39abea93645fd185a461651f122/href</a></iframe><p>Whatever you are doing, there is a way to test it.</p><h3>Conclusion</h3><p>Being modest is not always a good life strategy, but designing and building MODEST architectures is something you should consider. Apologies for this mischievous nonsense. I will be back with more serious thoughts on distributed tracing and OpenTelemetry shortly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a0a9978234b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/designing-and-building-modest-architectures-2a0a9978234b">Modest Architectures</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS Distro for OpenTelemetry with Jaeger]]></title>
            <link>https://medium.com/better-programming/using-aws-distro-for-opentelemetry-with-jaeger-acf4df3a8e37?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/acf4df3a8e37</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[tracing]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Tue, 10 May 2022 18:06:05 GMT</pubDate>
            <atom:updated>2022-05-10T18:06:05.708Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">Musings on Tracing</a></h4><h4>A tale of two Collectors</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s3D-JMF5HXwgvaJNQVU_CA.png" /><figcaption>AWS Distro for OpenTelemetry and Jaeger is destined to be a perfect match!</figcaption></figure><h3>Summary</h3><p>So far I’ve been talking about different aspects of <a href="https://betterprogramming.pub/trace-context-propagation-with-opentelemetry-b8816f2f065e">running OpenTelemetry in distributed architectures</a> and introduced <a href="https://betterprogramming.pub/introducing-jaeger-quick-start-deploying-on-aws-f0ee5d398e8a">Jaeger Quick Start</a> to simplify the deployment of this tracing platform in the AWS environment. Now it is time to bring everything together and demonstrate a complete example of using <a href="https://aws.amazon.com/otel/?otel-blogs.sort-by=item.additionalFields.createdDate&amp;otel-blogs.sort-order=desc">AWS Distro for OpenTelemetry</a> (ADOT) and <a href="https://www.jaegertracing.io/">Jaeger</a> as a tracing backend.</p><h3>The goal for the series</h3><p>It’s long overdue to clearly state the goal for this <a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">series of posts</a>. Better late than never though.</p><p>I want to persuade you that tracing is a practical and valuable concept, not just theoretically interesting. I’m approaching this from DevOpsy angle of infrastructure deployment and initial configuration — the boring topics we need to settle before we move on to more useful things.</p><p>In particular, I’m aiming for the audience that is familiar with AWS environment and has some experience with <a href="https://aws.amazon.com/xray/">AWS X-Ray</a> — the official tracing offering from AWS. Through <a href="https://opentelemetry.io/">OpenTelemetry</a> and other open-source products like <a href="https://www.jaegertracing.io/">Jaeger</a>, I demonstrate that you have an option to evaluate different tracing platforms. I’m arguing that by adopting OpenTelemetry you can even run multiple tracing platforms <em>side by side</em>, examining them and making a conscious decision about what works best for your organization.</p><h3>OpenTelemetry + Jaeger = ❤️</h3><p>Before we move on any further, a quick reminder of the OpenTelemetry Collector architecture (for in-depth discussion see <a href="https://towardsdev.com/approaching-opentelemetry-f138b9f0a9e4?source=friends_link&amp;sk=55c40cab3e52ba1a25347a39b552666e">Approaching OpenTelemetry</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*Q-9iyAHhDCQ_BHCa.png" /><figcaption>OpenTelemetry Collector architecture</figcaption></figure><p>OpenTelemetry Collector is the main component that ensures <a href="https://opentelemetry.io/docs/reference/specification/glossary/#signals"><em>telemetry signals</em></a> (traces, metrics, logs) are <a href="https://opentelemetry.io/docs/collector/configuration/#receivers"><em>received</em></a>, <a href="https://opentelemetry.io/docs/collector/configuration/#processors"><em>processed</em></a>, and <a href="https://opentelemetry.io/docs/collector/configuration/#exporters"><em>exported</em></a>. As an OpenTelemetry user, you are responsible for Collector <a href="https://opentelemetry.io/docs/collector/configuration/#service">pipelines</a> — you can pick from a <a href="https://opentelemetry.io/registry/">registry</a> of available receivers, processors, and exporters to build the exact pipelines you need.</p><blockquote>OpenTelemetry defines a list of <a href="https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors">recommended processors</a> — worth taking a look.</blockquote><p>Jaeger architecture is also built around <em>Collector</em>. Similar names are not a coincidence here — Jaeger founder, <a href="https://www.shkuro.com/">Yuri Shkuro</a> is also an OpenTelemetry co-founder and an active member of the community. <em>Jaeger Collector</em> can receive trace spans in <a href="https://www.jaegertracing.io/docs/1.33/deployment/#collector">various formats</a> and handles efficient storage of the data, using one of the <a href="https://www.jaegertracing.io/docs/1.33/architecture/#collector">supported storage backends</a> — at the time of writing these are Elasticsearch, Cassandra, and Kafka.</p><p>Bringing it all together, when the OpenTelemetry pipeline is configured to export traces to the Jaeger platform, the trace flow looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*ImRfkIg8EkRsTUjnY_bm6g.png" /><figcaption>OpenTelemetry trace flow with Jaeger as a tracing backend.</figcaption></figure><p>At this point you may ask a valid question — why do we need <em>another</em> Collector? Recall that this is a big deal operationally — think about deploying, scaling, patching, monitoring, and troubleshooting. Wouldn’t it be easier if we export traces to Jaeger storage backends directly from OpenTelemetry Collector?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*wq-mqWgmGFQ2Zvpqho29ig.png" /><figcaption>OpenTelemetry trace flow (imaginary) with direct export to Jaeger backend trace storage.</figcaption></figure><blockquote>Jaeger was created <strong>before</strong> OpenTelemetry. This explains why we have Jaeger Collector but do not have (yet) OpenTelemetry Exporters that support Jaeger storage backends.</blockquote><p>In fact, this is exactly the direction Jaeger team is <a href="https://github.com/jaegertracing/jaeger/issues/3500">thinking about</a>. However, we are not there yet and <em>both</em> Collectors have to be present. It may seem like a setback, but let’s go one step further to understand how AWS Distro for OpenTelemetry fits in.</p><blockquote>The referenced <a href="https://github.com/jaegertracing/jaeger/issues/3500">GitHub issue</a> is a very interesting read that helps to better understand relationships between OpenTelemetry and Jaeger as well as feature overlap between them.</blockquote><h3>ADOT + Jaeger = ?</h3><p>As we discussed above, OpenTelemetry Collector can be configured to export the traces to Jaeger Collector via the configuration file using the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerexporter">Jaeger exporter.</a> Can the same be done with ADOT?</p><p>In theory, it should be the same thing — ADOT is just an <a href="https://opentelemetry.io/docs/concepts/distributions/">OpenTelemetry distribution</a>, tailored for the AWS environment. Right?..</p><p>Not quite. Remember that an OpenTelemetry distribution can both <em>add</em> and <em>remove</em> certain OpenTelemetry components and ADOT is not an exception. It <em>adds</em> <a href="https://github.com/aws-observability/aws-otel-collector#adot-collector-built-in-components">AWS-specific components</a> but it also <em>removes</em> some from the base distro, mainly to keep the distro compact and reduce the testing surface. What’s important for today’s post — <em>there is no Jaeger exporter available in AWS Distro for OpenTelemetry</em>.</p><blockquote>Despite the fact that there <strong>is</strong> <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/jaegerexporter">Jaeger exporter for OpenTelemetry</a> available, it is not included into ADOT (yet?) and, therefore, there is no built-in way to configure ADOT-based applications to export spans to Jaeger.</blockquote><p>It means we can’t directly build ADOT -&gt; Jaeger trace processing pipeline. It may be disappointing when you first encounter this, but I’m going to argue below that this is actually not a (very) bad thing.</p><p>There are multiple ways to address this obstacle and export OpenTelemetry traces from the ADOT-enabled application to Jaeger:</p><ul><li>OpenTelemetry Collector in front of Jaeger</li><li>Custom ADOT Collector packaged with Jaeger exporter</li></ul><p>This post focuses primarily on the first approach as it can be used with the latest OTEL Collector releases, including all processor/exporters from the <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib">opentelemetry-collector-contrib</a> repository. We’ll cover the process of building a custom ADOT Collector in future posts.</p><h3>Gateway OTEL Collector architecture</h3><p>Instead of trying to export spans to Jaeger directly from ADOT, let’s introduce another OpenTelemetry Collector in front of the Jaeger platform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*2jbY4r9Qr7p0dKvK7vP1RA.png" /><figcaption>Trace flow with ADOT Collector, Gateway Collector and Jaeger as a tracing backend</figcaption></figure><p>The architecture above may feel wrong — now we have <em>two</em> OpenTelemetry Collectors, seemingly just to work around the fact that ADOT does not include Jaeger exporter. It may be indeed the case if you have a single service — but what if you have tens or hundreds of them?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*wRhzovv7J76G-HDepBhtNQ.png" /><figcaption>Large-scale architecture with Gateway Collector</figcaption></figure><p>Suddenly, the architecture with OpenTelemetry Collector deployed as a <a href="https://opentelemetry.io/docs/collector/getting-started/#gateway">gateway</a> looks much more sensible. Gateway Collector is a way to standardize your telemetry processing pipelines, instead of replicating the same configuration over and over in each service. With this architecture in place, ADOT Collectors act as “dumb” <a href="https://opentelemetry.io/docs/collector/getting-started/#agent">agents</a>, forwarding telemetry signals to the gateway Collector that in turn perform the main processing.</p><p>Here is what you can do with this architecture in place:</p><ul><li>Implement advanced sampling policies across your distributed architecture, including <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor">tail-based sampling</a></li><li><a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanmetricsprocessor">Capture metrics</a> based on the spans collected across your services. This is especially valuable as you can capture metrics based on the spans that will be dropped by your sampling rules.</li><li>Centralize export of telemetry signals (along with secret management) to external backends — such as Jaeger or DataDog</li><li>Efficient scaling — ADOT Collector is running close to service and, therefore, is supposed to be very lightweight to ensure that application performance is not affected by tracing. In contrast, gateway OpenTelemetry Collector can be deployed and scaled independently, based on the total telemetry volume across <em>all</em> services</li><li>Gateway Collector is a way to enforce org-wide security/networking controls and decouple telemetry <em>producers</em> (usually, service teams) from telemetry <em>management</em> (platform team).</li></ul><p>The obvious disadvantage of this architecture — it adds another hop to the telemetry pipeline and makes it more complex, especially from the operational point of view.</p><blockquote>I would argue that the advantages you get greatly outweigh the disadvantages the more services you have in your distributed architecture.</blockquote><h3>Good news — Jaeger Quick Start</h3><p>I hear you sigh deeply and reluctantly matter:</p><blockquote>OK. That gateway thing makes sense. I would try this out and compare Jaeger with X-Ray. However, it feels like this requires a lot of work and I’m not ready to invest into this right now.</blockquote><p>In the <a href="https://medium.com/better-programming/introducing-jaeger-quick-start-deploying-on-aws-f0ee5d398e8a">previous post</a>, I introduced <a href="https://github.com/kolomiets/quickstart-jaeger">Jaeger Quick Start</a> project that addresses this use case. The Quick Start automates the initial Jaeger deployment and configuration on AWS with reasonable defaults, so you can “start quickly”. For the purposes of this blog post, Jaeger Quick Start provides an option to deploy not only Jaeger itself but also a pre-configured OpenTelemetry Collector in front of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*HI2y4rbOW1CkiRNiWiHHIw.png" /><figcaption>Jaeger Quick Start boundaries and integration with ADOT Collector</figcaption></figure><p>This actually means that once you have Jaeger Quick Start deployed in your environment, you can use it straight away — no code changes in your ADOT services are necessary (yay!). All that is needed is to adjust the ADOT Collector configuration to forward the traces to the gateway Collector in front of Jaeger.</p><h3>ADOT Collector configuration</h3><p>For the demonstration part of this post I’m going to use a simple event-driven architecture we covered in detail in <a href="https://betterprogramming.pub/trace-context-propagation-with-opentelemetry-b8816f2f065e?source=friends_link&amp;sk=dbed25d85d4fc29d770fe677349c9e66">the previous post</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*w1-p9jG1_nv0KPmB.png" /><figcaption>The distributed architecture we want to trace with OpenTelemetry</figcaption></figure><p>The complete source code of this application along with the necessary deployment instructions can be found in <a href="https://github.com/kolomiets/tracing-playground/tree/main/opentelemetry-with-jaeger">kolomiets/tracing-playground</a> repository on GitHub.</p><p>As we are using ADOT Collector in the demo application, we can use <a href="https://aws-otel.github.io/docs/getting-started/lambda">OPENTELEMETRY_COLLECTOR_CONFIG_FILE</a> environment variable to override the default Collector configuration and add a custom one with the trace processing pipelines adjusted (compare this with the <a href="http://v/">default config</a>—the changes are marginal):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ea92a33b943607936191f642b24c4ef/href">https://medium.com/media/6ea92a33b943607936191f642b24c4ef/href</a></iframe><p>With this configuration in place, we export the same traces to both AWS X-Ray <strong>and</strong> Jaeger backends. This is very useful to evaluate both platforms.</p><blockquote>We use <a href="https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables">environment variable expansion</a> for the OTLP endpoint in the configuration above. This is a neat technique to keep your Collector configuration files simple and static. Expansion also allows you to reconfigure the endpoint by updating JAEGER_OTLP_ENDPOINT variable — no need to change the configuration file and redeploy the whole service.</blockquote><h3>So, can I see the traces, please?</h3><p>Yes!</p><p>Now, with all the configuration bits out of the way, we can run our demo application (see <a href="https://github.com/kolomiets/tracing-playground/tree/main/opentelemetry-with-jaeger#running-the-test">the instructions in GitHub repo</a>) and finally see the traces in both AWS X-Ray and Jaeger. Let’s start with AWS X-Ray:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dKKVIu7RJW7wxWLvvR94tA.png" /><figcaption>AWS X-Ray trace</figcaption></figure><p>This is the familiar trace we’ve seen when we examined <a href="https://betterprogramming.pub/trace-context-propagation-with-opentelemetry-b8816f2f065e?source=friends_link&amp;sk=dbed25d85d4fc29d770fe677349c9e66">context propagation with OpenTelemetry</a>. Nothing particularly new here. Let’s switch on to Jaeger:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fE857EHxfsPySsE2vfvcHg.png" /><figcaption>Jaeger UI with recent traces</figcaption></figure><p>Jaeger shows a number of captured traces — note different colour codes for services involved. If we dive deeper and open a trace, we’ll get span details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4J4L_lEjt2wIf-95FMJlSg.png" /><figcaption>Jaeger — Trace details</figcaption></figure><p>Each span contains <a href="https://opentelemetry.io/docs/instrumentation/go/manual/#span-attributes">OpenTelemetry span attributes</a> — the main source of information about a span. For example, here are the attributes for Kinesis.PutRecord span:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UVDUwGeZ-v3AB7CAzlmfaA.png" /><figcaption>Jaeger — Span attributes</figcaption></figure><p>Jaeger has an analogue of AWS X-Ray Service Map feature available at <em>System Architecture</em> tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*kajsjSlqudlRuE7h7cTeHw.png" /><figcaption>Jaeger — System Architecture</figcaption></figure><blockquote>The numbers on the edges show the total number of traces captured along the edge</blockquote><p>You may notice that Jaeger’s system architecture is different from what we’ve seen in AWS X-Ray. Nodes for the lambda functions are present but we do not see the nodes for AWS services: Lambda, SQS, Kinesis, SNS. It looks like we’ve been able to capture all the spans explicitly created in the lambda functions, but there are no spans emitted by AWS services. What is going on here?</p><p>The answer lies in how AWS services are integrated with AWS X-Ray. As mentioned in <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-services.html">the documentation</a>, some services (such as AWS Lambda or Amazon API Gateway) add additional nodes to the service map. In other words, there is built-in integration between certain AWS services and AWS X-Ray and OpenTelemetry can’t help us with these additional spans. This is why we don’t see them in Jaeger (as Jaeger receives only the spans that OpenTelemetry Collector gets).</p><p>This is not the only difference between the traces captured by AWS X-Ray and Jaeger, but probably the most visible one. I’d like to pause the discussion here and defer further analysis to the next post — this is a big topic on its own.</p><h3>Conclusion</h3><p>Before we wrap up, let me summarize the main ideas we’ve covered in this post.</p><p>OpenTelemetry Collector in a <a href="https://opentelemetry.io/docs/collector/getting-started/#gateway">gateway mode</a> is a good pattern to consider, especially for systems with a large number of services. This is a perfect way to standardize and enrich your telemetry processing pipelines, reducing telemetry overhead for each individual service.</p><p>AWS Distro for OpenTelemetry can be configured to export the traces not only to AWS X-Ray (which is the default behaviour) but to other backends as well. This enables you to run multiple tracing backends side by side — invaluable for migrations, evaluations, and POCs.</p><p><a href="https://github.com/kolomiets/quickstart-jaeger">Jaeger Quick Start</a> simplifies provisioning of the Jaeger telemetry platform in your AWS environment, ready to be plugged into your OpenTelemetry pipelines.</p><p>Traces captured by ADOT and exported to Jaeger <strong>are</strong> different from the traces you observe in AWS X-Ray. Partially this is because of direct <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-services.html">integrations between certain AWS services and AWS X-Ray</a>. The analysis of these differences will be covered in further posts of this series.</p><p>And, finally, the main achievement — with all the groundwork covered and codified in <a href="https://github.com/kolomiets/quickstart-jaeger">Jaeger Quick Start</a> and <a href="https://github.com/kolomiets/tracing-playground/">demo applications</a>, I stop talking about deploying and configuring Jaeger :) In the next posts, we will talk about the traces themselves and the benefits that OpenTelemetry provides.</p><h3>Resources</h3><ul><li><a href="https://kolomiets.github.io/quickstart-jaeger/">Jager Quick Start Deployment guide</a></li><li><a href="https://github.com/kolomiets/quickstart-jaeger"><strong>kolomiets/quickstart-jaeger</strong></a><strong> — </strong>Jaeger Quick Start project</li><li><a href="https://github.com/kolomiets/tracing-playground/blob/main/opentelemetry-with-jaeger/README.md"><strong>kolomiets/tracing-playground</strong></a> — the demo application we discussed</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acf4df3a8e37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/using-aws-distro-for-opentelemetry-with-jaeger-acf4df3a8e37">Using AWS Distro for OpenTelemetry with Jaeger</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Jaeger Quick Start — Deploying on AWS]]></title>
            <link>https://medium.com/better-programming/introducing-jaeger-quick-start-deploying-on-aws-f0ee5d398e8a?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0ee5d398e8a</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[jaeger]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Tue, 19 Apr 2022 16:03:22 GMT</pubDate>
            <atom:updated>2022-04-26T19:24:16.436Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">Musings on Tracing</a></h4><h3>Introducing Jaeger Quick Start — Deploying on AWS</h3><h4>A look at the end-to-end distributed tracing platform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1lhiVr8bUKGc8lklrOKcEw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rubaitulazad?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Rubaitul Azad</a> on <a href="https://unsplash.com/@rubaitulazad?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>A problem with OpenTelemetry and X-Ray</h3><p><a href="https://betterprogramming.pub/trace-context-propagation-with-opentelemetry-b8816f2f065e?source=friends_link&amp;sk=dbed25d85d4fc29d770fe677349c9e66">In the previous post</a>, we looked at AWS Distro for OpenTelemetry (ADOT) and how to use it to produce traces in event-driven architectures. ADOT exports the traces to AWS X-Ray by default — unfortunately, there are some limitations:</p><ul><li>X-Ray does not support <a href="https://opentelemetry.io/docs/concepts/data-sources/#logs">OpenTelemetry events</a> — a very useful concept that in many cases may replace more traditional logs</li><li>X-Ray does not support <a href="https://opentelemetry.io/docs/reference/specification/overview/#links-between-spans">OpenTelemetry links</a> — we’ve seen previously that links are very helpful for expressing causal relationships between spans</li><li>X-Ray has <a href="https://aws-otel.github.io/docs/getting-started/python-sdk/trace-manual-instr#creating-custom-spans">limitations on span kinds </a>— only spans of kind Server are converted into X-Ray segments</li></ul><p>As is often the case in OpenTelemetry world, it is possible to change the default configuration and <a href="https://opentelemetry.io/docs/collector/configuration/#exporters">export</a> the traces to a <em>different</em> tracing backend. But which one to use?</p><h3>Jaeger</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6KVikFIVj_sjhX43VKR_Q.png" /></figure><p>There are many tracing backends you may want to consider for your tracing platform — both commercial and open source. In this post, I’d like to bring your attention to <a href="https://www.jaegertracing.io/">Jaeger</a> — a <a href="https://www.cncf.io/projects/jaeger/">graduated project</a> from <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a> (CNCF). This is a respectable stamp of approval, considering other CNCF graduated projects. Just to name a few: <a href="https://kubernetes.io/">Kubernetes</a>, <a href="https://helm.sh/">Helm</a>, <a href="https://prometheus.io/">Prometheus</a>, <a href="https://www.fluentd.org/">Fluentd</a>.</p><p>Jaeger is an open-source, end-to-end distributed tracing platform. In a true cloud-native fashion, Jaeger can be deployed in a number of ways, for example:</p><ul><li><a href="https://www.jaegertracing.io/docs/1.33/deployment/">Docker images</a></li><li><a href="https://github.com/jaegertracing/jaeger-operator#getting-started">Kubernetes operator</a></li><li><a href="https://github.com/jaegertracing/helm-charts">Helm charts</a></li><li><a href="https://www.jaegertracing.io/docs/next-release/windows/">Windows service</a></li></ul><p>Jaeger team created a detailed <a href="https://www.jaegertracing.io/docs/next-release/deployment/">deployment guide</a> that answers many questions about the roles of different Jaeger components (<a href="https://www.jaegertracing.io/docs/1.33/architecture/#agent">Agent</a>, <a href="https://www.jaegertracing.io/docs/1.33/architecture/#collector">Collector</a>, and <a href="https://www.jaegertracing.io/docs/1.33/architecture/#query">Query</a> services), integration points with other services (Elasticsearch, Cassandra), and scaling considerations. However, even with that level of technical documentation available, deploying a production-ready Jaeger platform is not a small feat.</p><p>For large-scale deployments, especially in the AWS context, we expect a lot from a <a href="https://aws.amazon.com/architecture/well-architected/">well-architected</a> solution — among other things:</p><ul><li>Resiliency</li><li>Elasticity</li><li>Observability</li><li>Native integration with other AWS services</li><li>Reproducible, fully automated IaC deployments</li></ul><p><a href="https://github.com/jaegertracing/jaeger-operator#getting-started">Jaeger Kubernetes operator</a> helps with some of these traits, but then again — you need a Kubernetes cluster to start with. What if you don’t have one? Spinning up (and maintaining!) a new cluster to host a tracing platform may be way too difficult and expensive (both cost- and experience-wise).</p><p>Here goes the main question for this post — what would a Jaeger “quick start” look like on AWS?</p><h3>AWS Quick Starts — 101</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*HImzOvJnfoN8HOR0" /></figure><p>I used “quick start” above for a reason — <a href="https://aws.amazon.com/quickstart/">AWS Quick Starts</a> are “<em>automated reference deployments … [that] help you deploy popular technologies on AWS according to AWS best practices</em>”. This is exactly what I am aiming for with Jaeger on AWS.</p><p>AWS even provides <a href="https://aws-quickstart.github.io/">Quick Start Contributor’s Guide</a> to help with the authoring of the new Quick Starts. Without diving too deep into the details, let me say that a <em>Quick Start is a set of CloudFormation templates</em>.</p><p>And I must admit that these templates have to be <strong>solid</strong>. Quoting the documentation, the <a href="https://aws-quickstart.github.io/reqs.html#min">minimum architectural requirements</a> for a Quick Start are:</p><ul><li>Multi-AZ architecture (<a href="https://aws-quickstart.github.io/best-practices.html#ha">details</a>)</li><li>Support for the majority of AWS Regions (<a href="https://aws-quickstart.github.io/design.html#region-support">details</a>)</li><li>New VPC and existing VPC deployment options (<a href="https://aws-quickstart.github.io/design.html#deployment-options">details</a>)</li><li>Product instances in private subnets (<a href="https://aws-quickstart.github.io/best-practices.html#security">details</a>)</li><li>NAT gateways for outbound internet access from private subnets (<a href="https://aws-quickstart.github.io/templates-examples.html">details</a>)</li><li>Marketplace AMIs whenever possible; no prebaked AMIs (<a href="https://aws-quickstart.github.io/use-amis.html">details</a>)</li><li>AMI mappings; no hardcoded AMIs (<a href="https://aws-quickstart.github.io/use-amis.html">details</a>)</li><li>User-friendly parameter labels and groups (<a href="https://aws-quickstart.github.io/naming-parms.html">details</a>)</li><li>CIDR block lockdown for external admin access (<a href="https://aws-quickstart.github.io/best-practices.html#security">details</a>)</li><li>Security groups with principle of least privilege (<a href="https://aws-quickstart.github.io/best-practices.html#security">details</a>)</li><li>No software bits with deployment (<a href="https://aws-quickstart.github.io/design.html#handling-downloads">details</a>)</li><li>No hardcoded passwords (<a href="https://aws-quickstart.github.io/best-practices.html#security">details</a>)</li><li>No sensitive data in EC2 instance user data or other clear text (<a href="https://aws-quickstart.github.io/best-practices.html#security">details</a>)</li><li>No use of 0.0.0.0/0 for open remote management access (<a href="https://aws-quickstart.github.io/best-practices.html#security">details</a>)</li><li>No resources created automatically outside stack (<a href="https://aws-quickstart.github.io/design.html#error-handling">details</a>)</li></ul><p>An impressive list indeed. If you use <em>any</em> Quick Start to deploy your infrastructure, you can be sure that templates are pretty good.</p><p>Another thing I especially like about the Quick Starts ecosystem is <a href="https://aws-quickstart.github.io/design.html#modularity">modularity</a>. If there is a Quick Start that deploys a part of the infrastructure you want to use in your architecture (such as <a href="https://github.com/aws-quickstart/quickstart-aws-vpc">VPC Quick Start</a> or <a href="https://github.com/aws-quickstart/quickstart-amazon-aurora-postgresql">Aurora PostgreSQL Quick Start</a>) — you can <em>reference</em> an existing implementation. It’s like referencing a library in Python or NodeJS, only for your CloudFormation templates. This feature alone is groundbreaking enough to change your approach to IaC with CloudFormation.</p><p>Let’s be honest — how many times have you created a template to provision VPC, subnets, route tables, internet gateways, NAT gateways, endpoints, …? Don’t tell me you haven’t dreamed about off-the-shelf implementation you could just reuse without reinventing this wheel again? Now you can.</p><p>A final thing about Quick Starts (promise). Quick Start architectures can be very complex, especially if you consider the modularity aspect we covered above. Developing and maintaining CloudFormation templates for a large Quick Start may be a serious task without a solid automated testing framework. AWS Quick Start team recognized this challenge and introduced <a href="https://github.com/aws-ia/taskcat">TaskCat</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*Ux-e0B3rWkQ1Pv-T.png" /></figure><p>TaskCat automates linting and deployments for CloudFormation templates across multiple AWS regions. It means that you can ensure that your templates, however complex they are, can be successfully deployed in the target regions (and deleted— this aspect is often neglected during CloudFormation development)</p><p>With TaskCat it is possible to define “tests”, providing different parameters to CloudFormation at deployment time. The tool also generates a pass/fail report that can be integrated into your CI/CD tool, so you can run the tests as part of your PR validation process.</p><blockquote>Even if you are not going to use AWS Quick Starts for your IaC, I do recommend incorporating <a href="https://github.com/aws-ia/taskcat">TaskCat</a> into your workflow to automate tests for your CloudFormation templates.</blockquote><h3>Jaeger Quick Start</h3><blockquote>NOTE: This is NOT an official <a href="https://aws.amazon.com/quickstart/">AWS Quick Start</a>. The project is built according to <a href="https://aws-quickstart.github.io/">Quick Start Contributor’s Guide</a> but it is NOT approved by AWS and Jaeger teams yet.</blockquote><p>Now, once we defined what an AWS Quick Start is, let me finally introduce <a href="https://github.com/kolomiets/quickstart-jaeger">Jaeger Quick Start</a>!</p><p>Jaeger Quick Start helps you to deploy a <a href="https://docs.aws.amazon.com/whitepapers/latest/real-time-communication-on-aws/high-availability-and-scalability-on-aws.html">highly available</a> Jaeger — open-source, end-to-end distributed tracing platform on the Amazon Web Services (AWS) Cloud.</p><p>You can use the AWS CloudFormation templates included with the Quick Start to deploy Jaeger in your AWS account in about 10–30 minutes.</p><p>This Quick Start is for users who want a repeatable, customizable reference deployment for Jaeger using <a href="https://aws.amazon.com/cloudformation/">AWS CloudFormation</a>. You can also use the provided AWS CloudFormation templates as a starting point for your own implementation.</p><p>Here is a high-level architecture diagram of the Quick Start:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*HvxrPI6X6l1QEBiRdhsTZw.png" /><figcaption>Jaeger serverless architecture (with Elasticsearch backend)</figcaption></figure><p>The main highlights:</p><ul><li>Instead of running Jaeger containers in the Kubernetes cluster (with associated operational complexity), the Quick Start uses <a href="https://aws.amazon.com/fargate/">AWS Fargate</a> — serverless compute engine</li><li>The Quick Start deploys all the necessary Jaeger components (<a href="https://www.jaegertracing.io/docs/1.33/architecture/#collector">Collector</a>, <a href="https://www.jaegertracing.io/docs/1.33/architecture/#query">Query</a>, <a href="https://github.com/jaegertracing/spark-dependencies">Dependency Spark Job</a>) and configures them appropriately</li><li>The Quick Start supports two different storage backends — <a href="https://aws.amazon.com/opensearch-service/">Amazon OpenSearch</a> cluster for large-scale production deployments and in-memory storage suitable for experiments and proof-of-concept workloads</li><li>The Quick Start deploys <a href="https://aws.amazon.com/elasticloadbalancing/network-load-balancer/">Network Load Balancer</a> in public VPC subnets to distribute the traffic and expose necessary Jaeger ports. The load balancer can be either internet-facing or internal, accessible only to VPC clients</li><li>The Quick Start provides options to assign DNS names for Jaeger instance. Both Public and Private <a href="https://aws.amazon.com/route53/">Amazon Route 53</a> hosted zones are supported.</li><li>The Quick Start integrates with <a href="https://aws.amazon.com/certificate-manager/">AWS Certificate Manager</a> to add TLS to the Network Load Balancer</li><li>The Quick Start supports integration with <a href="https://aws.amazon.com/prometheus/">Amazon Managed Service for Prometheus</a> to expose Jaeger metrics</li><li>The Quick Start adds an optional integration with <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html">CloudWatch Container Insights</a> to monitor metrics for the Jaeger containers running on Fargate</li></ul><p>A detailed explanation of the features, architectural decisions, and complete parameter reference can be found in <a href="https://kolomiets.github.io/quickstart-jaeger/">Jaeger Quick Start Deployment Guide</a>.</p><h3>What’s next?</h3><p><a href="https://github.com/kolomiets/quickstart-jaeger">Jaeger Quick Start</a> is a new project — for the initial version I concentrated on supporting two storage backends (OpenSearch and in-memory) and, therefore, slightly different architectures. I’ve put together <a href="https://github.com/users/kolomiets/projects/1/views/1">Jaeger Quick Start roadmap</a> and raised a number of <a href="https://github.com/kolomiets/quickstart-jaeger/issues">GitHub issues</a> to indicate where I want to go next. My focus for the next milestone would be performance assessment, fine-tuning of the default parameters, and documentation improvements. Eventually, I plan to add support for another popular Jaeger storage backend — Cassandra or, in AWS world of managed services, <a href="https://aws.amazon.com/keyspaces/">Amazon Keyspaces</a>.</p><p>If you have experience with deploying and running Jaeger on AWS or want to contribute in any way— please do reach out. Anecdotes and any pain points you encountered are especially welcomed!</p><p>Before I wrap the post up though, let me put Jaeger Quick Start in a wider context of <a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">this series</a>, namely how it could be useful for our OpenTelemetry journey with event-driven architectures?</p><p>With a simple way to deploy Jaeger into AWS environment, you can <em>complement</em> (or even <em>replace</em>) AWS X-Ray as your target telemetry backend for application traces. Indeed, you can use Jaeger and AWS X-Ray together, run them side by side— in fact, I would really recommend this approach to actually see both platforms in action and make an educated decision about which one suits your needs best.</p><p>And this is exactly what we are going to cover in the next post — running both platforms side by side, discussing ADOT configuration changes necessary to support Jaeger, comparing the resulting traces, and outlining the main differences between the platforms.</p><h3>Resources</h3><ul><li><strong>Jaeger Quick Start</strong> project we discussed above is in <a href="https://github.com/kolomiets/quickstart-jaeger"><strong>kolomiets/quickstart-jaeger</strong></a> GitHub repository</li><li><a href="https://kolomiets.github.io/quickstart-jaeger/">Jager Quick Start Deployment guide</a></li><li><a href="https://aws.amazon.com/quickstart/?solutions-all.sort-by=item.additionalFields.sortDate&amp;solutions-all.sort-order=desc&amp;awsf.filter-content-type=*all&amp;awsf.filter-tech-category=*all&amp;awsf.filter-industry=*all">Registry of the official AWS Quick Starts</a>, i.e. building blocks you can use</li><li><a href="https://aws-quickstart.github.io/">Quick Start Contributor’s Guide</a> — documentation on how to create a new quick start</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0ee5d398e8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/introducing-jaeger-quick-start-deploying-on-aws-f0ee5d398e8a">Introducing Jaeger Quick Start — Deploying on AWS</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trace Context Propagation With OpenTelemetry]]></title>
            <link>https://medium.com/better-programming/trace-context-propagation-with-opentelemetry-b8816f2f065e?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8816f2f065e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Sat, 02 Apr 2022 20:31:00 GMT</pubDate>
            <atom:updated>2022-04-18T14:33:52.652Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">Musings on Tracing</a></h4><h4>Practical notes on using AWS Distro for OpenTelemetry</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MGmsWLwL0H6dox0p" /><figcaption>Photo by <a href="https://unsplash.com/@polarmermaid?utm_source=medium&amp;utm_medium=referral">Anne Nygård</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><pre><strong>Table of Contents</strong></pre><pre><a href="#03f1">Summary</a><br><a href="#c370">Trace context propagation</a><br><a href="#9687">End to end trace</a><br><a href="#e3e7">Dangling AWS Lambda traces</a><br><a href="#762b">Conclusion</a><br><a href="#456e">Resources</a></pre><blockquote>Until this point I was talking about OpenTelemetry and ADOT in general, without discussing specific programming languages. The following post is based on <a href="https://aws-otel.github.io/docs/getting-started/lambda/lambda-python">AWS managed Lambda layer for ADOT Python</a>. I believe that conclusions are generally the same for other languages.</blockquote><h3>Summary</h3><p>The aim for this post is exactly as outlined in <a href="https://medium.com/@dmitrykolomiets/trace-context-propagation-in-event-driven-architectures-ba9181b1b57?source=friends_link&amp;sk=ad98d9cc52bebc6b92074d908b2eb20e">the previous one</a> — demonstrate how we can manually pass the trace context around when AWS services do not support this natively (for example, with AWS Kinesis streams). This time we will do this with OpenTelemetry, not AWS X-Ray (we are going to use X-Ray console to review the resulting traces though).</p><h3>Trace context propagation</h3><p>Now when you know about OpenTelemetry collectors, distributions, and can tell the difference between <a href="https://aws.amazon.com/otel">ADOT</a> and <a href="https://azdot.gov/">ADOT </a>— we can finally talk about practical bits.</p><blockquote>Review <a href="https://medium.com/@dmitrykolomiets/approaching-opentelemetry-f138b9f0a9e4?source=friends_link&amp;sk=55c40cab3e52ba1a25347a39b552666e">Approaching OpenTelemetry</a> if you need a refresher on these topics.</blockquote><p>To demonstrate the trace context propagation with OpenTelemetry I am going to use <a href="https://aws-otel.github.io/docs/getting-started/lambda/lambda-python">ADOT Python Lambda layer</a> and <a href="https://github.com/open-telemetry/opentelemetry-python">OpenTelemetry Python SDK</a> for the following reasons:</p><ul><li>I used Python in <a href="https://medium.com/@dmitrykolomiets/trace-context-propagation-in-event-driven-architectures-ba9181b1b57?source=friends_link&amp;sk=ad98d9cc52bebc6b92074d908b2eb20e">the previous post</a> to demonstrate trace context propagation with X-Ray SDK. I figured it would be beneficial to see how the same application looks like with OpenTelemery</li><li>Python is one of the languages (Java is another) that supports <a href="https://aws-otel.github.io/docs/getting-started/python-sdk/trace-auto-instr">automatic instrumentation</a>. It is important to highlight the differences between automatic and manual instrumentation to understand what is happening behind the scenes.</li></ul><p>We will use the same application architecture we examined before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*4Sm4d9LckQ9sYYvaVOhsjA.png" /><figcaption>The distributed architecture we want to trace with OpenTelemetry</figcaption></figure><h4>Producer Lambda</h4><p>We start with Producer lambda — the most straightforward one. Let me show you the code first:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cebd88c4d27961521805c05776a0d485/href">https://medium.com/media/cebd88c4d27961521805c05776a0d485/href</a></iframe><p>This suspiciously simple implementation not only captures the traces for the lambda function, but also <em>instruments </em>and captures traces from boto3 library, handles interactions with ADOT Collector, and exports the traces to AWS X-Ray. This is the level of service that ADOT Lambda layers provide when <a href="https://aws-otel.github.io/docs/getting-started/python-sdk/trace-auto-instr">automatic instrumentation</a> is enabled!</p><p>More specifically, you need to do the following to enable automatic instrumentation:</p><ul><li>Add <a href="https://aws-otel.github.io/docs/getting-started/lambda/lambda-python#add-the-arn-of-the-lambda-layer">ADOT Lambda layer</a> to your function — this installs OpenTelemetry SDK, ADOT Collector, AWS-specific extensions, etc.</li><li><a href="https://aws-otel.github.io/docs/getting-started/lambda/lambda-python#enable-auto-instrumentation-for-your-lambda-function">Enable auto instrumentation</a> by adding AWS_LAMBDA_EXEC_WRAPPER variable with /opt/otel-instrument value</li></ul><p>Even without understanding everything that happens behind the scenes, this looks pretty good already.</p><h4>Automated instrumentation — trace structure</h4><p>As a sanity check, we can invoke Producer function in the console (with any payload) and see the resulting trace in X-Ray console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yp2nuye-5SCt8adv6JaA2g.png" /></figure><p>Let’s review the structure of the captured X-Ray trace — there will be an important difference when we move on to the manual instrumentation later.</p><blockquote>X-Ray service uses the concepts of <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-segments">segment </a>and <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-subsegments">subsegment</a> to denote parent/child relationships between units of work that comprise the trace. OpenTelemetry uses a concept of a <a href="https://opentelemetry.io/docs/concepts/data-sources/#traces">span</a> for the same purpose — there may be parent and child spans.<br>Even though we use X-Ray console to view the resulting traces, I am going to use OpenTelemetry terms in this post.</blockquote><p>The first two spans are added to the trace by the AWS Lambda service. It is important to understand that these two spans are captured and exported to X-Ray even <em>before </em>our Producer lambda is invoked.</p><blockquote>AWS Lambda exports these two spans <strong>directly </strong>to X-Ray. At the time of writing there is no way to configure AWS Lambda and specify another telemetry backend such Jaeger, Splunk, etc.<br>An implication — if OpenTelemetry is configured with another telemetry backend, AWS Lambda-specific spans will <strong>not </strong>be exported there.</blockquote><p>The next producer-function span is the span captured by OpenTelemetry — this is the span that is created by the automatic instrumentation startup script (remember that /opt/otel-instrument script we set as AWS_LAMBDA_EXEC_WRAPPER environment variable?).</p><p>If we select producer-function span, we will see the child producing_messages span we manually created in our lambda function. There are two additional spans for AWS SQS and AWS Kinesis services as well — these are captured from boto3 library. When automatic instrumentation is used, the startup script instruments the most common Python libraries to automatically capture spans from them. Note that <em>instrumentors</em> should be <a href="https://aws-otel.github.io/docs/getting-started/python-sdk/trace-auto-instr">installed and packaged</a> with your lambda function in order for this to work. You can find <a href="https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation">the comprehensive list of supported libraries on GitHub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZMBwCx95jo7-_UuVBgKCgw.png" /><figcaption>Producer function trace</figcaption></figure><p>Hope you have a better idea of what is happening when automatic instrumentation is used. Let’s move on to the next lambda.</p><h4>Consumer SQS Lambda function</h4><p>This function needs to extract the trace context from the SQS message and ensure that new spans generated by the function “continue” the original trace for the message. Again, we are using automatic instrumentation for this lambda. Here is the implementation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4ffd5c31e084ccc0ae9b036caeaa3e7/href">https://medium.com/media/d4ffd5c31e084ccc0ae9b036caeaa3e7/href</a></iframe><p>As with the Producer lambda, we do not need to bother with OpenTelemetry configuration too much. We extract traceId and spanId fields from SQS message — this part is identical to our <a href="https://medium.com/@dmitrykolomiets/trace-context-propagation-in-event-driven-architectures-ba9181b1b57?source=friends_link&amp;sk=ad98d9cc52bebc6b92074d908b2eb20e">previous trace propagation example with X-Ray</a>. There are a few things worth mentioning though.</p><p><strong>Server span kind<br></strong>Another interesting detail is a special SERVER span kind we assign to the consuming_sqs span. This is necessary to be able to see this span as a separate node in X-Ray service map. According to <a href="https://aws-otel.github.io/docs/getting-started/python-sdk/trace-manual-instr#creating-custom-spans">the documentation</a>:</p><blockquote>Note that only spans of kind Server are converted into X-Ray segments, all other spans are converted into X-Ray subsegments.</blockquote><p>Keep this obscure detail in mind if you plan to export OpenTelemetry traces to AWS X-Ray. If you use another telemetry backend — other <a href="https://opentelemetry.io/docs/instrumentation/js/api/tracing/#span-kind">span kinds</a> can be useful.</p><p><strong>Span links</strong><br>Note that we add an optional links parameter when we create the consuming_sqs span.</p><p>The idea is simple — each lambda invocation may contribute to <em>multiple </em>traces and it may be useful to have a way to correlate them. In our case, we have a lambda function triggered by SQS service. Here are the traces involved when the function is triggered with a batch of N messages:</p><ul><li><strong>Lambda invocation trace</strong>. When SQS service triggers the lambda function, an implicit trace is created. This is the “default” trace —trace that you normally see in the AWS X-Ray console. There will be two spans that AWS Lambda service emits by default (we discussed them above when we examined automatic instrumentation trace structure). This trace may also include spans for the lambda cold start or anything that happens before we start processing SQS messages</li><li><strong>SQS message traces</strong> —<em> at most</em> N of them. Remember that each SQS message may belong to a different trace and we want to “continue” that original trace when we process the message, not create a new one</li></ul><p>Therefore, when we process SQS message (adding spans to the original trace of the message) we would like to keep the link to the Lambda invocation trace — it may be particularly useful for troubleshooting scenarios.</p><p>Links between spans/traces may be a powerful way to express causality. It will be difficult to demonstrate these links in AWS X-Ray console as X-Ray does not support links (yet?). In the next post, I will show how the same OpenTelemetry trace looks like in a different telemetry backend (Jaeger) and will demonstrate the usefulness of the links then.</p><h4>Consumer Kinesis Lambda function</h4><p>The last Kinesis consumer lambda is the most tricky one mainly due to <a href="https://github.com/aws-observability/aws-otel-lambda/issues/238">the issue</a> in <a href="https://aws-otel.github.io/docs/getting-started/lambda/lambda-python">AWS managed Lambda layer for ADOT Python</a> that forced me to abandon automatic instrumentation (delete AWS_LAMBDA_EXEC_WRAPPER environment variable). This is a good thing as I will demonstrate how to add OpenTelemetry support without ADOT magic. Beware, there are many things to absorb:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d274720ebe020df4a318621ac54eed0/href">https://medium.com/media/7d274720ebe020df4a318621ac54eed0/href</a></iframe><p>Let’s cover OpenTelemetry initialization step by step:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b4577712b3baeca76c2f57f9c82ba65/href">https://medium.com/media/3b4577712b3baeca76c2f57f9c82ba65/href</a></iframe><p>The first thing we need to do is to initialize the <em>Tracer Provider</em> — this is the object that handles the collection of the traces and export to the OpenTelemetry Collector. When we initialize the provider we specify an ID generator <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-api-sendingdata.html#xray-api-traceids">compatible with X-Ray</a> (not necessary if X-Ray support is not needed) and other attributes we would like to attach to the spans. For this example, we set an explicit service name and capture other AWS Lambda-specific attributes (function ARN, name, memory/CPU allocation, etc.)</p><blockquote>The Tracer Provider can be initialized only <strong>once </strong>— if you try to set another provider, the second call will be ignored.</blockquote><p>Let’s unwrap the next line:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7c0e2919fa410f9afd7dbb58e6c0680/href">https://medium.com/media/b7c0e2919fa410f9afd7dbb58e6c0680/href</a></iframe><ul><li>add_span_processor —registers a span <em>processor</em> with the Tracer Provider. Processor is a construct OpenTelemetry introduces to pre-process data before it is exported (e.g. modify attributes or sample) or helps to ensure that data makes it through a pipeline successfully (e.g. batch/retry). A good summary OpenTelemetry processors can be found in <a href="https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor">Collector GitHub repository</a>.</li><li>BatchSpanProcessor — the batch processor accepts spans and places them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data</li><li>OTLPSpanExporter — exports spans in <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md">OpenTelemetry protocol</a> (OTLP) format. By default, the exporter connects to the local ADOT Collector instance running alongside your lambda code. It is the responsibility of the ADOT Collector to forward the spans further — by default, the collector is configured to pass them on to AWS X-Ray service.</li></ul><p>To recap — Tracer Provider manages the collection and export of the traces to ADOT Collector using BatchSpanProcessor and OTLPSpanExporter. With automatic instrumentation enabled all these low-level details are hidden, but now we have complete control over the processing and exporting of the spans. I will use this approach in future posts to demonstrate how we can extend capabilities provided by ADOT lambda layers and add support for additional telemetry backends.</p><p>Moving on:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ec03675491c9ac79a9f6689ce692d82/href">https://medium.com/media/7ec03675491c9ac79a9f6689ce692d82/href</a></iframe><p>This line enables instrumentation for boto3 API. With manual instrumentation it is our responsibility to instrument the libraries we are interested in.</p><p>Finally (the pun intended), you may have noticed this block of code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa84cbc058133dc81f6fb6d0d764692f/href">https://medium.com/media/aa84cbc058133dc81f6fb6d0d764692f/href</a></iframe><p>This is necessary to ensure that all spans are properly exported to the ADOT collector even in case of unhandled exception. This is particularly important when BatchSpanProcessor is used — remember that it batches multiple spans together before it forwards them to the collector. Without force_flush call, some of the spans may be lost (and, of course, these will be the most important ones)</p><p>This is it! Now you should have a decent mental model of OpenTelemetry initialization and basic configuration and understand the differences between automatic and manual instrumentation modes provided by ADOT lambda layer.</p><h3>End to end trace</h3><p>Before we wrap up we should run the Producer lambda function and check the resulting end-to-end trace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*99bEI2gUiov2QxsxqTmkXA.png" /></figure><p>Unlike my <a href="https://medium.com/@dmitrykolomiets/trace-context-propagation-in-event-driven-architectures-ba9181b1b57?source=friends_link&amp;sk=ad98d9cc52bebc6b92074d908b2eb20e">previous post</a>, I am not aware of any “hacks” or unsupported APIs I used in this example of trace propagation. This is a proper implementation according to OpenTelemetry specification. Even better — we can get end-to-end X-Ray traces with OpenTelemetry.</p><h3>Dangling AWS Lambda traces</h3><p>We briefly touched upon spans exported by AWS Lambda service above. These spans are clearly visible when we look at the complete service map in X-Ray:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4Jwm7UaOYxhF4lukCL3TQ.png" /></figure><p>Consumer lambdas emit OpenTelemetry spans that “continue” our main end-to-end trace. At the same time, we still have traces emitted by AWS Lambda service for these functions. These traces are detached from the main end-to-end trace or “dangling”. They are visible in X-Ray console because of the direct integration between AWS Lambda and AWS X-Ray services. Keep in mind that if you are going to use a different telemetry backend with OpenTelemetry, these AWS Lambda spans will <em>not</em> be exported.</p><h3>Conclusion</h3><p>We have covered a lot of ground in this post by showing how to integrate OpenTelemetry to event-driven AWS architectures and get complete end-to-end traces. We are still using AWS X-Ray as our main telemetry backend but we do not have to — with OpenTelemetry we have a way to export traces to another backend or event to multiple backends at the same time. This is what we are going to do in the next post — I will show you how to integrate with <a href="https://www.jaegertracing.io/">Jaeger </a>— a popular open-source distributed tracing backend.</p><h3>Resources</h3><ul><li>You can find the complete project we discussed above in <a href="https://github.com/kolomiets/tracing-playground/tree/main/opentelemetry-context-propagation">kolomiets/tracing-playground</a> GitHub repository</li><li>To get a feeling of the scale of OpenTelemetry ecosystem, take a look at <a href="https://opentelemetry.io/registry/">OpenTelemetry Registry</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8816f2f065e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/trace-context-propagation-with-opentelemetry-b8816f2f065e">Trace Context Propagation With OpenTelemetry</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Approaching OpenTelemetry]]></title>
            <link>https://towardsdev.com/approaching-opentelemetry-f138b9f0a9e4?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f138b9f0a9e4</guid>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[tracing]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Fri, 25 Mar 2022 10:17:00 GMT</pubDate>
            <atom:updated>2022-04-18T14:34:32.816Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">Musings on Tracing</a></h4><h4>Making sense of the “new” telemetry kid on the block</h4><blockquote>This is NOT a comprehensive guide to OpenTelemetry on AWS. There are much <a href="https://opentelemetry.io/docs/">better</a> <a href="https://aws.amazon.com/otel/">resources</a> to start with. Still, considering this is a huge topic, my aim is to get you started and give you a model you can use to reason about the modern telemetry architectures.</blockquote><h3>Summary</h3><p>I started working on this post, trying to demonstrate how to propagate a trace context in event-driven architectures with OpenTelemetry (similar to what I <a href="https://medium.com/@dmitrykolomiets/trace-context-propagation-in-event-driven-architectures-ba9181b1b57?source=friends_link&amp;sk=ad98d9cc52bebc6b92074d908b2eb20e">showed before with X-Ray</a>).</p><p>However, I quickly realized that understanding the underlying OpenTelemetry concepts is even more important than actual implementation. OpenTelemetry is a big and complex project and having a solid mental model would help to find a way through various repositories on GitHub, vendor-specific details and architectural patterns.</p><p>Therefore, I will start with a high-level overview of OpenTelemetry, touch on AWS Distro for OpenTelemetry and summarize why I think OpenTelemetry is such a big deal.</p><p>The actual code is coming in the next post!</p><h3>OpenTelemetry</h3><p>The best way to explain a new concept is to demonstrate how it is similar to something known and well understood. Let us start with an example of X-Ray traces.</p><p>Applications rarely communicate with AWS X-Ray service directly — instead, they rely on <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html">X-Ray Daemon</a>. The daemon receives raw traces from the application(s) and relays them to AWS X-Ray service (batching and compressing them for efficiency, retrying any transient errors from AWS APIs, etc). Applications don’t usually communicate with X-Ray Daemon directly either — there are <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-api.html">AWS X-Ray SDKs</a> for different languages that hide this complexity. Applications are concerned with higher-level concepts: defining segments/sugsegments, annotations, metadata, error details, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*Onab94rvVAzQfzKk7gLhdA.png" /><figcaption>High-level X-Ray (Lambda) architecture</figcaption></figure><p>OpenTelemetry has the same principal architecture (we are talking <em>very</em> high level here) — applications rely on OpenTelemetry SDK to pass the traces to <a href="https://opentelemetry.io/docs/collector/">OpenTelemetry Collector</a> which in turn can be integrated with a variety of backends, including AWS X-Ray.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*3KMnQJTKZjXQTzLdHdv5MA.png" /><figcaption>High-level OpenTelemetry (Lambda) architecture</figcaption></figure><p>An important distinction here — the AWS X-Ray daemon handles only traces whereas the OpenTelemetry collector can handle <strong>all</strong> <em>telemetry signals</em>: traces metrics, and logs.</p><p>Continuing the X-Ray analogy — you can deploy OpenTelemetry Collector as an <a href="https://opentelemetry.io/docs/collector/getting-started/#agent"><em>agent</em></a>, alongside each service (as in our Lambda example above — each Lambda function runs an instance of the collector) or as a <a href="https://opentelemetry.io/docs/collector/getting-started/#gateway"><em>gateway</em></a>, sharing it across the services (for example, in <a href="https://aws-otel.github.io/docs/setup/ecs">ECS</a> or <a href="https://aws-otel.github.io/docs/setup/eks">EKS</a> cluster). An interesting twist — and a very common scenario — agent and gateway deployment options can be used together. This may be particularly useful to enforce networking/security policies and/or centralize access to API authentication tokens required to relay telemetry information to external backends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*f1aSmzO0Hq0Yij30JsjJzA.png" /><figcaption>OpenTelemetry — agent and gateway deployment options</figcaption></figure><p>Still, nothing too exciting so far. You may even say that this is boring — another “generic” and “pluggable” SDK that promises a unified set of APIs with various backends support. Familiar promise, we’ve been there a number of times. But let’s continue, it will be (eventually) fun.</p><h3>AWS Distro for OpenTelemetry</h3><p>As it became clear from the previous section, OpenTelemetry Collector is a core component that ensures traces, metrics, and logs flow efficiently through the system. OpenTelemetry community even provides <a href="https://github.com/open-telemetry/opentelemetry-collector-releases">pre-built cross-platform collector distributions</a> with wide support of input (<em>receivers</em> in OpenTelemetry terms) and output integrations (<em>exporters</em>). Technically, all you need to do is to grab this binary and start using it.</p><p>The thing is — you probably don’t want to interact with Collector directly from your application. Remember our X-Ray analogy — you do not interact with X-Ray daemon, you use X-Ray SDK instead. And indeed, there are OpenTelemetry SDKs for different languages you can use. Then you need to ensure the collector is deployed properly — if, for example, you want to use OpenTelemetry across EC2 instances, Lambdas, EKS clusters, ECS services — you would need different Collector deployment packages. You would also need to install additional packages if you want to support a specific telemetry backend that is not included in the pre-built collector distribution. Ah, and it is very important to ensure that Collector starts and stops correctly — otherwise, if your application crashes, for example, you can lose some traces — and crash traces are often the most important ones).</p><p>Suddenly, the idea of running the Collector doesn’t look <em>that</em> appealing.</p><p>The good thing is that the community recognizes this problem and addresses it by providing so-called <a href="https://opentelemetry.io/docs/concepts/distributions/">OpenTelemetry Distributions</a>. Essentially, a <em>distribution</em> contains everything you need to run a particular version of the Collector in a particular environment (Python, Java, .NET, etc.) with particular integrations enabled. This is exactly what <a href="https://aws.amazon.com/otel">AWS Distro for OpenTelemetry</a> (ADOT) offers for applications running in the AWS environment.</p><blockquote>There are other <a href="https://opentelemetry.io/vendors/">vendor-specific OpenTelemetry distributions</a>. AWS is just one of them.</blockquote><p>ADOT includes a <a href="https://github.com/aws-observability/aws-otel-collector">custom build of OpenTelemetry Collector</a>. The main differences from the upstream Collector:</p><ul><li>ADOT Collector supports a <a href="https://github.com/aws-observability/aws-otel-collector#adot-collector-built-in-components">subset of features</a> that upstream OpenTelemetry Collector provides (for example, Jaeger exporter is not supported). This makes ADOT Collector package smaller and lighter on resources (important for on-demand environments, such as AWS Lambda)</li><li>ADOT Collector adds <a href="https://github.com/aws-observability/aws-otel-collector#adot-collector-aws-components">AWS-specific components</a> that are not included in OpenTelemetry Collector distributions (AWS X-Ray and Amazon CloudWatch exporters, AWS Container Insights receiver, etc.)</li></ul><h3>ADOT Lambda layers</h3><p>I mostly use Lambda functions for examples in this series as they pose some interesting challenges for trace context propagation in event-driven and streaming architectures. ADOT recognizes this and even provides convenient <a href="https://aws-otel.github.io/docs/getting-started/lambda">AWS Lambda Layers</a> that simplify the integration burden. On a very high level of my architecture diagrams a lambda function with ADOT layer looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*EkL7lg-lJuEPEa0qM7HtGQ.png" /><figcaption>Lambda function with ADOT Layer</figcaption></figure><p>By integrating ADOT Lambda layer into your functions, you address most of the difficulties I mentioned before with self-hosting the Collector. There are other benefits, such as automatic instrumentation of the common libraries your lambda may use (at the time of writing — only for Java and Python).</p><p>In some cases, you can get detailed traces without <em>any</em> changes in the application code. I would argue that this magic rarely works in practice outside of crafted or trivial examples (and will cover the limitations in the next post). Still, it is a great first iteration that gets you started. As the <a href="https://martinfowler.com/bliki/WaterfallProcess.html">classic said</a>:</p><blockquote>You should use iterative development only on projects that you want to succeed.<br><strong>Martin Fowler</strong></blockquote><h3>OpenTelemetry promises</h3><p>To conclude the post I put together a number of facts that make OpenTelemetry revolutionary and exciting, making it much bigger than another telemetry framework.</p><h4>OpenTelemetry is CNCF project</h4><p>OpenTelemetry is backed by <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a>. It is a great company to be in, just to name a few projects: <a href="https://www.cncf.io/projects/kubernetes/">kubernetes</a>, <a href="https://www.cncf.io/projects/prometheus/">Prometheus</a>, <a href="https://www.cncf.io/projects/jaeger/">Jaeger</a>. OpenTelemetry is likely a <em>very</em> big deal in the telemetry and tracing domain. Definitely worth your attention and a closer look.</p><h4>OpenTelemetry is not only about traces</h4><p>My analogy with X-Ray above may give you an impression that OpenTelemetry is just another tracing framework. It is true that OpenTelemetry supports traces, however, one of the main goals of the project is to <a href="https://thenewstack.io/modern-observability-is-a-single-braid-of-data/">change our practice of observability</a>. We often think that <strong>logs</strong>, <strong>metrics</strong> and <strong>traces</strong> are independent and separate views so we use different tools and frameworks to collect and manage these observability “pillars”. OpenTelemetry promotes an idea of “a single braid of structured data”, helping you to correlate traces, metrics and logs. We are so accustomed to this “classic” separation between logs, metrics, and traces that it is quite a mental shift. I will surely write more about this OpenTelemetry aspect in future posts.</p><h4>OpenTelemetry spurs innovative thinking</h4><p>First of all, OpenTelemetry is a language-agnostic <a href="https://github.com/open-telemetry/opentelemetry-specification"><em>specification</em></a>. The necessity to define concepts that are meaningful and useful for different programming languages and architectures requires huge efforts and insights from the community. At the time of writing (March 2022), OpenTelemetry is a very much alive specification. New ideas are constantly submitted and discussions in the specification repository on GitHub are full of treasures. If you ever thought that tracing is a difficult beast — you greatly underestimated the problem. See, for example:</p><ul><li><a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md">Messaging Systems</a></li><li><a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/cloudevents.md">Cloud Events</a></li><li><a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/faas.md">Semantic conventions for FaaS spans</a></li></ul><p>Observability is tough for modern architectures and having OpenTelemetry as a focal point of wisdom is very important for practitioners and researchers alike.</p><h4>OpenTelemetry encourages structured logs</h4><p>I have been a big fan of the structured logging approach for years. If I can make just a single change in the system, I would always pick structured logs over anything else. The ability to mine logs is crucial for efficient troubleshooting, especially if we are talking about the production environment.</p><p>An example I often use to make the point —<a href="https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries"> Elasticsearch slowlogs</a>. Here is a standard slowlogs log entry (yes, it is a <strong>single</strong> line):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/034e2cc231c1045a96188aedf0c3de59/href">https://medium.com/media/034e2cc231c1045a96188aedf0c3de59/href</a></iframe><p>And here is a structured version of the same log entry:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d00b7c3b5e4d12f83ce202da6f43f55/href">https://medium.com/media/0d00b7c3b5e4d12f83ce202da6f43f55/href</a></iframe><p>Now consider — what format would be easier to use to answer questions like:</p><ul><li>find queries for a specific index that take more than 10s and return more than 1k documents</li><li>find slow queries that use aggregations</li><li>find slow queries that ask for a date range that is more than a month</li></ul><p>Any modern log crunching tool (Amazon CloudWatch, DataDog, Splunk, NewRelic, you name it) supports JSON log entries and will give you the answers in no time. Good luck with the original unstructured log entry format.</p><p>OpenTelemetry is designed to promote this “structural” thinking about traces, metrics, and logs. It is finally <em>easier</em> to emit structured telemetry and get benefits from it. More than that — with OpenTelemetry you get structured telemetry <em>by default.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f138b9f0a9e4" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/approaching-opentelemetry-f138b9f0a9e4">Approaching OpenTelemetry</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trace context propagation in event-driven architectures]]></title>
            <link>https://towardsdev.com/trace-context-propagation-in-event-driven-architectures-ba9181b1b57?source=rss-3e42fa96c5e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba9181b1b57</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[tracing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-x-ray]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Dmitry Kolomiets]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 21:10:24 GMT</pubDate>
            <atom:updated>2022-04-18T14:34:59.261Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/@dmitrykolomiets/list/musings-on-tracing-3d7bd0d2093c">Musings on Tracing</a></h4><h3>Trace context propagation in event-driven architectures with AWS X-Ray</h3><blockquote>The tracing approach described in this post relies on implementation details of <a href="https://github.com/aws/aws-xray-sdk-python">AWS X-Ray SDK for Python</a>. The main purpose of the post is to be thought-provoking and spur discussions in the community about tracing in async/event-driven architectures.</blockquote><p>Tracing in distributed systems is often illustrated with RPC-like communications: HTTP requests, <a href="https://grpc.io/">gRPC</a> calls, etc. These communication patterns are <em>synchronous — </em>the client is blocked until a response from the server is received.</p><p>For the purposes of this post let’s introduce the term <em>synchronous architectures — </em>architectures that rely on synchronous communication patterns. This term covers a wide range of systems — from a simple client-server to a <a href="https://en.wikipedia.org/wiki/Service_mesh">service mesh</a> with hundreds of interacting microservices.</p><p><em>Trace context</em> is another important concept that we need to define. In its simplest form, trace context consists of two identifiers (see, for example, <a href="https://www.w3.org/TR/trace-context/">W3C definition of trace context</a>):</p><ul><li><strong>trace-id </strong>— unique ID of the whole trace</li><li><strong>parent-id </strong>— ID of the parent/previous</li></ul><p><em>Trace context propagation</em>, i.e. passing these two IDs from one service to another, can be achieved in different ways. For synchronous architectures request headers are often used (see, for example, <a href="https://w3c.github.io/trace-context/">W3C</a> or <a href="https://github.com/openzipkin/b3-propagation">Zipkin</a> specifications).</p><p>Due to the simplicity of trace context propagation in synchronous architectures, adding tracing is relatively easy and often can be achieved without instrumenting or modifying the source code of the application (see, for example, <a href="https://opentelemetry.io/docs/concepts/instrumenting/#automatic-instrumentation">automatic instrumentation</a> in <a href="https://opentelemetry.io/">OpenTelemetry</a>).</p><p>In this post, however, I want to discuss tracing in <em>asynchronous architectures</em> and demonstrate the problems that modern tracing tools have.</p><h3>Asynchronous architectures — the problem</h3><p>Let’s continue with the definitions. I will call an architecture <em>asynchronous</em> if it relies on communication patterns different from the traditional request/response paradigm. Let me give you some examples:</p><ul><li><a href="https://en.wikipedia.org/wiki/Event-driven_architecture">Event-driven architectures (EDA)</a>. In the AWS world think about services like <a href="https://aws.amazon.com/eventbridge/">Amazon EventBridge</a> or <a href="https://aws.amazon.com/sns/?whats-new-cards.sort-by=item.additionalFields.postDateTime&amp;whats-new-cards.sort-order=desc">Amazon SNS</a></li><li>Streaming architectures. Think <a href="https://kafka.apache.org/">Apache Kafka</a> or <a href="https://aws.amazon.com/kinesis/data-streams/">Amazon Kinesis Data Streams</a></li><li>Message Brokers. Examples may be <a href="https://www.rabbitmq.com/">RabbitMQ</a> or <a href="https://aws.amazon.com/amazon-mq/?amazon-mq.sort-by=item.additionalFields.postDateTime&amp;amazon-mq.sort-order=desc">Amazon MQ</a></li><li>There are other ways to implement asynchronous processing. Consider <a href="https://aws.amazon.com/sqs/">Amazon SQS</a> for example.</li></ul><p>As you can see there are plenty of architectures that do not rely on request/response communication patterns. In these systems trace context propagation may be a challenge. It either requires a custom ad-hoc approach or is not supported at all.</p><p>Let me illustrate the problem with a simple asynchronous architecture shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*oKMTMxtb1nkPIVefzePzGw.png" /><figcaption>Simple event-driven system architecture</figcaption></figure><p>There are multiple components on this diagram:</p><ul><li><em>Producer lambda</em> produces messages for <em>SQS queue</em> and <em>Kinesis stream</em></li><li>SQS queue triggers <em>Consumer SQS lambda</em></li><li>Kinesis stream triggers <em>Consumer Kinesis lambda</em></li><li>Both consumer lambdas publish a message in <em>SNS topic</em> when processing is complete</li></ul><p>If we try to collect traces for this architecture using <a href="https://aws.amazon.com/xray/">AWS X-Ray</a>, we will get <em>disconnected</em> traces, as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*u3HMy97gj2ICKhV6WeuG3w.png" /><figcaption>“Disconnected” X-Ray service map</figcaption></figure><p>You may have guessed what is the reason for this. Trace context propagation is challenging in asynchronous systems. There is no easy way to pass trace context along with SQS/Kinesis message and restore it on the other side when a message is picked up by consumer lambda.</p><p>Disconnected traces is a major problem for modern architectures. It makes it very difficult to build an accurate map of interacting microservices if there is no way to pass trace context around. It makes troubleshooting and reasoning about the system difficult. Even in the trivial example above generated traces are confusing. The situation is significantly worse in real systems with tens and hundreds of services emitting millions of messages/events.</p><h3>Explicit trace context propagation</h3><p>At the time of writing (March 2022) AWS X-Ray provides <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-services.html">support for trace propagation for some AWS services</a>. However, there are limitations:</p><ul><li>Amazon SQS <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html">does not support trace propagation for Lambda functions</a> (a common scenario)</li><li>Many asynchronous communication patterns are not supported (Amazon Kinesis Data Streams, Amazon MQ, 3rd party solutions, etc.)</li></ul><p>Not everything is lost, though. It turned out that with small tweaks it is possible to pass trace context along with the message and restore it later. The following code shows a simple Producer lambda implementation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f744246a74cacfd40ed1d1ffb642fbc5/href">https://medium.com/media/f744246a74cacfd40ed1d1ffb642fbc5/href</a></iframe><p>An important thing to note from the code above: we include <strong>traceId</strong> and <strong>segmentId</strong> properties (X-Ray trace context) in the message. We need these values to continue the trace when we process this message later.</p><p>Passing trace context along with the message is easy — restoring it is more involved. Take a look at the code below, especially at the<em> enable_lambda_xray_segments</em> decorator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da1852a49dc1fce892365ce0f14882f8/href">https://medium.com/media/da1852a49dc1fce892365ce0f14882f8/href</a></iframe><p>This is where we do something that is not entirely supported by X-Ray SDK. According to <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html">AWS documentation</a>:</p><blockquote><strong><em>Note<br></em></strong><em>It is not currently possible to connect a trace passed into an SQS queue to the Lambda consuming it on the other end. This is because although the trace header is propagated in the SQS message, </em><strong><em>you cannot set it in the Lambda because segments in Lambda are immutable</em></strong><em>.</em></blockquote><p>By replacing the current Lambda-specific tracing context with the “normal” X-Ray context, we make it possible to record new segments and link them to the trace context passed along with the message (see <em>recorder.in_segment</em> function call above).</p><p>With these tweaks in place, the traces X-Ray produces are much closer to the architecture of the system we defined above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUmsI4-Fwl3AtDpobV2uUA.png" /><figcaption>Complete “connected” service map that spans multiple lambda functions</figcaption></figure><p>What is better, the resulting trace contains all the details about segments and subsegments generated across multiple lambda functions. You can see the complete path of the message through the system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybM6XfkPg1si4FmzCNGREw.png" /><figcaption>Trace that captures segments across multiple Lambda functions</figcaption></figure><h3>Closing thoughts and further steps</h3><p>We’ve covered explicit trace context propagation using AWS X-Ray service. The same problems exist for other tracing systems (OpenTelemetry) in the context of asynchronous architectures.</p><p>The approach defined above can be used to produce “connected” X-Ray traces for services that are not natively supported (SQS with Lambda, Kinesis Data Streams, DynamoDB Streams, Amazon MQ, etc.).</p><p>Still, there are many open questions:</p><ul><li>What if a service <em>generates</em> N new messages for each message it processes? How the complete trace should look like?</li><li>What if a service receives a <em>batch</em> of messages (each belonging to a different trace) for processing (say, N messages from SQS queue)?</li><li>What if we have <em>cycles</em> in our event-driven system? How should we handle them in the context of traces?</li><li>What are cost/performance implications for a complete tracing solution for a large-scale event-driven system with [m|b]illions of events?</li></ul><p>As I said at the beginning, I’d like this introductory post to be thought-provoking. Tracing is an exciting area and lots of research is happening as you read this. I plan to continue the series with further examples of how some of the questions above can be answered in the context of AWS X-Ray and/or OpenTelemetry frameworks.</p><h3>Resources</h3><p>The complete example discussed in this post can be found in <a href="https://github.com/kolomiets/tracing-playground/tree/main/xray-context-propagation"><strong>kolomiets/tracing-playground</strong> repository</a>. It relies on <a href="https://aws.amazon.com/cdk/">AWS Cloud Development Kit</a> for the deployment — you will find the details in the README file.</p><p><a href="https://opentelemetry.io/">OpenTelemetry</a> is a project that generates lots of ideas in the domain of tracing. <a href="https://github.com/open-telemetry/opentelemetry-specification">Specification GitHub repository</a> contains lots of materials and discussions for anyone interested in the topic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba9181b1b57" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/trace-context-propagation-in-event-driven-architectures-ba9181b1b57">Trace context propagation in event-driven architectures</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>