<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yuri Grinshteyn on Medium]]></title>
        <description><![CDATA[Stories by Yuri Grinshteyn on Medium]]></description>
        <link>https://medium.com/@thegrinch?source=rss-84e227c7daf2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Kwoif53T33mPC9SolZO-RA.png</url>
            <title>Stories by Yuri Grinshteyn on Medium</title>
            <link>https://medium.com/@thegrinch?source=rss-84e227c7daf2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:56:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thegrinch/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building monitoring dashboards for fun and profit]]></title>
            <link>https://medium.com/google-cloud/building-monitoring-dashboards-for-fun-and-profit-2e04347e205e?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e04347e205e</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[dashboard]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 17:47:08 GMT</pubDate>
            <atom:updated>2020-11-18T17:47:08.228Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last <a href="https://medium.com/google-cloud/setting-up-cloud-operations-for-gke-a21b49979693">two</a> <a href="https://medium.com/google-cloud/troubleshooting-services-on-gke-872470e60d51">posts</a> , I explored the new “out of the box” GKE monitoring dashboard and using it to both set up alerting against an important resource and drilling down from an alert to figure out a problem. That dashboard is certainly great if you care about the entirety of your GKE fleet — but what if your primary charter is to maintain the reliability of a specific service? Well, for that, you’d need to create a service-specific dashboard. I’ve <a href="https://medium.com/google-cloud/automating-application-dashboard-creation-for-services-on-gke-istio-a55a5a79aa15?source=your_stories_page-------------------------------------">covered</a> creating dashboards before, and you can see me talk about automating their creation using Terraform in this <a href="https://www.youtube.com/watch?v=Rqu-2k9m814&amp;list=PLIivdWyY5sqLuKKx4pcdEAkJY1HevjVVm&amp;index=23&amp;t=1s">episode</a> of Stack Doctor. But now, there’s a <a href="https://cloud.google.com/monitoring/charts/dashboards#select-editor">new</a> Dashboard Editor — and I thought this would be a great opportunity to revisit this topic and use the new Editor to create an example dashboard capturing some key signals.</p><h3>What’s in a dashboard?</h3><p>As ever, the first question is — what should go on a dashboard? I covered this pretty well in both my <a href="https://medium.com/google-cloud/automating-application-dashboard-creation-for-services-on-gke-istio-a55a5a79aa15?source=your_stories_page-------------------------------------">post</a> and <a href="https://www.youtube.com/watch?v=Rqu-2k9m814&amp;list=PLIivdWyY5sqLuKKx4pcdEAkJY1HevjVVm&amp;index=23&amp;t=1s">video</a> on automating dashboard creation, but let’s review quickly. My recommendation here is usually based on the Monitoring Distributed Systems <a href="https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/">chapter</a> in the SRE <a href="https://landing.google.com/sre/sre-book/toc/">book</a>. If you have nothing else, start with the “golden signals” — traffic, errors, latency, and saturation.</p><p>From there, it’s just a matter of figuring out what is the best indicator for each of these. For example, if you have a way to count requests to your service, request count is a great way to measure traffic. If not, you can look at something like network usage for your workload. If you have a way to count responses based on status code, that’s a good way to count errors. If not — maybe counting log entries with a severity of ERROR or higher is good enough. You can even measure something like container restarts to track application crashes. If you’re tracking response latency using something like <a href="https://medium.com/google-cloud/custom-metrics-in-node-js-with-opentelemetry-and-prometheus-c10c8c0204d3--------------------">OpenTelemetry</a> — fantastic! If not — no worries — maybe you’re writing out transaction times in logs, and you can create a log-based metric to track it. And finally — you can always start with a basic infrastructure metric like CPU or memory use to track saturation.</p><p>The important thing is that you have a good at-a-glance representation of the health of your service.</p><h3>How do I dashboard?</h3><p>As I’ve previously <a href="https://www.youtube.com/watch?v=Rqu-2k9m814&amp;list=PLIivdWyY5sqLuKKx4pcdEAkJY1HevjVVm&amp;index=23&amp;t=1s">discussed</a>, you have 3 options when creating a new dashboard — using the UI, the API, or Terraform. This time, I wanted to focus on using the UI specifically because there’s a new Dashboard Editor!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AIVw1meykibnyPeA" /></figure><p>Following up on my post about troubleshooting, I figured I’d create a rather simple dashboard showing the basics of the health of my GKE workload. Let’s dive right in.</p><h3>New stuff!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UA-2v-JOMkm-pwkV" /></figure><p>It is immediately apparent that the new editor looks quite different. There’s a list of widgets on the left, and a grid view in the actual dashboard pane. For comparison, here’s what the editor used to look like when you went to create a new dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HuAx8F_2OxeGYSmU" /></figure><h3>Widgets</h3><p>Instead of using the ADD CHART link at the top right, you start by selecting a widget. There are three new widget types — Gauge, Scorecard, and Text. The other four (Line, Stacked Area, Stacked Bar, and Heatmap) are familiar and still there. Let’s take a look at the new ones!</p><h3>Gauge</h3><p>The Gauge widget finally delivers a way for you to represent a single number on a dashboard as an indicator of health — along with a way to color-code it so that you can see whether things are healthy at a glance. Here’s a basic example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YPTKxxchcQnC16sW" /></figure><p>This chart is showing how much of the CPU limit is being utilized by all containers averaged across a specific cluster (of course, you can use other aggregation types like Min/Max/Sum/99th percentile). One thing I really love about this is the ability to specify the ranges for the chart to change colors. For example, here’s how to set a warning threshold that will turn the chart yellow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MTsB_Z51tMf_14lW" /></figure><p>You can also set a danger threshold to turn it red:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*90L9nidMo_IcBjtd" /></figure><p>For my purpose, this is actually a pretty good way to represent the “saturation” signal — if the containers, on average, are not close to the limit, there should be plenty of capacity left. If I care about a specific service, I’d probably want to filter this down further:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*huh8hxMqLyy5fAxt" /></figure><h3>Scorecard</h3><p>Next up — the Scorecard widget! It also allows you to show a single numeric value on the dashboard. However, unlike a gauge, it tracks the value over time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uXCVzW8AOQ2g7w4u" /></figure><p>Similarly to a gauge, you can use warning and danger thresholds to have the widget change colors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n7AvXMlCRO-tvyfF" /></figure><p>You can also change how the value over time is displayed. The line is the default option, but you can also use a bar chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7NSzorHo1VaPDIph" /></figure><p>The Icon option will simply show you whether the value is within the desired range (with a green checkmark):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KLLRKVCD3bh388Xh" /></figure><p>Or out of the desired range with a red one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yYtptYE-ryXzmCHl" /></figure><p>For our purposes, I actually really like this widget for visualizing latency, for example. It’s an easy way to quickly see whether your service is meeting a specific performance objective.</p><h3>Text</h3><p>The final new widget type is Text. At first glance, it’s not much to get excited about:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rJiuWhu1LCtvEvuq" /></figure><p>But this widget really shines once you select the Markdown option — this essentially turns it into a rich text editor (provided you’re comfortable writing <a href="https://en.wikipedia.org/wiki/Markdown">Markdown</a>). This is a great way to, for example, add documentation directly to the dashboard that can describe exactly what a user is looking at.</p><p>For example, you can use this widget to document the dashboard you’ve created so far to help orient a new user like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h-sIuD7FioI6CzFb" /></figure><h3>Layout options</h3><p>By now, you might have noticed that you now have the option to do things like move widgets around and resize them. This is all part of the new “Mosaic” option in the editor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*P5CQ3cKJquhnWgg3" /></figure><p>By being able to resize charts as you see fit, you have a whole host of options available to you on how you want your dashboard to look — you’re no longer limited to just the basic columns view. For example — you can add a request count chart to your dashboard that spans both of the columns we’ve created so far:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xptQpjLpf5rs_ZJB" /></figure><p>And now, you have a dashboard that represents three of the four “golden signals” — traffic (as request count), saturation (as measured by CPU limit usage), and latency.</p><h3>Basic vs Advanced configuration</h3><p>The last new thing you should be aware of is the new distinction between configuring a chart using the Basic and Advanced configuration options. For example, if you add a Heatmap widget to your dashboard, you may see a message like this letting you know that you need to use the Advanced mode to configure it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bsr6KZeINjo05DZJ" /></figure><p>That option is at the top of the chart configuration options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*or4Ype1bkbhCKsnR" /></figure><p>While the Basic mode should serve most users in most cases, there are times when you may, for example, need to explicitly specify the chart alignment. Switching to Advanced mode lets you do just that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JoJV5PFQ2n5M0FiM" /></figure><p>For our heatmap, you can then select options like the preprocessing step and the alignment to get the exact chart you want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_1_6ecu3lidhatdG" /></figure><h3>In summary</h3><p>I am really excited about this new feature — and I’m really looking forward to seeing the new dashboards that y’all are going to create with the new widget, layout, and configuration options. Thanks for reading, and, as always — let me know what you’d like me to tackle next. These days more than ever — stay healthy out there!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e04347e205e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/building-monitoring-dashboards-for-fun-and-profit-2e04347e205e">Building monitoring dashboards for fun and profit</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Troubleshooting services on GKE]]></title>
            <link>https://medium.com/google-cloud/troubleshooting-services-on-gke-872470e60d51?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/872470e60d51</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Tue, 03 Nov 2020 17:22:59 GMT</pubDate>
            <atom:updated>2020-11-03T17:22:59.269Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last <a href="https://medium.com/google-cloud/setting-up-cloud-operations-for-gke-a21b49979693">post</a>, I reviewed the new GKE monitoring dashboard and used it to quickly find a GKE entity of interest. From there, I set up an alert on container restarts using the in-context “create alerting policy” link in the entity details pane. This time, I wanted to have a go at troubleshooting an incident using this setup.</p><h3>The setup</h3><h4>The app</h4><p>You can see the full code for the simple demo app I’ve created to test this <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/crashing-pod-demo/main.go">here</a>. The basic idea is that it exposes two endpoints — a / endpoint, which is just a “hello world”, and a /crashme endpoint, which uses Go’s os.Exit(1) to terminate the process. I then created a container image using Cloud Build and <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/crashing-pod-demo/deployment.yaml">deployed</a> it to GKE. Finally, I exposed the service with a load balancer.</p><p>Once the service was deployed, I checked the running pods:</p><pre>✗ kubectl get pods</pre><pre>NAME READY STATUS RESTARTS AGE</pre><pre>restarting-deployment-54c8678f79-gjh2v 1/1 Running 0 6m38s</pre><pre>restarting-deployment-54c8678f79-l8tsm 1/1 Running 0 6m38s</pre><pre>restarting-deployment-54c8678f79-qjrcb 1/1 Running 0 6m38s</pre><p>Notice that RESTARTS is at zero for each pod initially. Once I hit the /crashme endpoint, I saw a restart:</p><pre>✗ kubectl get pods</pre><pre>NAME READY STATUS RESTARTS AGE</pre><pre>restarting-deployment-54c8678f79-gjh2v 1/1 Running 1 9m28s</pre><pre>restarting-deployment-54c8678f79-l8tsm 1/1 Running 0 9m28s</pre><pre>restarting-deployment-54c8678f79-qjrcb 1/1 Running 0 9m28s</pre><p>I was able to confirm that each request to the endpoint resulted in a restart. However, I had to be careful to not do this too often — otherwise, the containers would go into CrashLoopBackOff, and it would take time for the service to be available again. I ended up using this simple loop in my shell (zsh) to trigger restarts when I needed them:</p><pre>while true;</pre><pre>  do</pre><pre>    curl <a href="http://$ip_address:8080/crashme;">http://$IP_ADDRESS:8080/crashme;</a></pre><pre>    sleep 45;</pre><pre>done</pre><h3>The alert</h3><p>The next step was to set up the alerting policy. Here is how I configured it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fAyZ52ZY7BeZTsFE" /></figure><p>I used the kubernetes.io/container/restart_count metric, filtered to the specific container name (as specified in the deployment yaml <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/crashing-pod-demo/deployment.yaml">file</a>), and configured the alert to trigger if any timeseries exceeded zero — meaning if any container restarts were observed.</p><p>The setup was done — I was now ready to test and see what happens!</p><h3>Testing alert</h3><p>When I was ready, I started the looped script to hit the /crashme endpoint every 45 seconds. The restart_count metric is sampled every 60 seconds, so it didn’t take very long for an alert to show up on the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_dpFfJripTmqLyUF" /></figure><p>I moused over the incident to get more information about it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ihUXko2bUA2bYiRy" /></figure><p>Already, this is an improvement over the previous version of this UI, where I couldn’t interact with the incident cards.</p><p>I then clicked on “View Incident”. This took me to the Incident details screen, where I could see the specific resources that triggered it. In my case, it was pointing to the container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7qrFVXGuwWPnmZtn" /></figure><p>I then clicked on View Logs to see the logs (in the new Logs Viewer!) — and sure enough, it’ was immediately apparent that the alert was triggered by the containers restarting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xU_egFlXUVEabSFw" /></figure><p>This is all very nicely tied together and makes troubleshooting during an incident much easier!</p><h3>In summary….</h3><p>I’m a big fan of the new GKE dashboard — I really like the new alerts timeline, I like that the incidents are clearly marked and that I can actually interact with them to get the full details of exactly what happened, all the way down to the container logs that tell me the actual problem.</p><p>Thanks for reading, and come back soon for more. As always, please let me know what other SRE or observability topics you’d like to see me take on. And now more than ever — stay healthy out there!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=872470e60d51" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/troubleshooting-services-on-gke-872470e60d51">Troubleshooting services on GKE</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up Cloud Operations for GKE]]></title>
            <link>https://medium.com/google-cloud/setting-up-cloud-operations-for-gke-a21b49979693?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a21b49979693</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Wed, 28 Oct 2020 22:34:17 GMT</pubDate>
            <atom:updated>2020-10-28T22:34:17.231Z</atom:updated>
            <content:encoded><![CDATA[<p>My interest in observability in Google Cloud developed in large part in the context of working with GCP customers running workloads on GKE, and one of my very first posts here <a href="https://medium.com/google-cloud/new-stackdriver-monitoring-for-kubernetes-part-1-a296fa164694">covered</a> using Stackdriver for those workloads. The very first <a href="https://youtu.be/lwBBAvPxO9c">episode</a> of <a href="https://www.youtube.com/playlist?list=PLIivdWyY5sqLuKKx4pcdEAkJY1HevjVVm">Stack Doctor</a> also went over what were at the time the “new” GKE monitoring capabilities. This was over two years ago, and there have been some great updates to those capabilities since then. I thought it was time to revisit Cloud Ops for GKE, have another look at the dashboards, and try out the new capabilities. Let’s dive in!</p><h3>The basics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NOtJ_noBbCs5CNZm" /><figcaption>New GKE monitoring dashboard</figcaption></figure><p>Right away, the new GKE monitoring dashboard looks very different from what we saw <a href="https://cloudplatform.googleblog.com/2018/05/Announcing-Stackdriver-Kubernetes-Monitoring-Comprehensive-Kubernetes-observability-from-the-start.html">released</a> at Kubecon in May of 2018. Instead of 3 tabs — infrastructure, workloads, and services — you now have lists of all the different entities in the workspace (which, as you may <a href="https://cloud.google.com/monitoring/workspaces">recall</a>, can aggregate monitoring information from multiple projects). The first thing I thought of when I saw this was “Well, that’s great, but there is a default namespace in every cluster, how will I ever find what I need?” As it turns out — someone has already thought of that!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vCs8-8b3PzeEq639" /></figure><p>There’s a really cool filtering feature that will actually help you to find exactly what you’re looking for. In this example, if you have a copy of the “frontend” service running in three different clusters — the filter lets you select the exact one you want. Once you apply the filter, the entire dashboard is filtered to match:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vz7gPX7hWkOSEoou" /></figure><p>One of the things that’s been kind of hard to do in the past is to get an aggregated view of the data. For example, what does my resource utilization look like across my namespace? This new view makes it really easy to get this kind of aggregation — and it will scale no matter how many resources are being aggregated, even if you have thousands of pods across hundreds of namespaces. But if you do want to see all of those resources — you can click View All.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9o8EYO52zQkImc62" /></figure><p>That will let you see all of the entities in that category. If that list is too overwhelming — you can filter from here, too!</p><p>You can still select a row in the table to get its details. For example, you can see metrics for a pod by clicking on it, which opens the details panel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0k9HUT9lIRqF144z" /></figure><p>From there, click on the Logs tab to get the logs for the pod:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tpfpBlh0LWbyCmwQ" /></figure><p>If needed — you can filter logs by severity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/0*QtkVlS-gcI9b4XSC" /></figure><p>Click the Open in Logging button:</p><p>That opens the (<a href="https://www.youtube.com/watch?v=8HJDnh8TVOM&amp;list=PLIivdWyY5sqLuKKx4pcdEAkJY1HevjVVm&amp;index=22&amp;t=0s">new</a>!) Logs Viewer with the query to get those logs pre-populated and executed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iJ4VABkjGQppKtFF" /></figure><p>One other thing I really like is that you can easily create an Alerting Policy from the Metrics tab of the details screen:</p><p>If you’ve been following me for a while, you probably know that I don’t necessarily think it’s a good idea to alert on these kinds of infrastructure metrics, but I can absolutely envision a use case where you might need to know about, for example, hitting memory limits or something like that.</p><h3>In summary…</h3><p>I’m really happy to see the team make great progress on this experience in the last two years, and I really like where this is going. Next time, I want to approach this from a different angle by forcing an alert/incident and seeing how useful it’s going to be for troubleshooting.</p><p>Thanks for reading, and stay healthy out there!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a21b49979693" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/setting-up-cloud-operations-for-gke-a21b49979693">Setting up Cloud Operations for GKE</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to alert on SLOs]]></title>
            <link>https://medium.com/google-cloud/how-to-alert-on-slos-2a5ce8c4e7dd?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a5ce8c4e7dd</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Mon, 28 Sep 2020 14:15:55 GMT</pubDate>
            <atom:updated>2020-09-28T14:15:55.186Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve spent quite a bit of time looking at defining and configuring SLOs in Service Monitoring. And lately, I’ve been getting lots of questions about what happens next — once the SLO is configured, folks want to know how to use alerting to be notified about potential, imminent, and in-progress SLO violations. Service Monitoring provides SLO error budget burn <a href="https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/alerting-on-budget-burn-rate">alerts</a> to accomplish just that, but using these alerts is not always intuitive. I set out to try these out for myself and document what I found along the way. Let’s see what happens!</p><h3>In theory…</h3><p>The SRE Workbook has a whole <a href="https://landing.google.com/sre/workbook/chapters/alerting-on-slos/">chapter</a> devoted to alerting on SLOs. I don’t think I need to reproduce it here, but there two classes of considerations that are very important:</p><ul><li>What is the relationship between your alert and the number of SLO events your service experiences?</li></ul><p><strong>— Precision</strong> is your true-positive rate — what fraction of your alerts will actually indicate an event you need to know about?</p><p><strong>— Recall</strong> is your “sensitivity” — what fraction of your events will actually result in an alert?</p><ul><li>What is the relationship between an alert and the event that triggered it?</li></ul><p><strong>— Detection time</strong> is the time between when the event started and the time of the alert firing</p><p><strong>— Reset time</strong> is the time between when the event ended and the time of the alert resetting</p><p>As you can imagine, your SLO itself AND the alert you configure can affect all of these. It’s impossible to optimize all of these (driving precision and recall to 100% AND detection time and reset time to zero) AND not drown in alerts, most of which will not be actionable. At the same time, it is not practical to wait until you’re out of error budget to alert — advance notice is necessary to be able to take appropriate action to ensure that the service stays within its daily/weekly/monthly error budget. Alerting on <strong>error budget burn</strong>, rather than on whether your service is within SLO for any specific short time period, is the way to get there.</p><p>The next obvious question — just how much of the error budget should be burned before triggering an alert? The most advanced approach recommended in the SRE workbook <a href="https://landing.google.com/sre/workbook/chapters/alerting-on-slos/">chapter</a> is to use multiwindow, multi-burn-rate alerts (see section 6). But to get started, I very much like the idea of two separate alerts with different burn rates, following the recommendation of “2% burn in 1 hour and 5% burn in 6 hours”.</p><p>Finally — and this is the hard part — how do you identify the exact burn rate and alerting window to get to the level of precision and reset time you need while minimizing operational overhead (responding to unneeded alerts) for your service? From the book:</p><blockquote>“For burn rate-based alerts, the time taken for an alert to fire is:</blockquote><blockquote>time to fire = (1-SLO) / error ratio * alerting window size * burn rate</blockquote><blockquote>The error budget consumed by the time the alert fires is:</blockquote><blockquote>budget consumed = burn rate * alerting window size / period”</blockquote><p>Let’s look at an example to clarify this. Let’s say you have an availability SLO of 99.9% over a rolling 28-day window. You’d like to alert if you’ve consumed 1% of your error budget in the previous hour. That means you want the alert to trigger if the error rate over the last hour is greater than target budget * (1-SLO) * (period/window).</p><p>However, Service Monitoring doesn’t allow you to specify an error rate — instead, you have to provide a <a href="https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/alerting-on-budget-burn-rate">burn rate threshold</a>. The burn rate is the rate at which your service consumes error budget. If the burn rate is 1.0, then at the end of the SLO evaluation period, you will have consumed exactly 100% of your error budget. If the burn rate is 2.0, then at the end of the period you will have consumed 200% of your error budget (or you will have consumed all of your error budget half-way through your evaluation period) — and so on. Using the equation above to calculate the burn rate, we get</p><p><em>burn rate = budget consumed * period / alerting window</em></p><p>In our example:</p><ul><li>SLO = 99.9%, or .999</li><li>Alerting window size = 1 hour</li><li>Budget consumed = 1% or .01</li><li>Period = 28 days or 672 hours</li></ul><p>This means that</p><p>burn rate = .01 * 672 / 1= 6.72</p><p>If we configure our alert to have a 60 minute lookback period and a burn rate threshold of 6.72, we can then calculate how quickly we would get an alert based on the error ratio our service experiences. For example, if our error rate goes up to 1%, time to fire = (1-.999) / .01 * 1 * 672 = 67.2 hours, which is almost 3 days — this is an alert with low recall!</p><p>If we want to improve our recall and have the alert fire right at 1 hour, we calculate the burn rate using 1 = (1-.999) / .01 * 1 * burn rate, which means that burn rate = 10. To confirm — time to fire = (1-.999) / .01 * 1 * 6.72 = 1! Our alert will fire in exactly one hour.</p><h3>In practice…</h3><p>I wanted to look at using Service Monitoring to implement a simpler approach (as described in section 5) — two separate alerts with different burn rates, following the recommendation of “2% burn in 1 hour and 5% burn in 6 hours”.</p><h3>The setup</h3><h3>Service and SLO</h3><p>To start, I needed a simple application whose error rate I could control precisely without redeploying code. You can see the code for the service <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/slo-alerting-demo/log-based/app-logging.js">here</a>. It’s a pretty basic Node.js Express application that writes log entries for every request and failure. I then configured log-based metrics to count both. I configured a service using the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kf3kTES0i4ejh9Hr" /></figure><p>Next, I needed to define the SLO. Because my service is using two different metrics for the “good” and “bad” filters, I could not figure out how to create such an SLO in the UI. As such, I needed to use the API — here’s the API call:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d219073ef01d656b8d3d96760c04d19/href">https://medium.com/media/5d219073ef01d656b8d3d96760c04d19/href</a></iframe><p>This created an SLO for “95% availability over a rolling 28 day period”. Here it is in the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GyqJVz7Dp8FhC4T1" /></figure><h3>Fault injection</h3><p>Next, I needed to inject a desired error rate into the service to be able to validate that increasing the error rate would result in error budget burn and trigger alerts. After some internal discussion, I identified a couple of options:</p><ul><li>Using Istio’s <a href="https://istio.io/latest/docs/tasks/traffic-management/fault-injection/">fault injection</a> capabilities</li><li>Having my app read an environment variable that could be managed by a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/configmap">ConfigMap</a></li></ul><p>I decided to use the latter. You can see my code <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/slo-alerting-demo/log-based/app-logging.js">here</a>; the gist of it is that I used Node’s process.ENV.&lt;variable name&gt; to have my code read the variable when servicing the request. The value of the variable itself is set in the ConfigMap — I followed these <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/">instructions</a> to create it from a basic .properties file. One additional complication I discovered is that the pod would only read the value of the environment variable at startup — so I needed to delete it and have the deployment recreate it when I wanted to change the value.</p><h3>Alerts</h3><p>Finally, I was ready to set up my alerts. As discussed earlier, I wanted to test two alerts:</p><ul><li>2% error budget burn in 1 hour</li><li>5% error budget burn in 6 hours</li></ul><p>As mentioned in the “theory” section, Service Monitoring expects two inputs for error budget burn alerts — a lookback duration and a burn rate threshold:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tal2aKnaWn-gKig-" /></figure><p>Because I knew I wanted to monitor error budget burn over an hour, the lookback duration was an easy decision — I set it to 60 minutes. However, I needed to calculate my burn rate threshold. I already showed how I calculated it for the “1% in 1 hour” burn example. This time, my inputs were as follows:</p><ul><li>Alerting window size = 1 hour</li><li>Budget consumed = 2% or .02</li><li>Period = 28 days or 672 hours</li></ul><p>To calculate burn rate, I used</p><p>burn rate = budget consumed * period / alerting window = .02 * 672 / 1 = 13.44</p><p>This is the burn rate threshold our alert policy expects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nqHkbL0yol3VhNPW" /></figure><p>Next, I needed to calculate the burn rate threshold for the “5% error budget burned in 6 hours” alert. This time:</p><ul><li>Alerting window size = 6 hours</li><li>Budget consumed = 5% or .05</li><li>Period = 28 days or 672 hours</li></ul><p>Which means that</p><p>burn rate = budget consumed * period / alerting window = .05 * 672 / 6 = 6.72 :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fbiMx77jyiVrxV6h" /></figure><p>Now, both of my alerts were ready to go. I was ready to introduce errors into the service and test the alerts.</p><h3>Testing alerts</h3><p>The first question I needed to answer was — what should the error rate be to trigger the alert in a reasonable time frame? As discussed above:</p><p>time to fire = (1-SLO) / error ratio * alerting window size * burn rate</p><p>This makes</p><p>error ratio = (1 — SLO) * alerting window size * burn rate / time to fire</p><p>For the first alert, my values were as follows:</p><ul><li>Desired time to fire = 1 hour</li><li>SLO = 95% or .95</li><li>Alerting window size = 1 hour</li><li>Burn rate = 13.44</li></ul><p>This meant that my error ratio would need to be at least 0.672, or 67.2% — that’s pretty high! I recreated the configmap with that value and deleted the pod, which was automatically recreated by the deployment. Almost immediately, I saw a decrease in my SLI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cte5WS2wFku47ivi" /></figure><p>Because I set my SLO higher than I intended, the service was already out of error budget, but the immediate drop was still exactly what I expected. My service also hadn’t been running for a full month, so I thought that the alert would trigger much faster than my calculation, because I hadn’t actually accumulated enough error budget at this point, and burning 2% would take no time at all.</p><p>Sure enough, I soon saw the error budget burn rate cross the threshold:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7pNyjCUhXi9jesvK" /></figure><p>And the alert generated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*nUdqe8ZyiGbHJEUH" /></figure><h3>Wrapping up…</h3><p>I’m really glad I took the time to figure this out AND document my journey along the way, and I hope you find this helpful. In particular, I found that there is not a lot of information available on exactly what an error budget burn rate is in product documentation, and it took some digging to really understand the math that goes into calculating it. Nevertheless, I very much like this approach to setting up alerts on SLOs. Thanks for reading, and please let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a5ce8c4e7dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/how-to-alert-on-slos-2a5ce8c4e7dd">How to alert on SLOs</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some time ago, I looked at using the Service Monitoring API to create basic SLOs against “out of…]]></title>
            <link>https://medium.com/google-cloud/some-time-ago-i-looked-at-using-the-service-monitoring-api-to-create-basic-slos-against-out-of-b3555b08036f?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3555b08036f</guid>
            <category><![CDATA[error-budget]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Mon, 31 Aug 2020 17:20:15 GMT</pubDate>
            <atom:updated>2020-09-02T18:38:35.034Z</atom:updated>
            <content:encoded><![CDATA[<h3>Creating SLOs with Terraform</h3><p>Some time ago, I <a href="https://medium.com/google-cloud/slos-with-stackdriver-service-monitoring-62f193147b3f?source=---------6------------------">looked</a> at using the Service Monitoring API to create basic SLOs against “out of the box” services like App Engine. This functionality has seen a lot of updates since then, and there’s now Terraform support for creating custom <a href="https://www.terraform.io/docs/providers/google/r/monitoring_service.html">services</a> and <a href="https://www.terraform.io/docs/providers/google/r/monitoring_slo.html">SLOs</a>. I wanted to have a go at this myself to see how it works.</p><h3>Creating the service</h3><p>SLO Monitoring does a great job of identifying services for you if you’re using things like Istio, App Engine, or Cloud Endpoints. But what if your service is on GCE, for example? In this case, you need to define it as a custom service, which will then allow you to define SLOs against it.</p><p>Here’s how to define a “<a href="https://www.terraform.io/docs/providers/google/r/monitoring_service.html">monitoring service</a>” in Terraform:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/026c402c934a672fd152b06846c47f13/href">https://medium.com/media/026c402c934a672fd152b06846c47f13/href</a></iframe><p>The service definition is actually very simple — you just provide a service ID unique to your project and a display name. Once you run “terraform apply”, the service is then visible in the Console:</p><p>From there, you can use the UI to create an SLO against it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DQ6DspOdYYKUFSWW" /></figure><p><strong>Note</strong> that you have to use “Other” as the metric — custom services don’t have an “out of the box” understanding of availability and latency. So, you need to have a good SLI for your service. You could use something like a log-based metric, a metric emitted by the Google Cloud Load Balancer if you’re using that, or a custom metric being written by the service. Let’s take a look at defining an SLO using the latter.</p><h3>Defining the SLO</h3><p>Here’s how to define an SLO using Terraform:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c0804519931f4b60a65c5efdf4ad410/href">https://medium.com/media/5c0804519931f4b60a65c5efdf4ad410/href</a></iframe><p>There are 3 main things to consider here:</p><ul><li>The <strong>basics</strong> — the resource ID, the SLO ID, the service you’re defining the SLO against, and the SLO display name.</li><li>The <strong>SLI</strong> — are you going to be using a request- or windows-based SLI? If request-based — how will you count total requests and differentiate between good and bad requests?</li></ul><p>In my example, I’m using a service that’s been instrumented to emit two separate metrics — one to count all requests and another to count errors. This makes things quite simple.</p><ul><li>The <strong>goal</strong> — what’s the actual target for your SLO?</li></ul><p>In this example, the goal is that 99% of requests are successful over a rolling 1-day period.</p><h3>Creating and validating</h3><p>At this point, run terraform plan to make sure everything is correct:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZpOo-vPEtZGgCu3JZqGyw.png" /><figcaption>Output of `terraform plan`</figcaption></figure><p>If everything looks correct, run terraform apply to create the service and the SLO(s):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uylMFw4-V93uKCqxYIQ_g.png" /><figcaption>Output of `terraform apply`</figcaption></figure><p><strong>Note</strong> that my file has two SLOs — a request-based one for availability and a windows-based one for latency. That’s why 3 resources are being created.</p><p>At this point, you can go back to the console and check your new service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*91GRnOYpPlT-_1qJ" /></figure><p>I’ve clearly not set my availability target correctly (or my service is having some serious issues) — I should absolutely revisit this before I take the next step to set up an error budget burn alert on this.</p><h3>Summary</h3><p>I’m really excited to see service and SLO support come to Terraform, and I hope lots of folks will take advantage of this to extend their automation capabilities. At this point, all of the major monitoring primitives can be created automatically once a project is up — this is great news! Thanks for reading, and let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3555b08036f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/some-time-ago-i-looked-at-using-the-service-monitoring-api-to-create-basic-slos-against-out-of-b3555b08036f">Some time ago, I looked at using the Service Monitoring API to create basic SLOs against “out of…</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Tracing and Logging with OpenTelemetry and Stackdriver]]></title>
            <link>https://medium.com/google-cloud/integrating-tracing-and-logging-with-opentelemetry-and-stackdriver-a5396fbc3e78?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5396fbc3e78</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[tracing]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Tue, 18 Feb 2020 17:16:02 GMT</pubDate>
            <atom:updated>2020-02-18T17:16:02.395Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the main benefits of using an all-in-one observability suite like Stackdriver is that it provides all of the capabilities you may need. Specifically, your metrics, traces, and logs are all in one place, and with the GA <a href="https://cloud.google.com/monitoring/docs/monitoring_in_console">release</a> of Monitoring in the Cloud Console, that’s more true than ever before. However, for the most part, each of these data elements are still mostly independent, and I wanted to attempt to try to unify two of them — traces and logs.</p><p>The idea for the project was inspired by the excellent work <a href="https://github.com/alexamies">Alex Amies</a> did in his <a href="https://cloud.google.com/solutions/troubleshooting-app-latency-with-cloud-spanner-and-opencensus">Reference Guide</a> on using OpenCensus to measure Spanner performance and troubleshoot latency. Specifically, he included an applog <a href="https://github.com/GoogleCloudPlatform/opencensus-spanner-demo/tree/master/applog">package</a> that integrated traces and logs in OpenCensus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0DmxlUedaGdJWbh1" /></figure><p>I wanted to follow my <a href="https://dev.to/yurigrinshteyn/distributed-tracing-with-opentelemetry-in-go-473h">post</a> on tracing with OpenTelemetry and attempt to create integrated traces and logs. Let’s dive in!</p><h3>The app</h3><p>I created a very basic Go app using the Mux router that:</p><ol><li>Receives a request on /.</li><li>Sleeps between 0 and 9 seconds.</li><li>Makes a backend call (to <a href="https://www.google.com/">https://www.google.com</a>)</li></ol><p>My intent was to create a root span with two children — one for the delay that simulates an internal process and another for the backend call.</p><h3>The code</h3><h3>Main function</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8aac76a405ad4221fb780ed6c0360cd0/href">https://medium.com/media/8aac76a405ad4221fb780ed6c0360cd0/href</a></iframe><p>The main function simply sets up my tracing and logging and uses the mainHandler to respond to requests on /.</p><h3>Tracing setup</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98e7351d67cd727e4a624bbfd0313737/href">https://medium.com/media/98e7351d67cd727e4a624bbfd0313737/href</a></iframe><p>The tracing set up is pretty straightforward — I’m simply using the exporter written by <a href="https://github.com/ymotongpoo">Yoshi Yamaguchi</a>, a fantastic Developer Advocate. It’s the same exporter I used in my post on tracing without any changes.</p><h3>Logging setup</h3><p>This is where things start to get interesting.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80d9d80323a4e6c4c5c6ff0687a9b611/href">https://medium.com/media/80d9d80323a4e6c4c5c6ff0687a9b611/href</a></iframe><p>I’ve largely lifted this from Alex’s <a href="https://github.com/GoogleCloudPlatform/opencensus-spanner-demo/blob/master/applog/applog.go">work</a>. The init function simply sets up the logging client.</p><h3>Writing logs</h3><p>This is where the trace/logging integration really happens.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83449d6f53796acb73f36c0a8ad99250/href">https://medium.com/media/83449d6f53796acb73f36c0a8ad99250/href</a></iframe><p>In Stackdriver, traces and logs can be connected by writing the span ID and the trace ID in the payload of the log message. Here, I’m using the context to extract both the span and trace and then extract their IDs. I then write them to the log payload. Here’s what a resulting log message looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*jH0_UUTOeWUpcEiy.png" /></figure><p>Notice that the spanId and trace fields are populated appropriately.</p><h3>Viewing traces</h3><p>I can run the app locally (after using gcloud auth application-default login to write default credentials) and send traffic to <a href="http://localhost:8080/">http://localhost:8080</a>. Here&#39;s a resulting trace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tCZ4LXP0sUCQq_lD.png" /></figure><blockquote><em>Note that the trace contains both Events — added with the </em><em>span.AddEvent() method - and logs, written as described above.</em></blockquote><h3>Viewing logs</h3><p>I can click on one of the log entries to see the full details of the log message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5mGzrjrn2Vk0Zz_l.png" /></figure><p>I can then click Open in Logs Viewer and see this log entry there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X2c8MfaSZELHAISe.png" /></figure><h3>In conclusion…</h3><p>I was very glad to see that this somewhat underappreciated functionality from OpenCensus still works in OpenTelemetry with minor changes. Specifically, I had to find the new APIs to use in printf() to extract the span from context and then get its span ID and trace ID, and this does not seem to be well documented. With that said, I hope this brief tutorial is useful to others looking to build a more integrated approach to observability with Stackdriver, especially in distributed systems. Many thanks for Alex for doing the original work on this, and thank you for reading!</p><p><em>Originally published at </em><a href="https://gist.github.com/295b0056b9ed4a4c39cd7fd616887bd4"><em>http://github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5396fbc3e78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/integrating-tracing-and-logging-with-opentelemetry-and-stackdriver-a5396fbc3e78">Integrating Tracing and Logging with OpenTelemetry and Stackdriver</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom metrics in Node.js with OpenTelemetry (and Prometheus)]]></title>
            <link>https://medium.com/google-cloud/custom-metrics-in-node-js-with-opentelemetry-and-prometheus-c10c8c0204d3?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c10c8c0204d3</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Mon, 27 Jan 2020 19:50:18 GMT</pubDate>
            <atom:updated>2020-01-27T19:50:18.816Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last <a href="https://dev.to/yurigrinshteyn/distributed-tracing-with-opentelemetry-in-go-473h">post</a>, I tackled my first project with <a href="https://opentelemetry.io/">OpenTelemetry</a> and built a basic demo to show how to use distributed tracing and the Stackdriver exporter. I chose Go for that exercise because at the time it was the only language that had a Stackdriver exporter for tracing available. This time, I wanted to attempt using OpenTelemetry for metric instrumentation and noticed that opentelemetry-go does not have a Stackdriver exporter for metrics ready yet. I attempted to use the Prometheus exporter instead, but could not figure out how to make it play nice with the Mux router and switched to Node.js instead.</p><p>So — here’s how I built a basic Hello World Node.js app and instrumented it with OpenTelemetry to expose basic “golden signals” metrics (request count, error count, and latency) to Prometheus.</p><p>As always, I’ve made my code available in a Github <a href="https://github.com/yuriatgoogle/stack-doctor/opentelemetry-metrics/demo">repo</a>. Let’s review the interesting parts.</p><h3>Imports and setup</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d65f3fbfc3a827f54d6a6e3b99cd5f8/href">https://medium.com/media/4d65f3fbfc3a827f54d6a6e3b99cd5f8/href</a></iframe><p>I’m only using three external packages — Express to handle requests and two OpenTelemetry packages — one to write the metrics, and another to export them to Prometheus. Requests to :8081/metrics will be handled by Prometheus.</p><blockquote><em>Note that I’ve overridden the default port that’s set in the Prometheus exporter </em><a href="https://github.com/open-telemetry/opentelemetry-js/blob/master/packages/opentelemetry-exporter-prometheus/src/prometheus.ts#L32-L38"><em>code</em></a><em>.</em></blockquote><h3>Metric definition</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a64b32ca8f080cce931c16fc34c76d75/href">https://medium.com/media/a64b32ca8f080cce931c16fc34c76d75/href</a></iframe><p>In the next part, I define the three metrics I want to track. Request count and error count are both monotonic counters, meaning that they can only be increased. Response latency is a non-monotonic gauge, since I want to track latency for every request. I am also using a single label because I could not figure how to get metrics to work properly with no labels.</p><h3>Request handling</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05c26f98646b9b4199a2e3c91eb4be96/href">https://medium.com/media/05c26f98646b9b4199a2e3c91eb4be96/href</a></iframe><p>At this point, the app is ready to accept incoming requests. Requests to :8081/metrics will be handled by Prometheus, and I&#39;m using Express to handle requests to :8080/. For every request, the app creates a timer ( measuredLatency) and keeps it running until it&#39;s ready to respond. It generates a random number between 0 and 100 and uses it to return an error half the time, in which case latency is not reported. If there&#39;s no error to return, the apps sleeps between 1 and 10 seconds, takes another time snapshot, calculates the elapsed time, and returns that value as the latency.</p><h3>Metrics endpoint</h3><p>When I run the app locally, I can then hit the /metrics endpoint and see the metrics I&#39;ve registered there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*fQOCVVmB_OhIpRuH.png" /></figure><p>At this point, I was ready to run the app and generate actual traffic to it. That required:</p><ul><li>creating a GKE cluster</li><li>installing Prometheus on it</li><li>deploying the app and configure the /metrics endpoint to be scraped</li></ul><p>Let’s dive in!</p><h3>Creating the cluster</h3><p>I took a very straightforward approach and used the Cloud Console to create a single-node cluster (that turned out to be the wrong choice) using an n1-standard node. I then used gcloud container clusters get-credentials &lt;cluster&gt; --zone=&lt;zone&gt; to get credentials for it to be able to access it from my local workstation.</p><pre>$ kubectl get nodes</pre><pre>NAME                                     STATUS   ROLES    AGE     VERSION</pre><pre>gke-small-cluster-pool-1-b40da891-fc1p   Ready    &lt;none&gt;   7h27m   v1.15.7-gke.23</pre><p>I took the simple option here:</p><pre>$ brew install helm <em># install helm, the package manager for Kubernetes</em></pre><pre>$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/</pre><pre>$ helm repo update</pre><pre>$ helm install prometheus stable/prometheus</pre><p>That was enough to get Prometheus up and running, but I had to increase the size of my cluster to get enough CPU to run everything.</p><pre>&gt; kubectl get pods</pre><pre>NAME                                            READY   STATUS    RESTARTS   AGE</pre><pre>prometheus-alertmanager-77995478d8-lh4c5        2/2     Running   0          6m34s</pre><pre>prometheus-kube-state-metrics-f96cbbf97-vc56g   1/1     Running   0          6m34s</pre><pre>prometheus-node-exporter-8xlwp                  1/1     Running   0          114s</pre><pre>prometheus-node-exporter-s2pnt                  1/1     Running   0          6m34s</pre><pre>prometheus-node-exporter-xbtrs                  1/1     Running   0          112s</pre><pre>prometheus-pushgateway-747dc5fb57-rw72p         1/1     Running   0          6m34s</pre><pre>prometheus-server-6f7467bfdd-5vfgx              2/2     Running   0          6m34s</pre><p>I verified that Prometheus was working by using port-forward:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LAqzJiFR_9cA5nqj.png" /></figure><h3>Deploying the app</h3><p>I created a simple Dockerfile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32ac724063020c2d3c50a7cd4f5496cf/href">https://medium.com/media/32ac724063020c2d3c50a7cd4f5496cf/href</a></iframe><p>I built the image using gcloud builds submit and created a deployment file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c535cd09dbed1afdde48bbb5f385b72/href">https://medium.com/media/7c535cd09dbed1afdde48bbb5f385b72/href</a></iframe><p>Next, I used kubectl apply -f to create the app and kubectl expose deployment --type=LoadBalancer to make it externally available. Once the external IP address was available, I configured a Stackdriver Uptime Check against it to generate a steady stream of traffic.</p><p>At this point, all of the setup was done. I accessed Prometheus again to confirm that the metrics I created using OpenTelemetry were available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oVTSOoteXtmw8Jx6.png" /></figure><h3>What have I learned?</h3><p>In my OpenCensus <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/opencensus-metrics/node/app.js">exercise</a>, I learned about using primitives like Views, Measurements, and Measures. I also learned how to create histograms and control aggregation. My very limited experience with OpenTelemetry has shown that this model seems to be somewhat simpler. In this exercise, I simply defined metrics and set their value when necessary. The exporter took care of the rest. While I probably missed out on a lot of detail, this initial experience makes things seem quite a bit easier to get started with, which is great for folks just getting started with this kind of instrumentation.</p><p>I’ve also realized that the project is still in relatively early stages. For example, there are no Stackdriver metric exporters available as I’m writing this. Trace exporters have been written for (at least) Go and Node.js, but metrics are not yet available. As such, Using this on GCP would require me to setup the Prometheus/Stackdriver integration using these <a href="https://cloud.google.com/monitoring/kubernetes-engine/prometheus#configuration">instructions</a></p><p>I greatly enjoyed learning how to use OpenTelemetry to create custom metric telemetry — using it to instrument my app to expose the “golden signals” was quite straightforward, thanks largely to the excellent examples provided in the OpenTelemetry Github repo. Thanks for reading — come back next time when I attempt to use OpenTelemetry to integrate tracing and logs in Stackdriver!</p><p><em>Originally published at </em><a href="https://gist.github.com/20e9affa4aaf2ba8adf413ca72ab5b76"><em>http://github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c10c8c0204d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/custom-metrics-in-node-js-with-opentelemetry-and-prometheus-c10c8c0204d3">Custom metrics in Node.js with OpenTelemetry (and Prometheus)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SLOs with Stackdriver Service Monitoring]]></title>
            <link>https://medium.com/google-cloud/slos-with-stackdriver-service-monitoring-62f193147b3f?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/62f193147b3f</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[stackdriver]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Wed, 22 Jan 2020 16:16:01 GMT</pubDate>
            <atom:updated>2020-01-22T16:16:01.356Z</atom:updated>
            <content:encoded><![CDATA[<p>Service Level Objectives or <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/">SLOs</a> are one of the fundamental principles of <a href="https://landing.google.com/sre/">site reliability engineering</a>. We use them to precisely quantify the reliability target we want to achieve in our service. We also use their inverse, <a href="https://landing.google.com/sre/sre-book/chapters/embracing-risk/#xref_risk-management_unreliability-budgets">error budgets</a>, to make informed decisions about how much risk we can take on at any given time. This lets us determine, for example, whether we can go ahead with a push to production or infrastructure upgrade.</p><p>However, Stackdriver has never given us the ability to actually create, track, alert, and report on SLOs — until now. The <a href="https://cloud.google.com/service-monitoring/">Service Monitoring</a> API was released to public beta at NEXT London in the fall, and I wanted to take the opportunity to try it out. Here’s what I found.</p><h3>The service</h3><p>Before I could create a service level objective, I needed a service. Because the initial release of the <a href="https://cloud.google.com/monitoring/service-monitoring/">API</a> only supports App Engine, Istio on GKE, and Cloud Endpoints, I thought I’d try the simplest option — App Engine Standard. I created a basic Hello World app in Go using the Mux router — here’s the <a href="https://github.com/yuriatgoogle/stack-doctor/tree/master/service-monitoring-demo/gae">code</a> for it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7202fbd1f59d273d0894b88555e4210c/href">https://medium.com/media/7202fbd1f59d273d0894b88555e4210c/href</a></iframe><p>I then created the app.yaml file:</p><pre>runtime: go112 # replace with go111 for Go 1.11</pre><p>Next, I deployed the app using `gcloud app deploy`. At this point, the build process was failing, and I ended up having to follow these <a href="https://github.com/golang/go/wiki/Modules#how-to-define-a-module">instructions</a> to define a module. I am not an expert in Go, and I’m guessing that this is a matter of local environment configuration that I just couldn’t be bothered to sort out. Nevertheless, I was able to deploy the app after following those instructions.</p><p>Finally, I set up a global Uptime Check to get a steady stream of traffic flowing to my new app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dgsxpZ7PATyexzKr" /></figure><h3>Defining the SLI</h3><p>Now that I had a service created, I was ready to proceed. I found the <a href="https://cloud.google.com/monitoring/service-monitoring/">documentation</a> on concepts very helpful, even as someone mostly familiar with this topic. For this exercise, I wanted to create a simple availability Service Level Indicator (SLI) to measure the percentage of “good” requests as a fraction of total. That required three decisions:</p><ul><li>How to count total requests</li><li>How to count “good” requests</li><li>What time frame to use for my SLI</li></ul><p>Thankfully, App Engine exposes a useful response count metric:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_x29foJPRnqGiC68" /></figure><p><strong>Note</strong> that this metric is not written if the GAE application is disabled (as I learned by attempting to simulate a failure by disabling the app).</p><p>This metric can further be filtered by response code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DvS7winl3LMrr-Z3" /></figure><p>I decided to use the unfiltered metric to count total requests and filter requests with a response code of 200 to count “good” requests for the sake of simplicity.</p><blockquote><strong>Note</strong> that this is likely far too simplistic for any production use. For example, this would count 404s as “bad” requests, when they are likely to be the result of misconfigured clients or even external scanners.</blockquote><h3>Creating the SLI and SLO</h3><p>At this point, I was ready to use the <a href="https://cloud.google.com/monitoring/service-monitoring/identifying-custom-sli">API</a> to define my SLO. As recommended in the “<a href="https://cloud.google.com/monitoring/service-monitoring/identifying-custom-sli#configure-sli">Building the SLI</a>” section, I used the Metrics Explorer to create a chart that showed my “total” request count:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YPWJv5wT82ut6LlQ" /></figure><p>From there, I was able to copy the JSON for the filter:</p><pre>&quot;metric.type=\&quot;appengine.googleapis.com/http/server/response_count\&quot; resource.type=\&quot;gae_app\&quot; resource.label.\&quot;module_id\&quot;=\&quot;default\&quot;&quot;</pre><p>I then modified the chart to only count the “good” requests, filtering on response_code=200 and copied that JSON:</p><pre>&quot;metric.type=\&quot;appengine.googleapis.com/http/server/response_count\&quot; resource.type=\&quot;gae_app\&quot; resource.label.\&quot;module_id\&quot;=\&quot;default\&quot; metric.label.\&quot;response_code\&quot;=\&quot;200\&quot;&quot;</pre><p>At this point, I was ready to build the SLI:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/056c9bcbf957d4c3007cfe78316d87d5/href">https://medium.com/media/056c9bcbf957d4c3007cfe78316d87d5/href</a></iframe><p>I chose the “requestBased” type of SLI, because I was looking to capture the fraction of good over total requests. The other options <a href="https://cloud.google.com/monitoring/service-monitoring/api-structures#sli-structs">include</a> <em>basic</em>, which might have been good enough for my purpose here, and “<em>window-based</em>”, which lets you count the number of periods during which the service meets a defined health threshold. I may come back and revisit the latter in another post.</p><p>From there, I defined the SLO:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f465b45912e9ea2debb818954a0da2d6/href">https://medium.com/media/f465b45912e9ea2debb818954a0da2d6/href</a></iframe><p>Finally, I submitted the request to the API using <a href="https://www.getpostman.com/">Postman</a> — you could do the same using the API Explorer or even curl. The response was successful and returned the SLO name in the body:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a89d43ae856f2f385824847ecff6efad/href">https://medium.com/media/a89d43ae856f2f385824847ecff6efad/href</a></iframe><h3>Alerting on SLO</h3><p>Now that my SLO was defined, I wanted to achieve two things — create an alert for SLO violation and figure out how to get a status without tripping an alert. I was able to use the UI to create an alerting policy using the “SLO BURN RATE” condition type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wOwOFr6oTWtFD_Ga" /></figure><p>When setting up the alerting policy, I ran into two fields whose meaning was not immediately clear to me. The first one is Lookback Duration. I was able to find an explanation in the <a href="https://cloud.google.com/monitoring/alerts/concepts-indepth#condition-types">documentation</a> — because burn rate is fundamentally a rate of change condition, you have the option of specifying a custom lookback window. For other rate of change conditions, the lookback is set to 10 minutes and cannot be changed. From the doc for rate of change conditions:</p><blockquote>The condition averages the values of the metric from the past 10 minutes, then compares the result with the 10-minute average that was measured just before the duration window. The 10-minute lookback window used by a metric rate of change condition is a fixed value; you can’t change it. However, you do specify the duration window when you create a condition.</blockquote><h3>Triggering alert</h3><p>Once my alerting policy was configured, I wanted to see what would happen if there was an availability issue. I rewrote the service to throw an error half the time:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e2238b82b66c462d9ef9bb410102192/href">https://medium.com/media/4e2238b82b66c462d9ef9bb410102192/href</a></iframe><p>and redeployed the app. Fairly quickly, I got an incident:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*odJAfJzA8bPCNqPl" /></figure><p>I was satisfied with this and redeployed the app with the original code to get it working again. In short order, the incident was resolved.</p><h3>Retrieving SLO Status</h3><p>Alerting on error budget burn is obviously necessary, but there will be times when we’ll need to know the status of our SLO long before an issue. As such, I needed a way to query the SLO data. I followed the <a href="https://cloud.google.com/monitoring/service-monitoring/timeseries-selectors">documentation</a> and used the<a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/list"> timeSeries.list </a>method of the monitoring API.</p><p>I thought of two primary questions I would want to be able to answer — what is the availability of my service over a given time period and what is the status of my error budget at a given point in time?</p><h3>SLO Status</h3><p>The first is answered using the “select_slo_health” time series. I sent a request to <a href="https://monitoring.googleapis.com/v3/projects/stack-doctor/timeSeries">https://monitoring.googleapis.com/v3/projects/&lt;project ID/timeSeries</a> with the following parameters:</p><pre>name:projects/&lt;project ID&gt;</pre><pre>filter:select_slo_health(&quot;projects/&lt;project number&gt;/services/gae:&lt;project ID&gt;_default/serviceLevelObjectives/&lt;SLO name&gt;&quot;)</pre><pre>interval.endTime:2020-01-06T17:17:00.0Z</pre><pre>interval.startTime:2020-01-05T17:17:00.0Z</pre><pre>aggregation.alignmentPeriod:3600s</pre><pre>aggregation.perSeriesAligner:ALIGN_MEAN</pre><p>I used the SLO name that was returned when I created the SLO in the previous steps. I could have also used a call to <a href="https://monitoring.googleapis.com/v3/projects/"><strong>https://monitoring.googleapis.com/v3/projects/</strong></a><strong>&lt;project ID&gt;/services/gae:&lt;project ID&gt;_default/serviceLevelObjectives </strong>to retrieve the SLOs I have configured against my default App Engine service.</p><p>I specified a 24hr interval to retrieve the data with an alignment of 1 hour using the mean aligner. If I was going to chart the data, I could have used a shorter alignment period and a more precise aligner, but this sufficed for my purposes. The results looked like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e9482d4c5113f0759c8d2c52652213d/href">https://medium.com/media/1e9482d4c5113f0759c8d2c52652213d/href</a></iframe><p>As expected, the output shows me the fractional ratio of good requests to total requests for each interval (that matches my alignmentPeriod) within the total interval (as specified by interval.startTime and endTime). For my service, each value was 1, meaning that 100% of the requests were good for each hourly interval.</p><h3>Error Budget Status</h3><p>The second question I wanted to answer is “how much error budget do I have left?” The operator for that is the select_slo_budget_fraction. The only change in the request is to change the filter:</p><pre>name:projects/&lt;project ID&gt;</pre><pre>filter:select_slo_budget_fraction(&quot;projects/&lt;project number&gt;/services/gae:&lt;project ID&gt;_default/serviceLevelObjectives/&lt;SLO name&gt;&quot;)</pre><pre>interval.endTime:2020-01-06T17:17:00.0Z</pre><pre>interval.startTime:2020-01-05T17:17:00.0Z</pre><pre>aggregation.alignmentPeriod:3600s</pre><pre>aggregation.perSeriesAligner:ALIGN_MEAN</pre><p>After making a request to the timeSeries.list method, I got the following return:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39161ea487e20e833105dd78843b26f7/href">https://medium.com/media/39161ea487e20e833105dd78843b26f7/href</a></iframe><p>As before, each “value” represents the fraction of the error budget remaining at that point. As my service is not burning error budget, the numbers stay at 1. I could have also used the select_slo_budget operator to get the actual remaining budget — the count of errors remaining.</p><h3>In conclusion…</h3><p>I hope you found this introduction to the Service Monitoring API useful. Thank you for reading, and let me know if you have any feedback. Until next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62f193147b3f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/slos-with-stackdriver-service-monitoring-62f193147b3f">SLOs with Stackdriver Service Monitoring</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Tracing with OpenTelemetry]]></title>
            <link>https://medium.com/google-cloud/distributed-tracing-with-opentelemetry-c0215ef38203?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0215ef38203</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Thu, 09 Jan 2020 22:51:24 GMT</pubDate>
            <atom:updated>2020-01-09T22:52:32.007Z</atom:updated>
            <content:encoded><![CDATA[<p>Toward the end of last year, I had the good fortune of publishing a reference <a href="https://cloud.google.com/solutions/using-distributed-tracing-to-observe-microservice-latency-with-opencensus-and-stackdriver-trace">guide</a> on using OpenCensus for distributed tracing. In it, I covered distributed tracing fundamentals, like traces, spans, and context propagation, and demonstrated using OpenCensus to instrument a simple pair of frontend/backend services written in Go. Since then, the OpenCensus and OpenTracing projects have merged into <a href="https://opentelemetry.io/">OpenTelemetry</a>, a “single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.” I wanted to attempt to reproduce the work I did in OpenCensus using the new project and see how much has changed.</p><h3>Objective</h3><p>For this exercise, I built a simple <a href="https://github.com/yuriatgoogle/stack-doctor/opentelemetry-tracing-demo">demo</a>. It consists of two services. The frontend service receives an incoming request and makes a request to the backend. The backend receives the request and returns a response. Our objective is to trace this interaction to determine the overall response latency and understand how the two services and the network connectivity between them contribute to the overall latency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*as-fF_4igPIFQApS.png" /></figure><p>In the original guide, the two services were deployed in two separate GKE clusters, but that is actually not necessary to demonstrate distributed tracing. For this exercise, we’ll simply run both services locally.</p><p>While the basic concepts are covered in the reference guide and in much greater detail in the Google Dapper research <a href="https://research.google.com/archive/papers/dapper-2010-1.pdf">paper</a>, it’s still worth briefly covering them here such that we can then understand how they’re implemented in the code.</p><p>From the reference guide:</p><blockquote><em>A </em>trace<em> is the total of information that describes how a distributed system responds to a user request. Traces are composed of </em>spans<em>, where each span represents a specific request and response pair involved in serving the user request. The </em>parent<em> span describes the latency as observed by the end user. Each of the </em>child<em> spans describes how a particular service in the distributed system was called and responded to, with latency information captured for each.</em></blockquote><p>This is well illustrated in the aforementioned research paper using this diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/0*CJdN9YG1o5BXsNuq.png" /></figure><p>Let’s take a look at how we can implement distributed tracing in our frontend/backend service pair using OpenTelemetry.</p><blockquote><strong><em>Note</em></strong><em> that most of this is adopted from the </em><a href="https://github.com/open-telemetry/opentelemetry-go/tree/master/example"><em>samples</em></a><em> published by OpenTelemetry in their Github </em><a href="https://github.com/open-telemetry/opentelemetry-go"><em>repo</em></a><em>. I made relatively minor changes to add custom spans and use the Mux router, rather than just basic HTTP handling.</em></blockquote><h3>Frontend code</h3><p>We’ll start by reviewing the frontend <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/opentelemetry-tracing-demo/go/frontend/frontend.go">code</a>.</p><h3>Imports</h3><p>First, the imports:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc5c9a6a9a0f17986f7e62d9f920eca6/href">https://medium.com/media/cc5c9a6a9a0f17986f7e62d9f920eca6/href</a></iframe><p>Mostly, we’re using a variety of OpenTelemetry libraries at this point. We’ll also use the <a href="https://github.com/gorilla/mux">Mux</a> router to handle HTTP requests (I mostly use it because it seem to be similar to Express in Node.js).</p><h3>Main function</h3><p>Next, let’s have a look at the main() function for our service:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e759a3e07bb4667af51288f513388ab/href">https://medium.com/media/1e759a3e07bb4667af51288f513388ab/href</a></iframe><p>As you can tell, this is pretty straighftorward. We initialize tracing right at the start and use a Mux router to handle a single route for requests to /. We then start the server on port 8080. I added an environment variable to check to see whether I&#39;m running the code locally to bypass the MacOS prompt to allow inbound network connections as per these <a href="https://medium.com/@leeprovoost/suppressing-accept-incoming-network-connections-warnings-on-osx-7665b33927ca">instructions</a>.</p><h3>Initialize tracing</h3><p>Next, let’s take a look at the initTracer() function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/faa86c10c190e628411f862ac803a030/href">https://medium.com/media/faa86c10c190e628411f862ac803a030/href</a></iframe><p>Here, we’re simply instantiating the Stackdriver exporter and setting the sampling parameter to capture every trace.</p><h3>Handle requests</h3><p>Finally, let’s look at the mainHandler() function that is called to handle requests to /.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a1ca179e49b925dd2d83f058bb35414/href">https://medium.com/media/4a1ca179e49b925dd2d83f058bb35414/href</a></iframe><p>Here, we’re setting the name of the tracer to “OT-tracing-demo” and starting a root span labeled “incoming call”. We then create a child span of that labeled “backend call” and pass the context to it. We then create a request to our backend server, whose location is defined in an env variable and inject our context into that request — we’ll see how that context is used in the backend in a bit. Finally, we make the request, get the status code, and output a confirmation message. Pretty straightforward!</p><p>A couple of things to note further:</p><ul><li>I am explicitly closing the child span, rather than using defer for more control over exactly when the timer is stopped.</li><li>I am adding events to spans for even more clear labeling.</li></ul><p>Now, let’s look at our <a href="https://github.com/yuriatgoogle/stack-doctor/blob/master/opentelemetry-tracing-demo/go/backend/backend.go">backend</a>.</p><h3>Backend code</h3><p>Much of the code here is very similar to the frontend — we use the same exact main() and initTracer() functions to run the server and initialize tracing.</p><h3>mainHandler</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27acb5b1febbd525fe1d1d704493fed3/href">https://medium.com/media/27acb5b1febbd525fe1d1d704493fed3/href</a></iframe><p>The mainHandler() function does look quite different. Here, we extract the span context from the incoming request, create a new request object using that context, and create a new span using that request context. We also add an event to our span for explicit labeling. Finally, we return &quot;OK&quot; to the caller and close our span. Again, I could have used defer span.End() instead of doing it explicitly.</p><blockquote><strong><em>Note</em></strong><em> the difference between span context and request context. This is specifically relevant when accepting incoming context and using it to create child spans. For further exploration of these two, take a look at the relevant </em><a href="https://opentracing.io/docs/best-practices/"><em>documentation</em></a><em> from OpenTracing.</em></blockquote><p>Now that we’ve seen how to implement tracing instrumentation in our code, let’s take a look at what this instrumentation creates. We can run both frontend and backend locally after setting the relevant environment variables for each and using gcloud auth login to log in to Google Cloud. Once we do that, we can hit the frontend on <a href="http://localhost:8080/">http://localhost:8080</a> and issue a few requests. This should immediately result in traces being written to Stackdriver:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xrfWskTjgndyMSOV.png" /></figure><p>You can see the span names we specified in our code and the events we added for clearer labeling. One additional thing I was pleasantly surprised by is that OpenTelemetry explicitly adds steps for the HTTP/networking stack, including DNS, connecting, and sending and receiving data.</p><p>I greatly enjoyed attempting to reproduce the work I did in OpenCensus with OpenTelemetry and eventually found it understandable and clear, especially once I was pointed to the tracer.Start() method to create child spans. Come back next time when I attempt to use the stats features of OpenTelemetry to create custom metrics. Until then!</p><p><em>Originally published at </em><a href="https://gist.github.com/39d92d18d1369cf2cc87b5ee5602d7c7"><em>http://github.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0215ef38203" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/distributed-tracing-with-opentelemetry-c0215ef38203">Distributed Tracing with OpenTelemetry</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Application Dashboard Creation for Services on GKE/Istio]]></title>
            <link>https://medium.com/google-cloud/automating-application-dashboard-creation-for-services-on-gke-istio-a55a5a79aa15?source=rss-84e227c7daf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a55a5a79aa15</guid>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Yuri Grinshteyn]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 07:16:01 GMT</pubDate>
            <atom:updated>2020-01-24T20:15:11.790Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>One of the more interesting concepts I’ve been hearing more and more about both from customers and from folks I respect and follow in the industry is the idea of “monitoring as code”. This is, of course, a subset of the “everything as code” movement, but it’s something that really resonates with me. Specifically, I have talked to lots of folks recently who are interested in automating the setup of the monitoring configuration when new services or new projects are rolled out. That generally includes two main things — alerting and dashboards. Automating the creation of alerting policies in Stackdriver has been available for some time through the use of the relevant <a href="https://cloud.google.com/monitoring/alerts/using-alerting-api">API</a> or <a href="https://www.terraform.io/docs/providers/google/r/monitoring_alert_policy.html">Terraform</a>. Now, the same automation is available for Stackdriver dashboards!</p><h3>Dashboards</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*ZtSss6lG3HOJp7zO" /><figcaption>Credit to <a href="https://twitter.com/ronsoak">https://twitter.com/ronsoak</a></figcaption></figure><p>It has been my long-held opinion that dashboards in general are over-emphasized in monitoring because they don’t really help with problem detection or problem resolution. The former is better addressed via well-defined SLOs and alerting, and the latter via good observability and ad-hoc querying capabilities. Nevertheless, dashboards are widely used by nearly everyone involved in service reliability and are a must-have capability in any monitoring setup. Stackdriver dashboards are very popular, and most of my customer conversations include helping folks visualize data in one way or another. One of the most common questions I do field is essentially “what should go in our dashboard?”</p><p>Thankfully, the answer to that has been well documented in the Monitoring Distributed Systems <a href="https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/">chapter</a> in the SRE <a href="https://landing.google.com/sre/sre-book/toc/">book</a>. If you have nothing else, start with the “golden signals” — traffic, errors, latency, and saturation.</p><p>We can take a basic set of services running in GKE and managed by the Istio service mesh as an example. I’ve previously written on the value of Istio for observability and monitoring and its integration with Stackdriver <a href="https://medium.com/google-cloud/istio-and-stackdriver-59d157282258">here</a>. Separately, I created a tutorial for building a dashboard showing “golden signal” data from such a system using Grafana <a href="https://cloud.google.com/community/tutorials/visualizing-metrics-with-grafana">here</a>. In general, we are after something that looks like this (<strong>note</strong> that I’m using the<a href="https://cloud.google.com/stackdriver/docs/release-notes#December_04_2019"> beta release</a> of Monitoring in the Cloud Console here):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Z68KozduXEld55z" /><figcaption>Services Dashboard in Stackdriver</figcaption></figure><p>Let’s take a look at how to create a dashboard focused on the “golden signals” of service health.</p><h3>Application Dashboard</h3><h3>Request Rates</h3><p>The first “golden signal” is “traffic” — essentially a measure of how much user activity the service is responding to. Thankfully, Istio provides this natively for every service in the mesh. The metric is called “server_request_count”, and the first chart in my dashboard, “Request Rates by Service”, uses that metric and groups the results by the “destination_service_name” label. Here’s its configuration in detail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cN1yvlfFI8Y_P9_-" /><figcaption>Request Rates Chart</figcaption></figure><h3>Errors</h3><p>The next chart, “Errors by Service” uses the same metric and grouping options; the only difference is that the data is then filtered to only count requests where the response code is not 200. This is a rough approach, as it does include 3xx redirects and 4xx errors, which are often the result of misconfigured or misbehaving clients, but it suffices to illustrate the point. Here’s the configuration for that chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fB8byR5a4nSHrdtg" /><figcaption>Errors Chart</figcaption></figure><h3>Latency</h3><p>The last chart again uses an Istio metric — this time, it’s “Server Response Latencies”, grouped by “destination service name” and using the 99th percentile aggregation. I am not filtering out errors for this one, though it might be a good idea. Here’s how it’s configured:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-4UBCZY8iHeFHB5D" /><figcaption>Latency Chart</figcaption></figure><h3>Automation</h3><p>Creating such a dashboard manually in Stackdriver is not difficult, assuming that the data is available, but it is <a href="https://landing.google.com/sre/sre-book/chapters/eliminating-toil/">toil</a> that should be automated if possible. With the beta release of the Dashboards API, it now is! Users can also use the API to, for example, copy dashboards between Workspaces or create additional standardization via automation. Let’s take a look at the details.</p><h3>API</h3><p>The API is documented <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards">here</a>, but the general idea is pretty straightforward — we simply call the projects.dashboards.create method passing a Dashboards object, which contains a name, description, and a set of Widget objects that specify the charts themselves.</p><h3>Structure</h3><p>Because Widgets are the basic building blocks we create a dashboard from, let’s start there. We already have our chart definitions, and it’s just a matter of converting them to the JSON representation <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards">expected</a> by the API. We start by defining the dashboard itself, which needs:</p><ol><li><strong>Name</strong>, formatted as “/projects/&lt;project ID or number&gt;/dashboards/&lt;ID&gt;”— for creation, we can leave this empty</li><li><strong>displayName</strong> — the name to be shown when the dashboard is actually created</li><li>A “<strong>root</strong>” object — the actual content to be displayed, which contains a Widget</li><li>Default <strong>options</strong> to be applied to the dashboard at load time</li></ol><p>The root is where we define the actual content of the dashboard. For our example, we need to create a dashboard with two columns in it and three charts. So the root of the dashboard is a <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#gridlayout">GridLayout</a> with two columns. It then needs to include three additional <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#widget">widgets</a>, each of which will specify an <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#xychart">xyChart</a>. Each chart needs to specify a <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#dataset">DataSet</a> object, which in turn uses a <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#timeseriesfilter">TimeSeriesFilter</a> to actually query the data.</p><h3>Implementation</h3><p>Let’s take a look at how we can actually implement our dashboard using this information.</p><h3>Request Rates</h3><p>Here’s the JSON definition for the Request Rates chart. I created this manually, but you can use the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards/create">API Explorer</a> to create your own baseline and simply populate it with parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNoj9Hu2PX47P_5eTWdP0A.png" /><figcaption>Request rates chart definition</figcaption></figure><p>Some things to note in the defintion:</p><ul><li>The <strong>filter</strong> specifies the resource and metric. This is also where we would specify, for example, the cluster name, service name, or other attributes if needed.</li><li>We’re using the <a href="https://cloud.google.com/blog/products/management-tools/stackdriver-tips-and-tricks-understanding-metrics-and-building-charts">default aligner and the sum reducer</a> to simply total up the requests and group them by the destination_service_name metric label.</li></ul><h3>Error Rates</h3><p>Next, we need to define our error rates chart. Here’s that definition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*aZMj05sbHZk7JtH0hytF7A.png" /><figcaption>Error rates chart definition</figcaption></figure><p>Note that this time we’re using an additional filter to only count requests where the response code is not 200. Otherwise, this definition is the same as the request counts chart.</p><h3>Latencies</h3><p>Finally, we define the latencies chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zmZkbZa6HH9yNULbMWKQcg.png" /><figcaption>Latency chart definition</figcaption></figure><p>We’re using the 99th percentile reducer AND filtering the data to just successful requests.</p><h3>Dashboard</h3><p>It’s time to put it all together — here’s the full dashboard <a href="https://gist.github.com/yuriatgoogle/c3b50c13f88b5fd776034f6879024533">definition</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fae224e5fec9f06d7554a6e155b5e68b/href">https://medium.com/media/fae224e5fec9f06d7554a6e155b5e68b/href</a></iframe><p>Note that we’re specifying a 4-column gridLayout, which will actually results in a 3-column dashboard. We also specify each chart as a widget in the layout. We submit this as a POST request body to <a href="https://monitoring.googleapis.com/v1/%7Bparent%7D/dashboards"><strong>https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/dashboard</strong></a><strong>s </strong>(as per the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards/create">documentation</a>) and get this dashboard as a result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Y3IwJoAMMY6g1Sw" /><figcaption>Services Dashboard created via API call</figcaption></figure><p>Voila! We now have a way to automate the creation of dashboards in our workspaces!</p><h3>In conclusion…</h3><p>I hope you find this useful and start creating and automating your own dashboards using the API based on this simple example. In my next post, I’m going to repeat this exercise and create a dashboard to visualize the health of my Kubernetes cluster that the services are running on. Until then — thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a55a5a79aa15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/automating-application-dashboard-creation-for-services-on-gke-istio-a55a5a79aa15">Automating Application Dashboard Creation for Services on GKE/Istio</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>