<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Charles on Medium]]></title>
        <description><![CDATA[Stories by Charles on Medium]]></description>
        <link>https://medium.com/@charles.baer?source=rss-91a2d71c8e2c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BTT2ZFdoCEz_s2GY.jpg</url>
            <title>Stories by Charles on Medium</title>
            <link>https://medium.com/@charles.baer?source=rss-91a2d71c8e2c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:43:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@charles.baer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 2: Building a Dashboard for a data processing pipeline with the Stackdriver Dashboard API]]></title>
            <link>https://medium.com/google-cloud/part-2-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-69e3260b5ec3?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/69e3260b5ec3</guid>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Thu, 26 Dec 2019 16:20:02 GMT</pubDate>
            <atom:updated>2019-12-26T16:20:02.891Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Dashboard for a data processing pipeline with the Stackdriver Dashboard API</h3><p>This is part 2 of a 2 part series.</p><ul><li><a href="https://medium.com/@charles.baer/part-1-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-3cb14a27cd59">Part 1</a>: covers identifying recommended Stackdriver Monitoring metrics to use for a data processing pipeline</li><li>Part 2: covers how to <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards">using the Stackdriver Dashboards API</a> to implement the charts and dashboards described in part 1 from a JSON template.</li></ul><h3>First, why automate?</h3><p>One of the common requests that I hear from the DevOps and SRE teams using Stackdriver Monitoring is that they want to automate dashboard deployment like they do with most infrastructure or environment set-up. It’s the popular the config-as-code approach to reduce human <a href="https://landing.google.com/sre/sre-book/chapters/eliminating-toil/">toil</a>! Another common request that I hear, this time from dashboard users, is that users want to be able to share the configuration for a dashboard that they have developed so that a colleague can use the dashboard as a template. The recently released <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards">Stackdriver Dashboards API</a> provides the solution to both of these use cases.</p><h3>Stackdriver Dashboards API</h3><p>The Stackdriver Monitoring API provides a resource called <em>projects.dashboards</em> which provides a familiar set of methods: <em>create</em>, <em>delete</em>, <em>get</em>, <em>list</em> and <em>patch</em>. These methods follow the REST semantics, accept JSON payloads and are consistent with other Google Cloud REST-based APIs.</p><p>The rest of this post requires an understanding of the details of how dashboards, charts and metrics are used in the Stackdriver Monitoring console. You can read the <a href="https://cloud.google.com/monitoring/charts/">Creating charts</a> section in the docs to find all the details.</p><h3>Building the dashboard JSON payload the easy way</h3><p>I mentioned that the Dashboards API provides both a <em>create</em> and a <em>get</em> method. Because building a dashboard in JSON from scratch requires detailed knowledge of the API data model and corresponding JSON syntax, a much easier approach is to build the dashboard in the Dashboards section of Stackdriver Monitoring and then use the API to expore the JSON representation of the dashboard. Once you have the JSON, you can then use the <em>create</em> method to create another dashboard based on the JSON. I strongly recommend this approach as an easy way to get started.</p><h3>The dashboard JSON payload</h3><p>In order for you to create a dashboard via the Dashboards API, you need the define several bits of JSON payload data for which you can use to call the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards/create">projects.dashboards.create</a> method. Here’s an example JSON payload:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/166321e833a7d796452297a48fcc0e5d/href">https://medium.com/media/166321e833a7d796452297a48fcc0e5d/href</a></iframe><ul><li>displayName — the human-readable name of the dashboard</li><li>gridLayout — the container for the widgets</li><li>widgets — the container for the chart items</li><li>dataSets — this includes the details used to gather the specific data in a timeSeriesFilter object including the metric name, metric filters and how the metric is aggregated.</li><li>xAxis, yAxis — definitions affecting the presentation of the axes</li><li>chartOptions — definitions affecting the mode of the chart</li></ul><h3>Using projects.dashboards.create</h3><p>There are many ways to call the projects.dashboards.create method. In this example, you can easily use the “Try this API” functionality directly in the API documentation to test out the API calls*.</p><p>*Please note that you should already have a Stackdriver Monitoring Workspace defined and the GCP project id for the project that contains the Workspace.</p><p>I have already created a <a href="https://gist.github.com/charlesbaer/d4f29e73684facc6c53d90ae9ba7f55f">JSON dashboard template</a> including all 6 of the charts described in <a href="https://medium.com/@charles.baer/part-1-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-3cb14a27cd59">part 1</a> of this series. You can use this JSON payload as a template for your own dashboard.</p><ol><li>To create the dashboard, click the blue “TRY IT!” button to open the “Try this API” feature on the right-hand side of the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards/create">projects.dashboards.create</a> method.</li><li>Enter a value for the parent input form in the pattern “projects/<em>YOUR_PROJECT_ID</em>” replacing your own GCP project id that contains the Stackdriver Workspace where you want to create the dashboard for the “<em>YOUR_PROJECT_ID</em>” string value.</li><li>Highlight the default values in the Request Body input form and then copy/paste the JSON dashboard template</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*du5caoQXcn6RMsIIPYHFKQ.png" /></figure><p>4. Click the “EXECUTE” button at the bottom of the page. If all goes well, you should see a green HTTP “200” response code along with the JSON description of the dashboard that you just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*m033x-2ntRhss1eCaxh2qw.png" /></figure><p>5. Open the <a href="https://console.cloud.google.com/monitoring/dashboards">Dashboards section</a> in the Stackdriver Monitoring console to review your newly created dashboard. Find the “Data Processing Dashboard Template” and click the name to open the dashboard.</p><p>If you have already deployed Pub/Sub, Dataflow and BigQuery resources, you should already see values in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHnZpALu8YpPUDct7JcHmw.png" /></figure><h3>Practical use cases for the dashboards template</h3><p>I started this series with 2 common use cases:</p><ol><li>DevOps/SRE team members creating or updating dashboards</li><li>users sharing dashboard templates</li></ol><p>So, how can you use the Dashboards API for the use cases?</p><p>In the first case, dashboard JSON templates can be constructed, by exporting an existing dashboard from the Stackdriver Monitoring console, and then the JSON may be checked into a source repository like github or Cloud Source Repos. From there, you can deploy the dashboard by calling the Dashboards API as a part of your standard deployment process by pulling the JSON template file from the repo.</p><p>In the second case, you can create a dashboard from an existing dashboard from the Stackdriver Monitoring console and then use the Dashoards API to export the JSON template. You can then share that template either via repo or however you normally share files with your colleagues.</p><p>In both cases, having a pre-built template dashboard to start should make it easier to build and share Stackdriver Monitoring dashboards.</p><h3>Conclusion</h3><p>In this series, I have detailed a reasonable set of metrics and charts for a data processing pipeline and then described a detailed approach for deploying the dashboard template using the Dashboards API.</p><h3>Useful resources</h3><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/docs/apis">Stackdriver Monitoring API</a></li><li><a href="https://landing.google.com/sre/books/">Site Reliability Engineering books</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69e3260b5ec3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/part-2-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-69e3260b5ec3">Part 2: Building a Dashboard for a data processing pipeline with the Stackdriver Dashboard API</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: Building a Dashboard for a data processing pipeline with the Stackdriver Dashboard API]]></title>
            <link>https://medium.com/google-cloud/part-1-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-3cb14a27cd59?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cb14a27cd59</guid>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Thu, 26 Dec 2019 16:18:57 GMT</pubDate>
            <atom:updated>2019-12-26T16:18:57.407Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Dashboard for a data processing pipeline using the Stackdriver Dashboard API</h3><p>This is part 1 of a 2 part series.</p><ul><li>Part 1: covers identifying recommended Stackdriver Monitoring metrics to use for a data processing pipeline</li><li><a href="https://medium.com/@charles.baer/part-2-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-69e3260b5ec3">Part 2</a>: covers how to <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards">using the Stackdriver Dashboards API</a> to implement the charts and dashboards described in part 1 from a JSON template.</li></ul><h3>What should go in the Dashboard</h3><p>I spend a lot of time talking to DevOps and SRE teams that are using or are considering how to use Stackdriver. One of the consistent questions that I receive is guidance on what metrics should be monitored.</p><p>The best overall guidance that I’ve seen on this topic comes from the Site Reliability Engineering book under <a href="https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/">Chapter 6 — Monitoring Distributed Systems</a>. In the chapter, they discuss the “Four Golden Signals” which you should consider monitoring in your system.</p><p>The Four Golden Signals are:</p><ol><li><strong>Latency</strong> — the time it takes for your service to fulfill a request</li><li><strong>Traffic</strong> — how much demand is directed at your service</li><li><strong>Errors</strong> — the rate at which your service fails</li><li><strong>Saturation</strong> — a measure of how close to fully utilized for the service’s resources</li></ol><p>You can use these monitoring categories when considering the important metrics to monitor in your system or even data processing pipeline. For the purposes of monitoring, you can treat the pipeline as the “service” to be monitored. This means that you can consider the metrics in each product component and how those metrics can be applied to the “Four Golden Signals”. In the remainder of this post, I cover how you can map metrics to charts for an example data processing pipeline deployed in GCP.</p><h3>The sample data processing pipeline</h3><p>I extended a sample data processing pipeline based on this <a href="https://github.com/GoogleCloudPlatform/gcf-gce-usage-monitoring">reference guide</a> “<em>Monitoring a Compute Engine footprint with Cloud Functions and Stackdriver</em>” adding a simple Cloud Dataflow template for <a href="https://cloud.google.com/dataflow/docs/guides/templates/provided-streaming#pubsub-topic-to-bigquery">PubSub topic to BigQuery</a>. The architecture described in the reference guide builds an inventory of Compute Engine instances and then writes the results to Stackdriver Monitoring.</p><p>In this post, the Cloud Functions are simply a data source for our Pub/Sub, Cloud Dataflow and BigQuery data processing pipeline. The architecture is comprised of a series of Cloud Functions connected via Pub/Sub which write the results to Stackdriver Monitoring. The addition of the Cloud Dataflow component allowed me to write the results to BigQuery in addition to Stackdriver Monitoring for the purposes of this pipeline example.</p><p>For the purposes of this example, I am focusing on the metrics to monitor the Pub/Sub entry point, Cloud Dataflow and BigQuery components which are highlighted in the red boxes in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8TdKtK3zY5Ka0eWAs2uIg.png" /></figure><p>You can generalize this pipeline to the following steps:</p><ol><li>Send metric data to a Pub/Sub topic</li><li>Receive data from a Pub/Sub subscription in Cloud Dataflow</li><li>Write the results to BigQuery</li></ol><p>With the “Four Golden Signals” monitoring framework and the simple data processing pipeline, I can make recommendations about specific metrics and which of the monitoring signals should be monitored in the next sections.</p><p>The screenshot below shows the 6 different charts on a dashboard which provides coverage for the data processing pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0eCfrdWzJi8fumDJkFW9zQ.png" /></figure><h3>Traffic</h3><p>Traffic represents how many requests are being serviced over a given time. A common way to measure traffic is requests/second. I chose to build 3 different charts for the 3 technologies in the data processing pipeline architecture (Pub/Sub, Cloud Dataflow and BigQuery) to make it easier to read because the y axis scales turned out to be orders of magnitude different for each metric. You may choose to include them on a single chart for simplicity.</p><h3>Dataflow traffic chart</h3><p>Stackdriver Monitoring provides many different metrics for Cloud Dataflow which you can find in the metrics <a href="https://cloud.google.com/monitoring/api/metrics_gcp#gcp-dataflow">documentation</a>. Broadly, they are categorized into overall job metrics like job/status or job/total_vcpu_time and processing metrics like job/element_countand job/estimated_byte_count.</p><p>Since we’re looking to monitor the traffic through Cloud Dataflow, the job/element_countwhich represents “<em>The number of elements added to the pcollection so far</em>” aligns well with measuring the amount of traffic. Importantly, the metric will increase with an increase in the volume of traffic. Thus, it’s a reasonable metric to use to understand the traffic coming into a pipeline.</p><p>The screenshot below captures the Cloud Dataflow traffic chart in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*RmAWLUbMbteUcgC_SR0EJw.png" /></figure><h3>Pub/Sub traffic chart</h3><p>Stackdriver Monitoring metrics for Pub/Sub are categorized into topic, subscription and snapshot metrics. Both subscription and topic metrics may be used to chart the traffic since they represent both sides of a messages published to Pub/Sub.</p><p>Since I want to see the amount of incoming traffic, looking at the metrics around the inbound topics that receive the data is a reasonable choice. Specifically, the topic/send_request_countwhich represents the “<em>Cumulative count of publish requests, grouped by result” </em>aligns well with measuring the amount of traffic.</p><p>The screenshot below captures the Pub/Sub traffic chart in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*YlSefRYoDpbOF2T-UaOuDg.png" /></figure><h3>BigQuery traffic chart</h3><p>Stackdriver Monitoring metrics for BigQuery are categorized into bigquery_project, bigquery_dataset and query metrics.</p><p>Since I would like to see the amount of incoming traffic, looking at the metrics related to uploaded data is a reasonable choice. Specifically, the storage/uploaded_bytesaligns well with measuring incoming traffic to BigQuery.</p><p>The screenshot below captures the BigQuery traffic chart in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*354Gr-B29N9S7yuaReQ5Qg.png" /></figure><h3>Latency</h3><p>Latency represents how long it takes to service a request over a given time. A common way to measure latency is time required to service a request in seconds. In this sample architecture with Pub/Sub, BigQuery and Cloud Dataflow, the metrics that may be useful to understand latency may indicate how long it takes to go through the Cloud Dataflow or steps in the Cloud Dataflow, how long a message is unacknowledged in Pub/Sub and how long it takes to insert records into BigQuery.</p><h3>System lag chart</h3><p>Since I’d like to see the amount of time that it takes to service requests, looking at the metrics related to processing time and lag area reasonable choices. Specifically, the job/data_watermark_age which represents “<em>The age (time since event timestamp) of the most recent item of data that has been fully processed by the pipeline</em>” and the job/system_lag which represents “<em>The current maximum duration that an item of data has been awaiting processing, in seconds</em>” aligns well with measuring the time taken to be processed through the Cloud Dataflow pipeline.</p><p>The screenshot below captures the Cloud Dataflow system lag chart in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*0E8NxzM7C6fIzzNTZbT3aQ.png" /></figure><h3>Saturation</h3><p>Saturation represents how utilized the resources are that run your service. Saturation is meant to monitor a metric that show when the system may begin being constrained. In this sample architecture with Pub/Sub, BigQuery and Cloud Dataflow, the metrics that may be useful to understand saturation are the oldest unacknowledged messages (if processing slows down, then the messages will remain in Pub/Sub longer) and in Cloud Dataflow, the watermark age of the data (if processing slows down, then messages will take longer to get through the pipeline).</p><h3>Saturation chart</h3><p>Since I’d like to see when the service is approaching provisioned capacity, one assumption that I can make is that the time to process a given message will slow down as the system approaches fully utilizing its resources. This may not always be the case for data processing pipelines in general. Since I am processing asynchronously with Pub/Sub and Cloud Dataflow, this assumption is a reasonable one.</p><p>Specifically, the job/data_watermark_age which we used above and the topic/oldest_unacked_message_age_by_region which represents “<em>Age (in seconds) of the oldest unacknowledged message in a topic</em>” aligns well with measuring the increases in Cloud Dataflow processing time and time for the pipeline to receive/acknowledge input messages from Pub/Sub.</p><p>The screenshot below captures the Saturation chart for Pub/Sub and Cloud Dataflow in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*4uV8SV75Hbapbnl9fyYiXg.png" /></figure><h3>Errors</h3><p>Errors represents application errors, infrastructure errors or failure rates. The point here is to monitor a metric that shows an increased error rate when errors are encountered. In this sample architecture with Pub/Sub, BigQuery and Cloud Dataflow, the metrics that may be useful to understand saturation are the errors reported in the logs for Pub/Sub, Cloud Dataflow and BigQuery.</p><h3>Data processing pipeline errors chart</h3><p>Since I’d like to see the error rated for the service, I can look at the errors that are reported in the logs for the services included in the architecture.</p><p>Specifically, the log_entry_count which represents the &quot;<em>Number of log entries</em>” specific for each of the 3 services aligns well with measuring the increases in the number of errors.</p><p>The screenshot below captures the Errors chart for Pub/Sub, Cloud Dataflow and BigQuery in the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*IKgL96LJXeaC9m8J9yF-TA.png" /></figure><h3>Using the dashboard</h3><p>In this post, I have described an approach for selecting metrics for a data processing pipeline based on the “Four Golden Signals”. You can easily build this dashboard yourself by hand in the Dashboards section of Stackdriver Monitoring console. However, an even better approach is to use a dashboard template. Read <a href="https://medium.com/@charles.baer/part-2-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-69e3260b5ec3">part 2</a> of this series to learn how to deploy this dashboard from a JSON template.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cb14a27cd59" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/part-1-building-a-dashboard-for-a-data-processing-pipeline-with-the-stackdriver-dashboard-api-3cb14a27cd59">Part 1: Building a Dashboard for a data processing pipeline with the Stackdriver Dashboard API</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAS/COM: An architects guide to system design]]></title>
            <link>https://medium.com/google-cloud/sas-com-an-architects-guide-to-system-design-9830e3623964?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9830e3623964</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Mon, 20 May 2019 20:30:59 GMT</pubDate>
            <atom:updated>2019-05-20T20:42:50.740Z</atom:updated>
            <content:encoded><![CDATA[<h3>SAS/COM: An architect’s guide to system design</h3><p>I spend a lot of time thinking through different architecture designs and some time ago I developed a set of design principles that I call SAS/COM, for short. It’s an acronym for <em>Scalability</em>, <em>Availability</em>, <em>Security</em>, <em>Cost</em>, <em>Operations</em> and <em>Maintenance</em>. I’ll make the claim that any architecture has to consider these principles during the design. In this post, I’ll describe why each one of these criteria is important. Since I spend most of my productive time working with Google Cloud, I’ll share examples related to that product set.</p><h3><strong>Scalability</strong></h3><p>Understanding the volume, velocity and variety of user traffic helps you understand the sheer size of the problem. It’s a a simple place to start your design.</p><p>Knowing the number of queries per second (qps), latency requirements, variability in usage patterns, size of the requests and geographical origins all help understand how to structure an app and infrastructure. On top of that, understanding how these requirements may change over time is also a critical to design a system that continues to work as traffic scales. These requirements impact the selection of the platform, caching requirements, storage technologies and how to design an overall app architecture.</p><p>For example, if you have highly variable traffic, then a PaaS product like App Engine that can automatically scale up and down based on traffic is a good consideration. In fact, going deeper, you may want to leverage the pre-warming features in App Engine to provide the fastest ability to scale to meet demand. For low latency apps, you’ll want to cache information both at the browser and app and leverage a CDN like CDN for static content. For storing app state, you’ll want to choose a low-latency database for app storage. You’ll also want to think about separating the app into a frontend and a backend and building the service using microservices or RPCs. Compute options for microservices on GCP range from Google Kubernetes Engine to App Engine or even Cloud Functions. For apps with global user bases, you’ll want to consider locating your frontend as close to the user as possible to reduce latency while ensuring the frontend/backend services communication is robust. Google Cloud global load balancer provides an easy way to efficiently route user traffic.</p><h3><strong>Availability</strong></h3><p>Why does availability matter? It matters because you’ll need to design your system to offer the right availability. The Site Reliability Engineers (SREs) have a lot to say in their <a href="https://landing.google.com/sre/books/">books</a> and <a href="https://cloud.google.com/blog/products/cre">blog series</a> about designing scalable systems with the appropriate level of availability and are worth a read if you haven’t already. Understanding the availability requirements allows you to design a system with the appropriate amount of uptime, but not more. Increasing uptime can be costly and so, it’s important to understand the availability requirements from the business. A high traffic, global web site or service has very different availability requirements than your internal analytics pipeline. Both are important, but the public website/service may have higher availability requirements than your internal analytics pipeline.</p><p>When you develop an app, it’s a very good idea to formally capture the availability requirements by defining a <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/">Service Level Objective </a>(SLO) even if you don’t have a formal Service Level Agreement (SLA) with users or customers. SLOs help drive the design of the app by requiring use of storage with specific availability or performance characteristics. This also helps to define the plan for monitoring the app which will be important once the app is operational.</p><p>Cloud service providers, including Google Cloud, offer SLAs for their services. Building your app on top of cloud infrastructure means that you need to consider the underlying cloud infrastructure SLAs.</p><p>For highly available systems, you need to select a compute, storage and network service design that will provide high availability by providing redundancy. For example, you’ll want to include features such as Google Cloud’s global load-balancers, multi-regional Cloud Storage and redundant Cloud Interconnects for any cloud-to-on-prem communication. These compute, storage and network options each offer redundancy. Using a load-balancer means that you can spread the traffic load across a group of instances or a set of services on a GKE cluster. Using the multi-region option for Cloud Storage or Cloud Datastore means that your content is automatically replicated into multiple locations which provides a higher availability guarantee. Building redundancy into a cloud-to-on-prem communication channel can be done by setting up 2 Cloud Interconnects to your on-prem. All of these design choices are driven by the availability requirements.</p><h3><strong>Security</strong></h3><p>Security and privacy are a critical components of any system design regardless of where the system is deployed. You may have very specific requirements based on your organization’s policies, compliance or regulatory requirements. Keep in mind that security is a very broad term that can include very broad requirements such as “all data must be encrypted at rest and in transit with keys that you control” or very specific requirements such as “all TLS connections must use the TLS_RSA_WITH_AES_256_CBC_SHA SSLv3 cipher”. Or you may have policies that govern how PII data is used in your apps. Your organization may also have a team that certifies cloud services for usage which can be an important consideration for app design. Taken as a whole, security and privacy are broad topics and deeply technical considerations that you’ll need to consider during app design.</p><p>For example, one common design requirement is that Kubernetes clusters or Compute Engine VM instances should not be exposed through public IPs. In Google Cloud, this policy is implemented via an <a href="https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints">organizational policy constraint</a>. Another common requirement is to connect to on-prem networks with private IP space. In Google Cloud using VPC Service Controls, Private Google Access and Cloud Interconnect together allow you to continue using private IP space while making use of Google Cloud services. Requirements related to data storage regions services also may impact the way that you choose to architect the app. As an example, many of the services in Google Cloud are <a href="https://cloud.google.com/docs/geography-and-regions#regions_and_zones">zonal and regional services</a> which allow you to be specific about where the app is running and stores data. Other apps may require compliance certifications including BAA-covered services. For example, in Google Cloud, you can select from the broad list of <a href="https://cloud.google.com/security/compliance/hipaa-compliance/">BAA-covered services</a> while designing your app. Understanding these and all security-related requirements can dramatically influence the design of your infrastructure and app.</p><h3><strong>Cost</strong></h3><p>The efficiency of a design can have a material impact on the development and cost of an app. Inefficient designs may include inefficient database calls, unnecessary service calls and non-optimal caching each of which may incur additional cost. Since most cloud storage services are billed as usage-based, the more services that you use, the more that you pay. This can definitely offer advantages for workloads that scale up and down over time though it can also make the overall cost more difficult to estimate the cost before you build your app.</p><p>Some important items to consider are the differences in costs between different storage options, egress costs across regions and compute costs. Other items to consider are per-operation costs of individual components such as load-balancer costs, storage operations and logging volumes. In fact, cost is one of the reasons that serverless technologies are increasingly popular.</p><p>For example, if you deploy a microservices app that makes service calls across a Google Cloud region, you will incur egress traffic charges. Picking the storage class can also have a dramatic impact on the cost depending on the current and projected future volumes. As an example, US multiregional coldline storage costs $0.007/GB while the US multiregional standard storage costs $0.026 (as of current pricing today <a href="https://cloud.google.com/storage/pricing">here</a>). Thats a 3X difference in costs and can become material quickly as storage sizes grow over time. As a last example, picking an instance or GKE cluster has a different cost than using a serverless option such as Cloud Run for workloads that may scale up and down. Considering the cost is a critical component of your system design.</p><h3><strong>Operations</strong></h3><p>Observability is key design consideration for your app. This area focuses on running the app which is commonly done by the DevOps team or, as this function is known at Google, the <a href="https://landing.google.com/sre/">Site Reliability Engineering</a> (SRE) team. Thinking about important metrics such as Service Level Indicators (SLIs) and your app’s Service Level Objectives (SLOs) during the design phase can help you include the required instrumentation in your design. The SRE /DevOps team will be very interested in how your app can be monitored and what the dashboard levels mean for the identification and resolution of errors.</p><p>For example, Stackdriver Monitoring provides infrastructure and app-level monitoring for Google Cloud services. In Stackdriver, you can use the provided GCP infrastructure metrics or write your own custom metrics to track your app’s Service Level Indicators. In fact, for App Engine and Istio-enabled apps, Stackdriver’s Service Monitoring will automatically track your SLIs and report against your defined SLOs. Stackdriver alerting and dashboards provide SREs detailed insights into specific metrics of interest for your app.</p><p>For debugging and troubleshooting, it’s important to consider the logging strategy and the use of application performance management (APM) tools. The use of APM tools usually require some instrumentation at the code level and should be considered as a part of the software development lifecycle.</p><p>For example, Stackdriver’s APM tools include Profiler, Debugger and Trace and are integrated with OpenCensus. Each provide useful information for the SRE or development team to use when troubleshooting or analyzing an app’s performance.</p><h3><strong>Maintenance</strong></h3><p>The design of an app can have a very significant impact on the deployment time/cost. Developing a design that meets the current requirements, can scale with differing traffic loads and that is easy to modify in the future as new requirements surface, make the app easier to maintain over time.</p><p>For example, if you are using the Bigtable for your app storage, then carefully designing the key will mean that the app meets current requirements, will scale with higher traffic volumes and will be easy to change as the app requirements evolve over time.</p><p>A common way to help maintain development velocity while reducing deployment time/cost is to use CI/CD tooling. Using CI/CD tooling means that the development teams can design their tests and apps to use a standard process which also makes the apps more maintainable because changes are easier to deploy. Getting code to production faster, with less errors and with less human time makes developers more productive and reduces costs.</p><p>For example, you may set-up a CI/CD pipeline using <a href="https://cloud.google.com/solutions/jenkins-on-kubernetes-engine">Jenkins with Google Kubernetes Engine</a>. There are many different ways to set-up a pipeline, but the end result should provide an automated way to build, test and deploy your code. From a design perspective, it’s important to think about your code being released through the CI/CD pipeline and develop automated testing to take advantage of the automation.</p><h3><strong>Conclusion</strong></h3><p>Evaluating the requirements against a rubric of <em>Scalability</em>, <em>Availability</em>, <em>Security</em>, <em>Cost</em>, <em>Operations</em> and <em>Maintenance </em>helps design systems that scale appropriately, meet user SLOs, protect sensitive data, optimize cost, are well instrumented and take advantage of deployment automation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9830e3623964" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/sas-com-an-architects-guide-to-system-design-9830e3623964">SAS/COM: An architects guide to system design</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackdriver Monitoring Automation Part 6: Uptime Checks with Terraform]]></title>
            <link>https://medium.com/google-cloud/stackdriver-monitoring-automation-part-6-uptime-checks-with-terraform-76cb25b996a4?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/76cb25b996a4</guid>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Tue, 05 Mar 2019 16:08:26 GMT</pubDate>
            <atom:updated>2019-09-11T17:28:12.577Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is part 6 in the Stackdriver Automation series. In this post, I will walk through the steps that you can use to automate the management of Stackdriver Monitoring Uptime Checks with Terraform.</p><h3>Uptime Checks</h3><p><a href="https://cloud.google.com/monitoring/uptime-checks/">Uptime Checks</a> lets you verify the availability of your service via HTTP, HTTPS or TCP health checks. Stackdriver provides this service by accessing your application’s frontend from locations around the world and reporting on the results such as latency and response codes. You can use this service as a way to understand whether your users can access the service from various global locations.</p><h3>Uptime Checks Configuration</h3><p>As an example, I created an Uptime Check based on the apache infrastructure that I described in parts 1 and 2 of this series. The application itself is a simple website being hosted by apache servers behind a load-balancer. I created a simple HTTP Uptime Check to understand whether my users could access the service.</p><pre>provider &quot;google&quot; {<br>  project = &quot;abab-cdcd-023991&quot;<br>  region  = &quot;us-central1&quot;<br>  zone    = &quot;us-central1-c&quot;<br>}</pre><pre>resource &quot;google_monitoring_uptime_check_config&quot; &quot;http&quot; {<br>  display_name = &quot;1 - Website uptime check [global]&quot;<br>  timeout = &quot;10s&quot;<br>  period = &quot;60s&quot;</pre><pre>  http_check {<br>    path = &quot;/&quot;<br>    port = &quot;80&quot;<br>  }</pre><pre>  monitored_resource {<br>    type = &quot;uptime_url&quot;<br>    labels = {<br>      host = &quot;35.241.47.194&quot;<br>    }<br>  }</pre><pre>}</pre><p>You can find the full configuration files on the <a href="https://github.com/charlesbaer/stackdriver-automation">github repo</a>.</p><p>After the standard initialization, the terraform apply does all the work and creates the Uptime Check.</p><pre>$ terraform apply</pre><pre>An execution plan has been generated and is shown below.<br>Resource actions are indicated with the following symbols:<br>  + create</pre><pre>Terraform will perform the following actions:</pre><pre>+ google_monitoring_uptime_check_config.http<br>      id:                               &lt;computed&gt;<br>      display_name:                     &quot;01 - Website uptime check [global]&quot;<br>      http_check.#:                     &quot;1&quot;<br>      http_check.0.path:                &quot;/&quot;<br>      http_check.0.port:                &quot;80&quot;<br>      monitored_resource.#:             &quot;1&quot;<br>      monitored_resource.0.labels.%:    &quot;1&quot;<br>      monitored_resource.0.labels.host: &quot;35.241.47.194&quot;<br>      monitored_resource.0.type:        &quot;uptime_url&quot;<br>      name:                             &lt;computed&gt;<br>      period:                           &quot;60s&quot;<br>      project:                          &lt;computed&gt;<br>      timeout:                          &quot;10s&quot;</pre><pre>Plan: 1 to add, 0 to change, 0 to destroy.</pre><pre>Do you want to perform these actions?<br>  Terraform will perform the actions described above.<br>  Only &#39;yes&#39; will be accepted to approve.</pre><pre>Enter a value: yes</pre><pre>google_monitoring_uptime_check_config.http: Creating...<br>  display_name:                     &quot;&quot; =&gt; &quot;01 - Website uptime check [global]&quot;<br>  http_check.#:                     &quot;&quot; =&gt; &quot;1&quot;<br>  http_check.0.path:                &quot;&quot; =&gt; &quot;/&quot;<br>  http_check.0.port:                &quot;&quot; =&gt; &quot;80&quot;<br>  monitored_resource.#:             &quot;&quot; =&gt; &quot;1&quot;<br>  monitored_resource.0.labels.%:    &quot;&quot; =&gt; &quot;1&quot;<br>  monitored_resource.0.labels.host: &quot;&quot; =&gt; &quot;35.241.47.194&quot;<br>  monitored_resource.0.type:        &quot;&quot; =&gt; &quot;uptime_url&quot;<br>  name:                             &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>  period:                           &quot;&quot; =&gt; &quot;60s&quot;<br>  project:                          &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>  timeout:                          &quot;&quot; =&gt; &quot;10s&quot;<br>google_monitoring_uptime_check_config.http: Creation complete after 1s (ID: projects/abab-cdcd-023991/uptimeCheckConfigs/01-website-uptime-check-global)</pre><pre>Apply complete! Resources: 1 added, 0 changed, 0 destroyed.</pre><p>Once the uptime checks were created, I used the Stackdriver Monitoring console to verify that the uptime checks had been successfully created. Keep in mind that when you first create uptime checks, it can take up to 5 mins to start reporting status according to the <a href="https://cloud.google.com/monitoring/uptime-checks/">docs</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qJqwwsYIiExz76UluN4xg.png" /></figure><p>Now, the uptime checks are active and can be used in conjunction with alerting policies to notify you when the uptime checks fail or when the latency is outside of tolerance.</p><h3>Conclusion: The Stackdriver Monitoring Automation Series</h3><p>This concludes the Stackdriver Monitoring Automation series. In this series, I have included the steps that I took and the methods that I used to create Stackdriver Monitoring components via Google Cloud Deployment Manager and Terraform. In the posts 1 and 4, I created a Stackdriver Group used to group resources to monitor as a single entity. In posts 2 and 5, I created Stackdriver Alerting Policies to define when to send an alert and what alert to send. In posts 3 and 6, I created Stackdriver Uptime Checks to provide a basic picture of end-user experience. You can use these steps and config files as templates to automate the deployment of Stackdriver Monitoring resources in your environment.</p><p>Read more about Stackdriver Monitoring Automation in the other posts in the series and references below.</p><ul><li><a href="https://medium.com/google-cloud/stackdriver-automation-part-1-stackdriver-groups-8e51f0aa9b03">Stackdriver Monitoring Automation Part 1: Stackdriver Groups</a></li><li><a href="https://medium.com/@charles.baer/stackdriver-automation-part-2-alerting-policies-9f42068603c4">Stackdriver Monitoring Automation Part 2: Stackdriver Alerting Policies</a></li><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-3-uptime-checks-476b8507f59c">Stackdriver Monitoring Automation Part 3: Uptime Checks</a></li><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-4-stackdriver-groups-with-terraform-910289d16d08">Stackdriver Monitoring Automation Part 4: Stackdriver Groups with Terraform</a></li><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-5-alerting-policies-ff77b19b4b97">Stackdriver Monitoring Automation Part 5: Stackdriver Alerting Policies with Terraform</a></li></ul><p>References:</p><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">Stackdriver Monitoring API docs</a></li><li><a href="https://cloud.google.com/monitoring/api/v3/filters">Stackdriver Monitoring filters</a></li><li><a href="https://cloud.google.com/monitoring/api/metrics_gcp">Stackdriver Monitoring metrics</a></li><li><a href="https://www.terraform.io/docs/providers/google/provider_reference.html">Terraform Google Provider</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76cb25b996a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-6-uptime-checks-with-terraform-76cb25b996a4">Stackdriver Monitoring Automation Part 6: Uptime Checks with Terraform</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackdriver Monitoring Automation Part 5: Alerting Policies]]></title>
            <link>https://medium.com/google-cloud/stackdriver-monitoring-automation-part-5-alerting-policies-ff77b19b4b97?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff77b19b4b97</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Tue, 05 Mar 2019 16:08:05 GMT</pubDate>
            <atom:updated>2019-03-05T16:44:31.791Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is part 5 in the Stackdriver Automation series. In this post, I will walk through the steps that you can use to automate the management of Stackdriver Monitoring Alerting Policies with Terraform.</p><h3>SLI Alerting Metrics</h3><p>Head over to <a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-2-alerting-policies-9f42068603c4">part 2</a> for the details of the alerting scenarios implemented with these Alerting Policies. To recap, the main Stackdriver Alerting conditions, notifications and documentation that I selected were the following:</p><p><strong>Conditions</strong></p><p><em>For Availability &amp; Quality:</em></p><ul><li>L7 load balancer error request count = 0/sec</li></ul><p><em>For Latency:</em></p><ul><li>L7 Load balancer total latency &gt; 100ms</li><li>L7 load balancer Frontend RTT latency &gt; 50ms</li><li>L7 load balancer Backend RTT latency &gt; 50ms</li></ul><p><em>For Volume:</em></p><ul><li>L7 load balancer request count &gt; 100/sec</li></ul><p><strong>Notification channels</strong></p><ul><li>Email: <a href="/service/mailto:apache-website-oncall@example.com">website-oncall@example.com</a></li><li>Email: support-website@example.com</li></ul><p><strong>Documentation</strong></p><ul><li>Specific text describing each alerting policy</li></ul><h3>Notification Channels Configuration</h3><p>The 5 alerting conditions described above translated into 5 alerting policies and 2 separate notification channels. First, I created the notification channels configuration. Don’t forget to change your project values if you run in your own project.</p><pre>provider &quot;google&quot; {<br>  project = &quot;abab-cdcd-023991&quot;<br>  region  = &quot;us-central1&quot;<br>  zone    = &quot;us-central1-c&quot;<br>}</pre><pre>resource &quot;google_monitoring_notification_channel&quot; &quot;email0&quot; {<br>  display_name = &quot;Website Oncall&quot;<br>  type = &quot;email&quot;<br>  labels = {<br>    email_address = &quot;<a href="/service/mailto:website-oncall@example.com">website-oncall@example.com</a>&quot;<br>  }<br>}</pre><pre>resource &quot;google_monitoring_notification_channel&quot; &quot;email1&quot; {<br>  display_name = &quot;Website Support&quot;<br>  type = &quot;email&quot;<br>  labels = {<br>    email_address = &quot;<a href="/service/mailto:support-website@example.com">support-website@example.com</a>&quot;<br>  }<br>}</pre><pre>output &quot;email0_id&quot; {<br>  value = &quot;${google_monitoring_notification_channel.email0.name}&quot;<br>}</pre><pre>output &quot;email1_id&quot; {<br>  value = &quot;${google_monitoring_notification_channel.email1.name}&quot;<br>}</pre><h3>Creating the Notification Channels</h3><p>Creating the Notification Channels requires 2 simple steps. First, initialize Terraform.</p><pre>$ terraform init</pre><pre>Initializing provider plugins...</pre><pre>The following providers do not have any version constraints in configuration,<br>so the latest version was installed.</pre><pre>To prevent automatic upgrades to new major versions that may contain breaking<br>changes, it is recommended to add version = &quot;...&quot; constraints to the<br>corresponding provider blocks in configuration, with the constraint strings<br>suggested below.</pre><pre>* provider.google: version = &quot;~&gt; 2.1&quot;</pre><pre>Terraform has been successfully initialized!</pre><p>Then, the standard Terraform apply does all the work.</p><pre>$ terraform apply</pre><pre>google_monitoring_notification_channel.email1: Refreshing state... (ID: projects/abab-cdcd-023991/notificationChannels/12532464389112270325)<br>google_monitoring_notification_channel.email0: Refreshing state... (ID: abab-cdcd-023991/notificationChannels/11497409249297891355)</pre><pre>An execution plan has been generated and is shown below.<br>Resource actions are indicated with the following symbols:<br>  ~ update in-place</pre><pre>Terraform will perform the following actions:</pre><pre>~ google_monitoring_notification_channel.email0<br>      labels.email_address: &quot;<a href="/service/mailto:website-oncall@example.com">website-oncall@example.com</a>&quot; =&gt; &quot;<a href="/service/mailto:website-oncall-a@example.com">website-oncall-a@example.com</a>&quot;</pre><pre>~ google_monitoring_notification_channel.email1<br>      labels.email_address: &quot;<a href="/service/mailto:support-website@example.com">support-website@example.com</a>&quot; =&gt; &quot;<a href="/service/mailto:support-website-a@example.com">support-website-a@example.com</a>&quot;</pre><pre>Plan: 0 to add, 2 to change, 0 to destroy.</pre><pre>Do you want to perform these actions?<br>  Terraform will perform the actions described above.<br>  Only &#39;yes&#39; will be accepted to approve.</pre><pre>Enter a value: yes</pre><pre>google_monitoring_notification_channel.email1: Modifying... (ID: projects/abab-cdcd-023991/notificationChannels/12532464389112270325)<br>  labels.email_address: &quot;<a href="/service/mailto:support-website@example.com">support-website@example.com</a>&quot; =&gt; &quot;<a href="/service/mailto:support-website-a@example.com">support-website-a@example.com</a>&quot;<br>google_monitoring_notification_channel.email0: Modifying... (ID: projects/abab-cdcd-023991/notificationChannels/11497409249297891355)<br>  labels.email_address: &quot;<a href="/service/mailto:website-oncall@example.com">website-oncall@example.com</a>&quot; =&gt; &quot;<a href="/service/mailto:website-oncall-a@example.com">website-oncall-a@example.com</a>&quot;<br>google_monitoring_notification_channel.email1: Modifications complete after 1s (ID: projects/abab-cdcd-023991/notificationChannels/12532464389112270325)<br>google_monitoring_notification_channel.email0: Modifications complete after 2s (ID: projects/abab-cdcd-023991/notificationChannels/11497409249297891355)</pre><pre>Apply complete! Resources: 0 added, 2 changed, 0 destroyed.</pre><pre>Outputs:</pre><pre>email0_id = projects/abab-cdcd-023991/notificationChannels/11497409249297891355<br>email1_id = projects/abab-cdcd-023991/notificationChannels/12532464389112270325</pre><h3>Alerting Policies Configuration</h3><p>I took the outputs from the notification channels (email0_id and email1_id above) and added them as local variables in the alerting policies configuration. The 5 alerting conditions described above translated into 5 alerting policies. Please note that I’ve only included 2 alerting policies here for the sake of readability. You can find the full configuration files on the <a href="https://github.com/charlesbaer/stackdriver-automation">github repo</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1599e4f97c8c0e656e878bd2f377ce38/href">https://medium.com/media/1599e4f97c8c0e656e878bd2f377ce38/href</a></iframe><p>After the standard initialization, the terraform apply does all the work and creates the Alerting Policies.</p><pre>$ terraform apply</pre><pre>google_monitoring_notification_channel.email0: Refreshing state... (ID: projects/abab-cdcd-023991/notificationChannels/6314605697446264142)<br>google_monitoring_alert_policy.alert_policy0: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/48947729567500594)<br>google_monitoring_alert_policy.alert_policy4: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/16071394893024794286)<br>google_monitoring_alert_policy.alert_policy2: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/5670542644520942633)<br>google_monitoring_alert_policy.alert_policy3: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/6104269770159121403)<br>google_monitoring_notification_channel.email1: Refreshing state... (ID: projects/abab-cdcd-023991/notificationChannels/10463049156064412054)<br>google_monitoring_alert_policy.alert_policy1: Refreshing state... (ID: projects/abab-cdcd-023991/alertPolicies/5985603450478517474)</pre><p>Once the alerting policies were created, I used the Stackdriver Monitoring console to verify that the alerting policies had been successfully created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TDEqejUcKN9VIOvDoe80A.png" /></figure><p>This concludes part 5 of the series. Read more about Stackdriver Monitoring Automation with Terraform in the other posts in the series and references below.</p><ul><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-4-stackdriver-groups-with-terraform-910289d16d08">Stackdriver Monitoring Automation Part 4: Stackdriver Groups with Terraform</a></li><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-6-uptime-checks-with-terraform-76cb25b996a4">Stackdriver Monitoring Automation Part 6: Stackdriver Uptime Checks with Terraform</a></li></ul><p>References:</p><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">Stackdriver Monitoring API docs</a></li><li><a href="https://cloud.google.com/monitoring/api/v3/filters">Stackdriver Monitoring filters</a></li><li><a href="https://cloud.google.com/monitoring/api/metrics_gcp">Stackdriver Monitoring metrics</a></li><li><a href="https://www.terraform.io/docs/providers/google/provider_reference.html">Terraform Google Provider</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff77b19b4b97" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-5-alerting-policies-ff77b19b4b97">Stackdriver Monitoring Automation Part 5: Alerting Policies</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackdriver Monitoring Automation Part 4: Stackdriver Groups with Terraform]]></title>
            <link>https://medium.com/google-cloud/stackdriver-monitoring-automation-part-4-stackdriver-groups-with-terraform-910289d16d08?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/910289d16d08</guid>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Tue, 05 Mar 2019 16:07:47 GMT</pubDate>
            <atom:updated>2019-03-05T16:43:43.950Z</atom:updated>
            <content:encoded><![CDATA[<p>In my quest for improving life through automation, I wrote a 3-part series about Stackdriver Monitoring Automation that used Google Cloud Deployment Manager to deploy Monitoring resources. I recently sat down to translate that automation into Terraform since Stackdriver Monitoring support has been added to Terraform. You can use these steps to automate the deployment of Stackdriver Monitoring resources in your environment using Terraform.</p><p>In this addition to the previous series, I have included the Terraform configuration for the same 3 scenarios described in the original<a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-1-stackdriver-groups-8e51f0aa9b03"> Stackdriver Monitoring Automation</a> series and won’t repeat the scenarios here. This post covers Stackdriver Groups while <a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-5-alerting-policies-ff77b19b4b97">part 5</a> and <a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-6-uptime-checks-with-terraform-76cb25b996a4">part 6</a> cover Alerting Policies and Uptime Checks, respectively.</p><h3>What’s available for automation in Terraform for Stackdriver Monitoring?</h3><p>The following <a href="https://www.terraform.io/docs/providers/google/index.html">components</a> are available as Terraform data sources and therefore can be used with the automation. The Monitoring coverage has expanded to cover the 4 main Stackdriver configurations also available via Monitoring API.</p><ul><li>Stackdriver Groups</li><li>Alerting Policies</li><li>Notification Channels</li><li>Uptime Checks</li></ul><h3>Setting up Terraform for Monitoring</h3><p>I decided to make use of <a href="https://cloud.google.com/shell/">Cloud Shell</a> to run my Terraform scripts because Cloud Shell provides an easy command line interface and automatically provides the Terraform provider.</p><p>The first step is to configure the Google provider replacing each entry with the appropriate values for my GCP environment. Here’s the format described in the Terraform <a href="https://www.terraform.io/docs/providers/google/provider_reference.html">docs</a>.</p><pre>provider &quot;google&quot; {<br>  credentials <strong>=</strong> &quot;${file(&quot;account_key.<strong>json</strong>&quot;)}&quot;<br>  project     <strong>=</strong> &quot;abab-cdcd-023991&quot;<br>  region      <strong>=</strong> &quot;us-central1&quot;<br>  zone        = &quot;us-central1-c&quot;</pre><pre>}</pre><p>I removed the credentials line in my configuration because Cloud Shell provides the credentials by default and thus, didn’t require a service account key file. The IAM roles necessary for the deployment were already granted to me as the project owner. You will need the appropriate IAM permissions if you are not the project owner.</p><pre>provider &quot;google&quot; {<br>  project = &quot;abab-cdcd-023991&quot;<br>  region  = &quot;us-central1&quot;<br>  zone    = &quot;us-central1-c&quot;<br>}</pre><h3>The Configuration Files</h3><p>I created 3 separate groups: Apache, prod and qa based on the Stackdriver resource tags. All apache instances are included in the Apache Group based on the app=website tags. Only the instances tagged with env=qa and env=prod were included in the qa and prod Groups, respectively.</p><p>The qa and prod Groups specify Apache as the parent which tells Stackdriver Monitoring that these are subgroups. I used a reference to refer to parent names for each of the subgroups.</p><pre>provider &quot;google&quot; {<br>  project = &quot;abab-cdcd-023991&quot;<br>  region  = &quot;us-central1&quot;<br>  zone    = &quot;us-central1-c&quot;<br>}</pre><pre>resource &quot;google_monitoring_group&quot; &quot;apache_parent&quot; {<br>  display_name = &quot;Apache&quot;<br>  filter = &quot;metadata.user_labels.app=has_substring(\&quot;website\&quot;)&quot;<br>}</pre><pre>resource &quot;google_monitoring_group&quot; &quot;apache_prod_subgroup&quot; {<br>  display_name = &quot;prod&quot;<br>  filter = &quot;metadata.user_labels.env=\&quot;prod\&quot;&quot;<br>  parent_name =  &quot;${google_monitoring_group.apache_parent.name}&quot;<br>}</pre><pre>resource &quot;google_monitoring_group&quot; &quot;apache_qa_subgroup&quot; {<br>  display_name = &quot;qa&quot;<br>  filter = &quot;metadata.user_labels.env=\&quot;qa\&quot;&quot;<br>  parent_name =  &quot;${google_monitoring_group.apache_parent.name}&quot;<br>}</pre><p>You can find the main.tf files on the <a href="https://github.com/charlesbaer/stackdriver-automation">github repo</a>.</p><p><a href="https://github.com/charlesbaer/stackdriver-automation">charlesbaer/stackdriver-automation</a></p><h3>Creating the Groups and Subgroups</h3><p>Adding the Stackdriver Groups requires 2 simple steps. First, initialize Terraform.</p><pre>$ terraform init</pre><pre>Initializing provider plugins...</pre><pre>The following providers do not have any version constraints in configuration,<br>so the latest version was installed.</pre><pre>To prevent automatic upgrades to new major versions that may contain breaking<br>changes, it is recommended to add version = &quot;...&quot; constraints to the<br>corresponding provider blocks in configuration, with the constraint strings<br>suggested below.</pre><pre>* provider.google: version = &quot;~&gt; 2.1&quot;<br>Terraform has been successfully initialized!</pre><p>Then, the standard Terraform apply does all the work.</p><pre>$ terraform apply<br>google_monitoring_group.apache_parent: Refreshing state... (ID: projects/abab-cdcd-023991/groups/4800045303134799755)<br>google_monitoring_group.apache_prod_subgroup: Refreshing state... (ID: projects/abab-cdcd-023991/groups/1346673955451905122)<br>google_monitoring_group.apache_qa_subgroup: Refreshing state... (ID: projects/abab-cdcd-023991/groups/2305611457471642078)</pre><pre>An execution plan has been generated and is shown below.<br>Resource actions are indicated with the following symbols:<br>  + create</pre><pre>Terraform will perform the following actions:</pre><pre>+ google_monitoring_group.apache_parent<br>      id:           &lt;computed&gt;<br>      display_name: &quot;Apache&quot;<br>      filter:       &quot;metadata.user_labels.app=has_substring(\&quot;website\&quot;)&quot;<br>      name:         &lt;computed&gt;<br>      project:      &lt;computed&gt;</pre><pre>+ google_monitoring_group.apache_prod_subgroup<br>      id:           &lt;computed&gt;<br>      display_name: &quot;prod&quot;<br>      filter:       &quot;metadata.user_labels.env=\&quot;prod\&quot;&quot;<br>      name:         &lt;computed&gt;<br>      parent_name:  &quot;${google_monitoring_group.apache_parent.name}&quot;<br>      project:      &lt;computed&gt;</pre><pre>+ google_monitoring_group.apache_qa_subgroup<br>      id:           &lt;computed&gt;<br>      display_name: &quot;qa&quot;<br>      filter:       &quot;metadata.user_labels.env=\&quot;qa\&quot;&quot;<br>      name:         &lt;computed&gt;<br>      parent_name:  &quot;${google_monitoring_group.apache_parent.name}&quot;<br>      project:      &lt;computed&gt;</pre><pre>Plan: 3 to add, 0 to change, 0 to destroy.</pre><pre>Do you want to perform these actions?<br>  Terraform will perform the actions described above.<br>  Only &#39;yes&#39; will be accepted to approve.</pre><pre>Enter a value: yes</pre><pre>google_monitoring_group.apache_parent: Creating...<br>  display_name: &quot;&quot; =&gt; &quot;Apache&quot;<br>  filter:       &quot;&quot; =&gt; &quot;metadata.user_labels.app=has_substring(\&quot;website\&quot;)&quot;<br>  name:         &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>  project:      &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>google_monitoring_group.apache_parent: Creation complete after 1s (ID: projects/abab-cdcd-023991/groups/3905630164280032312)<br>google_monitoring_group.apache_qa_subgroup: Creating...<br>  display_name: &quot;&quot; =&gt; &quot;qa&quot;<br>  filter:       &quot;&quot; =&gt; &quot;metadata.user_labels.env=\&quot;qa\&quot;&quot;<br>  name:         &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>  parent_name:  &quot;&quot; =&gt; &quot;projects/abab-cdcd-023991/groups/3905630164280032312&quot;<br>  project:      &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>google_monitoring_group.apache_prod_subgroup: Creating...<br>  display_name: &quot;&quot; =&gt; &quot;prod&quot;<br>  filter:       &quot;&quot; =&gt; &quot;metadata.user_labels.env=\&quot;prod\&quot;&quot;<br>  name:         &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>  parent_name:  &quot;&quot; =&gt; &quot;projects/abab-cdcd-023991/groups/3905630164280032312&quot;<br>  project:      &quot;&quot; =&gt; &quot;&lt;computed&gt;&quot;<br>google_monitoring_group.apache_qa_subgroup: Creation complete after 1s (ID: projects/abab-cdcd-023991/groups/7315859232068601075)<br>google_monitoring_group.apache_prod_subgroup: Creation complete after 2s (ID: projects/abab-cdcd-023991/groups/457136244756364624)</pre><p>Once the Groups were created, I used the Stackdriver Monitoring console to verify that the Apache, qa and prod Groups had been successfully created. Notice that the prod and qa subgroups appear under the Apache group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YamaABlnUgqHx552yCSzyA.png" /></figure><p>This concludes part 4 of the series. Read more about Stackdriver Monitoring Automation with Terraform in the other posts in the series and references below.</p><ul><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-5-alerting-policies-ff77b19b4b97">Stackdriver Monitoring Automation Part 5: Stackdriver Alerting Policies with Terraform</a></li><li><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-6-uptime-checks-with-terraform-76cb25b996a4">Stackdriver Monitoring Automation Part 6: Stackdriver Uptime Checks with Terraform</a></li></ul><p>References:</p><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">Stackdriver Monitoring API docs</a></li><li><a href="https://cloud.google.com/monitoring/api/v3/filters">Stackdriver Monitoring filters</a></li><li><a href="https://cloud.google.com/monitoring/api/metrics_gcp">Stackdriver Monitoring metrics</a></li><li><a href="https://www.terraform.io/docs/providers/google/provider_reference.html">Terraform Google Provider</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=910289d16d08" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-4-stackdriver-groups-with-terraform-910289d16d08">Stackdriver Monitoring Automation Part 4: Stackdriver Groups with Terraform</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackdriver Monitoring Automation Part 3: Uptime Checks]]></title>
            <link>https://medium.com/google-cloud/stackdriver-monitoring-automation-part-3-uptime-checks-476b8507f59c?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/476b8507f59c</guid>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[deployment-manager]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Fri, 28 Sep 2018 19:32:42 GMT</pubDate>
            <atom:updated>2019-09-13T15:36:45.514Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is part 3 in the Stackdriver Automation series. In <a href="https://medium.com/google-cloud/stackdriver-automation-part-1-stackdriver-groups-8e51f0aa9b03">part 1</a> and <a href="https://medium.com/google-cloud/stackdriver-automation-part-2-alerting-policies-9f42068603c4">part 2</a>, I covered automating the management of Stackdriver Groups and Alerting Policies, respectively. In this post, I will walk through the steps that you can use to automate the management of Uptime Checks. Head over to part 1 for the background and prerequisites.</p><h3>Uptime Checks</h3><p><a href="https://cloud.google.com/monitoring/uptime-checks/">Uptime Checks</a> lets you verify the availability of your service via HTTP, HTTPS or TCP health checks. Stackdriver provides this service by accessing your application’s frontend from locations around the world and reporting on the results such as latency and response codes. You can use this service as a way to understand whether your users can access the service from various global locations.</p><p>As an example, I created an Uptime Check based on the apache infrastructure that I described in parts 1 and 2. The application itself is a simple website being hosted by apache servers behind a load-balancer. I created a simple HTTP Uptime Check to understand whether my users could access the service.</p><p>The <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs/create">projects.uptimeCheckConfigs.create</a> API lists the following values that are required to create an Uptime Check.</p><pre>{<br>  &quot;name&quot;: string,<br>  &quot;displayName&quot;: string,<br>  &quot;period&quot;: string,<br>  &quot;timeout&quot;: string,<br>  &quot;contentMatchers&quot;: [<br>    {<br>      object(ContentMatcher)<br>    }<br>  ],<br>  &quot;selectedRegions&quot;: [<br>    enum(UptimeCheckRegion)<br>  ],<br>  &quot;isInternal&quot;: boolean,<br>  &quot;internalCheckers&quot;: [<br>    {<br>      object(InternalChecker)<br>    }<br>  ],</pre><pre>// Union field resource can be only one of the following:<br>  &quot;monitoredResource&quot;: {<br>    object(MonitoredResource)<br>  },<br>  &quot;resourceGroup&quot;: {<br>    object(ResourceGroup)<br>  }<br>  // End of list of possible types for union field resource.</pre><pre>// Union field check_request_type can be only one of the following:<br>  &quot;httpCheck&quot;: {<br>    object(HttpCheck)<br>  },<br>  &quot;tcpCheck&quot;: {<br>    object(TcpCheck)<br>  }<br>  // End of list of possible types for union field check_request_type.<br>}</pre><p>I used the “Try this API” sidebar in the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs/create">projects.uptimeCheckConfigs.create</a> docs to test out the values. I separated the templates into jinja templates and yaml files so that I could reuse the jinja templates for any other Uptime Checks.</p><p><strong>stackdriver_uptimecheckconfigs.jinja</strong></p><pre>{% set PREFIX = env[&quot;deployment&quot;] %}<br>{% set UPTIME_CHECKS = properties[&quot;uptimechecks&quot;] %}<br>{% set PROJECT = env[&quot;project&quot;] %}</pre><pre>resources:<br>{% for uptimecheck in UPTIME_CHECKS %}<br>- name: {{ PREFIX }}-uptimecheck-{{ loop.index }}<br>  type: gcp-types/monitoring-v3:monitoring.projects.uptimeCheckConfigs.create<br>  properties:<br>    parent: projects/{{ PROJECT }}<br>    displayName: {{ uptimecheck.name }}<br>    period: {{ uptimecheck.period }}<br>    timeout: {{ uptimecheck.timeout }}<br>    monitoredResource: {{ uptimecheck.monitoredResource }}<br>    httpCheck: {{ uptimecheck.httpCheck }}<br>{% endfor %}</pre><p>Notice that I created the yaml with the ability to create multiple Uptime Checks under the uptimechecks block. I created a standard httpCheck against the URL for my load-balancer which is represented as a monitoredResource object in the API.</p><p><strong>stackdriver_uptimecheckconfigs.yaml</strong></p><p>Please the YOUR_PROJECT_ID value with the value of your GCP project id along with your URL value of the host</p><pre>imports:<br>- path: stackdriver_uptimecheckconfigs.jinja<br>resources:<br>- name: create_uptimechecks<br>  type: stackdriver_uptimecheckconfigs.jinja<br>  properties:<br>    uptimechecks:<br>    - name: &quot;1 - Website uptime check [global]&quot;<br>      period: &quot;60s&quot;<br>      timeout: &quot;10s&quot;<br>      monitoredResource:<br>        type: &quot;uptime_url&quot;<br>        labels:<br>          project_id: YOUR_PROJECT_ID<br>          host: &quot;35.241.47.194&quot;<br>      httpCheck:<br>        path: &quot;/&quot;<br>        port: 80</pre><p>You can find the jinja and yaml files on the <a href="https://github.com/charlesbaer/stackdriver-automation">github repo</a>.</p><p>The last step was to use the gcloud command line below to actually create the Stackdriver Uptime Check.</p><pre>$ gcloud deployment-manager deployments create website-uptimecheck --config stackdriver_uptimecheckconfigs.yaml </pre><pre>                                                                                          <br>Create operation operation-1537814541833-576a2597f1329-1faf3a15-f226e553 completed successfully.<br>NAME                         TYPE                                                                   STATE      ERRORS  INTENT<br>website-uptimecheck-uptimecheck-1  gcp-types/monitoring-v3:monitoring.projects.uptimeCheckConfigs.create  COMPLETED  []</pre><p>Once the uptime checks were created, I used the Stackdriver Monitoring console to verify that the uptime checks had been successfully created. Keep in mind that when you first create uptime checks, it can take up to 25 mins to start reporting status according to the <a href="https://cloud.google.com/monitoring/uptime-checks/management#before_you_begin">docs</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KNNyBNEApUdEv45bZD-JXg.png" /></figure><p>Now, the uptime checks are active and can be used in conjunction with alerting policies to notify you when the uptime checks fail or when the latency is outside of tolerance.</p><h3>Conclusion: The Stackdriver Monitoring Automation Series</h3><p>This concludes the Stackdriver Monitoring Automation series. In this series, I have included the steps that I took and the methods that I used to create Stackdriver Monitoring components via Google Cloud Deployment Manager. In the first post, I created a Stackdriver Group used to group resources to monitor as a single entity. In the second post, I created Stackdriver Alerting Policies to define when to send an alert and what alert to send. In this third post, I created Stackdriver Uptime Checks to provide a basic picture of end-user experience. You can use these steps and config files as templates to automate the deployment of Stackdriver Monitoring resources in your environment.</p><p>Read more about Stackdriver Monitoring Automation in the other posts in the series and references below.</p><ul><li><a href="https://medium.com/google-cloud/stackdriver-automation-part-1-stackdriver-groups-8e51f0aa9b03">Stackdriver Automation Part 1: Stackdriver Groups</a></li><li><a href="https://medium.com/@charles.baer/stackdriver-automation-part-2-alerting-policies-9f42068603c4">Stackdriver Automation Part 2: Stackdriver Alerting Policies</a></li></ul><p>References:</p><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">Stackdriver Monitoring API docs</a></li><li><a href="https://cloud.google.com/monitoring/api/v3/filters">Stackdriver Monitoring filters</a></li><li><a href="https://cloud.google.com/monitoring/api/metrics_gcp">Stackdriver Monitoring metrics</a></li><li><a href="https://cloud.google.com/deployment-manager/docs/">Google Cloud Deployment Manager</a></li><li><a href="https://github.com/GoogleCloudPlatform/deploymentmanager-samples">Deployment Manager Samples</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=476b8507f59c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-3-uptime-checks-476b8507f59c">Stackdriver Monitoring Automation Part 3: Uptime Checks</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackdriver Monitoring Automation Part 2: Alerting Policies]]></title>
            <link>https://medium.com/google-cloud/stackdriver-monitoring-automation-part-2-alerting-policies-9f42068603c4?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f42068603c4</guid>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[deployment-manager]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Fri, 28 Sep 2018 19:32:17 GMT</pubDate>
            <atom:updated>2018-09-28T19:42:32.761Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is part 2 in the Stackdriver Automation series. In <a href="https://medium.com/google-cloud/stackdriver-automation-part-1-stackdriver-groups-8e51f0aa9b03">part 1</a>, I covered automating the management of Stackdriver Groups. In this post, I will walk through the steps that you can use to automate the management of Alerting Policies. Head over to part 1 for the background and prerequisites.</p><h3>Alerting Policies</h3><p><a href="https://cloud.google.com/monitoring/alerts/">Alerting Policies</a> lets you define a set of conditions that will generate a notification whenever that event is triggered. The notification can optionally contain documentation and metadata to provide additional information to the person receiving the alert. Alerting policies can be stand alone or can be attached to Stackdriver Groups.</p><p>There are several different notification channels including email and SMS notification as well and integrations with 3rd parties such as PagerDuty, HipChat and Slack. Alternatively, you can use a webhook to call a service that you define whenever a notification is generated. The types of <a href="https://cloud.google.com/monitoring/alerts/concepts-indepth#condition-types">alerting conditions</a> together with the monitoring filters provide a rich set of options to tailor you notifications.</p><p>As an example, I created alerting based on the apache infrastructure that I described in part 1. The application itself is a simple website being hosted by apache servers behind a load-balancer. Selecting rational alerting requires consideration of the application’s users and the application architecture. I turned to the <a href="https://landing.google.com/sre/interview/ben-treynor.html">Site Reliability Engineering</a> (SRE) methods to select the metrics used for alerting.</p><p>SRE has a concept of service level indicators (SLIs) as a way to track your application’s performance metrics as measured by user impact. Good SLIs should make it easy to know when the performance of your application is making your users frustrated and when your users are having a good experience using the app. The main SLI categories are latency, availability, volume and quality. This article uses these SLIs to drive the alerting. For a production application, the selection of SLIs should be done with more rigor. This was a demo app after all!</p><h3>SLI Alerting Metrics</h3><p>The main Stackdriver Alerting conditions, notifications and documentation that I selected were the following:</p><p><strong>Conditions</strong></p><p><em>For Availability &amp; Quality:</em></p><ul><li>L7 load balancer error request count = 0/sec</li></ul><p><em>For Latency:</em></p><ul><li>L7 Load balancer total latency &gt; 100ms</li><li>L7 load balancer Frontend RTT latency &gt; 50ms</li><li>L7 load balancer Backend RTT latency &gt; 50ms</li></ul><p><em>For Volume:</em></p><ul><li>L7 load balancer request count &gt; 100/sec</li></ul><p><strong>Notification channels</strong></p><ul><li>Email: <a href="/service/mailto:apache-website-oncall@example.com">website-oncall@example.com</a></li><li>Email: support-website@example.com</li></ul><p><strong>Documentation</strong></p><ul><li>Specific text describing each alerting policy</li></ul><p>Now that I determined the specific metrics to monitor and the notification method, I needed to translate those rules into the format accepted by the Stackdriver AlertingPolicies API.</p><p>The <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies/create">projects.alertPolicies.create</a> API lists the following values that are required to create the Alerting Policy.</p><pre>{<br>  &quot;name&quot;: string,<br>  &quot;displayName&quot;: string,<br>  &quot;documentation&quot;: {<br>    object(Documentation)<br>  },<br>  &quot;userLabels&quot;: {<br>    string: string,<br>    ...<br>  },<br>  &quot;conditions&quot;: [<br>    {<br>      object(Condition)<br>    }<br>  ],<br>  &quot;combiner&quot;: enum(ConditionCombinerType),<br>  &quot;enabled&quot;: boolean,<br>  &quot;notificationChannels&quot;: [<br>    string<br>  ],<br>  &quot;creationRecord&quot;: {<br>    object(MutationRecord)<br>  },<br>  &quot;mutationRecord&quot;: {<br>    object(MutationRecord)<br>  }<br>}</pre><p>The alerting policies are quite flexible and offer the ability to filter, group, aggregate and then measure the metrics change over time. I experimented with creating alerting policies in the UI and then used the “Try this API” sidebar in the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies/create">projects.alertPolicies.create</a> docs to view the resulting configuration. That approach along with the Stackdriver Monitoring docs helped to translate the 5 alerting conditions that I had selected into 5 alerting policies. I separated the templates into jinja templates and yaml files so that I could reuse the jinja templates for any other Alerting Policies.</p><p><strong>stackdriver_alertingpolicies.jinja</strong></p><pre>{% set PREFIX = env[&quot;deployment&quot;] %}<br>{% set NOTIFICATION_EMAILS = properties[&quot;notificationEmails&quot;] %}<br>{% set POLICIES = properties[&quot;policies&quot;] %}<br>{% set PROJECT = env[&quot;project&quot;] %}<br>{% set DEFAULT_MIME_TYPE = &quot;text/markdown&quot; %}</pre><pre>resources:<br>{% for email in NOTIFICATION_EMAILS %}<br>- name: {{ PREFIX }}-email-{{ loop.index }}<br>  type: gcp-types/monitoring-v3:projects.notificationChannels<br>  properties:<br>    name: projects/{{ PROJECT }}<br>    type: email<br>    displayName: {{ email.displayName }}<br>    labels:<br>      email_address: {{ email.emailAddress }}<br>    enabled: true<br>{% endfor %}</pre><pre>{% for policy in POLICIES %}<br>- name: {{ PREFIX }}-alertingpolicy-{{ loop.index }}<br>  type: gcp-types/monitoring-v3:projects.alertPolicies<br>  properties:<br>    displayName: {{ PREFIX }}-{{ policy.name }}<br>    documentation:<br>      content: {{ policy.documentation.content }}<br>      mimeType:  {{ DEFAULT_MIME_TYPE }}<br>    combiner: OR<br>    conditions:<br>{% for condition in policy.conditions %}<br>    - displayName: {{ condition.displayName }}<br>      conditionThreshold:<br>        filter: {{ condition.filter }}<br>        comparison: {{condition.comparison }}<br>        duration: {{ condition.duration }}<br>        thresholdValue: {{ condition.thresholdValue }}<br>        trigger: {{ condition.trigger }}<br>        aggregations: {{ condition.aggregations }}<br>{% endfor  %}        <br>    notificationChannels:<br>{% for notification in NOTIFICATION_EMAILS %}<br>    - $(ref.{{ PREFIX }}-email-{{ loop.index }}.name)<br>{% endfor %}<br>    enabled: true<br>{% endfor %}</pre><p>Notice that I created the yaml with multiple notification channels and policies. I separated the email notification creation from the policies and added the same email notifications to each policy in the jinja template.</p><p><strong>Stackdriver_alertingpolicies.yaml</strong></p><p>Please note that I’ve only included 2 alerting policies here for the sake of readability. See the <a href="https://github.com/charlesbaer/stackdriver-automation">github</a> repo for the full yaml file.</p><p><a href="https://github.com/charlesbaer/stackdriver-automation">charlesbaer/stackdriver-automation</a></p><pre>imports:<br>- path: stackdriver_alertingpolicies.jinja<br>resources:<br>- name: create_alertingpolicies<br>  type: stackdriver_alertingpolicies.jinja<br>  properties:<br>    notificationEmails:<br>    - emailAddress: <a href="/service/mailto:website-oncall@example.com">website-oncall@example.com</a><br>      displayName: &quot;Website Oncall&quot; <br>    - emailAddress: <a href="/service/mailto:support-website@example.com">support-website@example.com</a><br>      displayName: &quot;Website Support&quot;  <br>    policies:<br>    - name: &quot;1 - Availability - Google Cloud HTTP/S Load Balancing Rule - Request count (filtered) [COUNT]&quot;   <br>      conditions: <br>      - filter: &quot;metric.type=\&quot;loadbalancing.googleapis.com/https/request_count\&quot; resource.type=\&quot;https_lb_rule\&quot; metric.label.response_code!=\&quot;200\&quot;&quot; <br>        comparison: &quot;COMPARISON_GT&quot;<br>        duration: &quot;60s&quot; <br>        thresholdValue: 1 <br>        trigger:<br>          count: 1<br>        aggregations:<br>        - alignmentPeriod: &quot;60s&quot;<br>          perSeriesAligner: &quot;ALIGN_RATE&quot;<br>          crossSeriesReducer: &quot;REDUCE_COUNT&quot;<br>        displayName: &quot;Google Cloud HTTP/S Load Balancing Rule - Request count (filtered) [COUNT]&quot;<br>      documentation:<br>        content: &quot;The load balancer rule ${condition.display_name} has generated this alert for the ${metric.display_name}.&quot;<br>    - name: &quot;2 - Latency - Google Cloud HTTP/S Load Balancing Rule - Total Latency (filtered) [99 percentile]&quot;   <br>      conditions: <br>      - filter: &quot;metric.type=\&quot;loadbalancing.googleapis.com/https/total_latencies\&quot; resource.type=\&quot;https_lb_rule\&quot; resource.label.url_map_name=\&quot;web-map\&quot;&quot; <br>        comparison: &quot;COMPARISON_GT&quot;<br>        duration: &quot;60s&quot; <br>        thresholdValue: 100 <br>        trigger:<br>          count: 1<br>        aggregations:<br>        - alignmentPeriod: &quot;60s&quot;<br>          perSeriesAligner: &quot;ALIGN_PERCENTILE_99&quot;<br>        displayName: &quot;Google Cloud HTTP/S Load Balancing Rule - Total Latency (filtered) [99 percentile]&quot;<br>      documentation:<br>        content: &quot;The load balancer rule ${condition.display_name} has generated this alert for the ${metric.display_name}.&quot;</pre><p>You can find the jinja and yaml files on the <a href="https://github.com/charlesbaer/stackdriver-automation">github repo</a>.</p><p>The last step was to use the gcloud command line below to actually create the Stackdriver Alerting Policies.</p><pre>$ gcloud deployment-manager deployments create stackdriver-alertingpolicies-apache --config stackdriver_alertingpolicies.yaml</pre><pre>Create operation operation-1537809007185-576a10f9b0c68-751256f4-45889de0 completed successfully.<br>NAME                                                  TYPE                                                   STATE      ERRORS  INTENT<br>stackdriver-alertingpolicies-apache-alertingpolicy-1  gcp-types/monitoring-v3:projects.alertPolicies         COMPLETED  []<br>stackdriver-alertingpolicies-apache-alertingpolicy-2  gcp-types/monitoring-v3:projects.alertPolicies         COMPLETED  []<br>stackdriver-alertingpolicies-apache-alertingpolicy-3  gcp-types/monitoring-v3:projects.alertPolicies         COMPLETED  []<br>stackdriver-alertingpolicies-apache-alertingpolicy-4  gcp-types/monitoring-v3:projects.alertPolicies         COMPLETED  []<br>stackdriver-alertingpolicies-apache-alertingpolicy-5  gcp-types/monitoring-v3:projects.alertPolicies         COMPLETED  []<br>stackdriver-alertingpolicies-apache-email-1           gcp-types/monitoring-v3:projects.notificationChannels  COMPLETED  []<br>stackdriver-alertingpolicies-apache-email-2           gcp-types/monitoring-v3:projects.notificationChannels  COMPLETED  []</pre><p>Once the alerting policies were created, I used the Stackdriver Monitoring console to verify that the alerting policies had been successfully created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b1P4hJjcDoz_-_ss" /></figure><h3>Testing the alerting</h3><p>I intentionally set the metric values low to make it easy to test the alerting. In this example, picking a low latency value for alerting generates alerts that you can use to verify that the alerts arrive successfully and include the details that you intended. Here’s a sample of an alert that I received.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*hq_Vqi8f5yNmqr9PCQUtVw.png" /></figure><p>Whenever the condition that caused the alert was resolved, the alert was automatically resolved. Here’s the sample of the email alert resolution below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*ETNEmjmQP9Kb6TWK2Qfgsw.png" /></figure><p>This concludes part 2 of the series. Read more about Stackdriver Monitoring Automation in the other posts in the series and references below.</p><ul><li><a href="https://medium.com/google-cloud/stackdriver-automation-part-1-stackdriver-groups-8e51f0aa9b03">Stackdriver Automation Part 1: Stackdriver Groups</a></li><li><a href="https://medium.com/@charles.baer/stackdriver-automation-part-3-uptime-checks-476b8507f59c">Stackdriver Automation Part 3: Stackdriver Uptime Checks</a></li></ul><p>References:</p><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">Stackdriver Monitoring API docs</a></li><li><a href="https://cloud.google.com/monitoring/api/v3/filters">Stackdriver Monitoring filters</a></li><li><a href="https://cloud.google.com/monitoring/api/metrics_gcp">Stackdriver Monitoring metrics</a></li><li><a href="https://cloud.google.com/deployment-manager/docs/">Google Cloud Deployment Manager</a></li><li><a href="https://github.com/GoogleCloudPlatform/deploymentmanager-samples">Deployment Manager Samples</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f42068603c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-2-alerting-policies-9f42068603c4">Stackdriver Monitoring Automation Part 2: Alerting Policies</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stackdriver Monitoring Automation Part 1: Stackdriver Groups]]></title>
            <link>https://medium.com/google-cloud/stackdriver-monitoring-automation-part-1-stackdriver-groups-8e51f0aa9b03?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e51f0aa9b03</guid>
            <category><![CDATA[stackdriver-monitoring]]></category>
            <category><![CDATA[stackdriver]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[deployment-manager]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Fri, 28 Sep 2018 19:31:57 GMT</pubDate>
            <atom:updated>2018-09-28T19:43:20.320Z</atom:updated>
            <content:encoded><![CDATA[<p>Many developers and organizations recognize that automation is critical to the long-term management of cloud infrastructure and applications. Developers, Site Reliability Engineers and Operations teams just getting started with Stackdriver Monitoring may wonder what components can be used with automation. I recently went through the exercise of automating the creation of Stackdriver Monitoring components for every new project created within an organization.</p><p>In this series, I have included the steps that I took and the methods that I used to create the automation. You can use these steps to automate the deployment of Stackdriver Monitoring resources in your environment. This post covers Stackdriver Groups while <a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-2-alerting-policies-9f42068603c4">part 2</a> and <a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-3-uptime-checks-476b8507f59c">part 3</a> cover Alerting Policies and Uptime Checks, respectively.</p><h3>What’s available for automation in Stackdriver Monitoring?</h3><p>The following components are available via API and therefore can be used with the automation.</p><ul><li>Stackdriver Groups</li><li>Alerting Policies</li><li>Uptime Checks</li></ul><p>The Stackdriver Monitoring APIs are available via <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">REST API</a> and <a href="https://cloud.google.com/monitoring/api/ref_v3/rpc/">gRPC</a>. The same REST APIs can be used in <a href="https://cloud.google.com/deployment-manager/docs/">Google Cloud Deployment Manager</a> or even the <a href="https://cloud.google.com/sdk/gcloud/">gcloud</a> command line. Using Deployment Manager makes it easy to treat the configuration as code and further automate the management of the Stackdriver Monitoring components. I used the Deployment Manager to implement the automation.</p><h3>1. Prerequisites</h3><p>To get started, I setup a Stackdriver Workspace and at least 1 GCP project associated with the Workspace. Workspaces were formerly called Stackdriver Accounts until 9/12/2018. See this <a href="https://cloud.google.com/blog/products/management-tools/using-stackdriver-workspaces-help-manage-your-hybrid-and-multicloud-environment">blog post</a> for more details.</p><p>I used a GCP environment that had many different components already in use including Cloud Storage, BigQuery, Compute Engine, Kubernetes Engine, Cloud Functions and Cloud ML Engine. Using an environment with many components in use made it easier for me to the monitoring components.</p><h3><strong>2. Stackdriver Groups</strong></h3><p><a href="https://cloud.google.com/monitoring/groups/">Stackdriver Groups</a> lets you define and monitor logical groups of resources in Stackdriver Monitoring. These include resources such as VM instances and containers. Groups can then be used to monitor a set of resources as a single entity. Groups can also include subgroups which can be used to logically build a groups hierarchy.</p><p>As an example, I create a group to monitor all of the resources related to a set of apache instances running on GCE. When I deployed the GCE resources, I added the tags app:website, env:prod to each of the GCE instances. You can see the labels attached to the instance below.</p><pre>$ gcloud compute instances describe wwwhttp11 --zone us-west1-b</pre><pre>canIpForward: false<br>cpuPlatform: Intel Broadwell<br>creationTimestamp: &#39;2018-09-24T13:16:33.497-07:00&#39;<br>deletionProtection: false<br>...<br>id: &#39;1488468442473625840&#39;<br>kind: compute#instance<br>labelFingerprint: FrEu9_zPopQ=<br><strong>labels:<br>  app: website<br>  env: prod</strong></pre><p>Then, when I created the Stackdriver Group, I could filter on the tags to include those specific resources in the group. This also highlight the dynamic abilities of Groups. I could also have used a name or resource type to filter as well. I also broke the groups into subgroups based on the env tag which allowed me to segregate the apache instances by environment.</p><p>The <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.groups/create">projects.groups.create</a> API lists the following values that are required to create the Group.</p><pre>{<br> “name”: string,<br> “displayName”: string,<br> “parentName”: string,<br> “filter”: string,<br> “isCluster”: boolean<br>}</pre><p>I used the “Try this API” sidebar in the <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.groups/create">projects.groups.create</a> docs to test out the API and ensure that I had the right values. This is an easy way to sanity check configuration values before adding them to deployment manager or via code.</p><p>I used the following values to test creating a simple Group:</p><pre>{<br>  &quot;displayName&quot;: “Apache test”,<br>  &quot;parentName&quot;: “”,<br>  &quot;filter&quot;:”metadata.user_labels.app=has_substring(\&quot;website\&quot;)”,<br>  &quot;isCluster&quot;: false<br>}</pre><p>The “Try this API” process works well for testing the API though I wanted to create a repeatable process that I could automate. This is where Deployment Manager comes in. I created the Deployment Manager configuration files to supply the values to the projects.groups.create API. I separated the templates into jinja templates and yaml files so that I could reuse the jinja templates for any other Groups.</p><p><strong>stackdriver_groups.jinja</strong></p><pre>resources:<br>- name:  {{ env[&quot;name&quot;] }}<br>  type: gcp-types/monitoring-v3:projects.groups <br>  properties:<br>    name: &quot;projects/{{ env[&quot;project&quot;] }}&quot;<br>    displayName: {{ properties[&quot;group_display_name&quot;] }}<br>    parentName: &quot;{{ properties[&quot;group_parent_name&quot;] }}&quot; <br>    filter: {{ properties[&quot;group_filter&quot;] }}<br>    isCluster: {{ properties[&quot;group_is_cluster&quot;] }}</pre><p>Notice that I created 3 separate groups in the yaml file: Apache, prod and qa based on the tags. All apache instances were included in the Apache Group based on the app=website tags. Only the instances tagged with env=qa and env=prod were included in the qa and prod Groups, respectively. The qa and prod Groups specify Apache as the parent which tells Stackdriver Monitoring that these are subgroups.</p><p><strong>stackdriver_groups.yaml</strong></p><pre>imports: <br>- path: stackdriver_groups.jinja<br>resources:<br>- name: create-apache-group<br>  type: stackdriver_groups.jinja <br>  properties:<br>    group_display_name: &quot;Apache&quot;<br>    group_parent_name: &quot;&quot;<br>    group_filter: &quot;metadata.user_labels.app=has_substring(\&quot;website\&quot;)&quot;<br>    group_is_cluster: false<br>- name: create-apache-prod-group<br>  type: stackdriver_groups.jinja<br>  properties:<br>    group_display_name: &quot;prod&quot;<br>    group_parent_name: $(ref.create-apache-group.name)<br>    group_filter: &quot;metadata.user_labels.env=\&quot;prod\&quot;&quot;<br>    group_is_cluster: false<br>    metadata:<br>      dependsOn:<br>        create-apache-group<br>- name: create-apache-qa-group<br>  type: stackdriver_groups.jinja<br>  properties:<br>    group_display_name: &quot;qa&quot;<br>    group_parent_name: $(ref.create-apache-group.name)<br>    group_filter: &quot;metadata.user_labels.env=\&quot;qa\&quot;&quot;<br>    group_is_cluster: false<br>    metadata:<br>      dependsOn:<br>        create-apache-group</pre><p>You can find the jinja and yaml files on the <a href="https://github.com/charlesbaer/stackdriver-automation">github repo</a>.</p><p>The last step was to use the gcloud command line below to actually create the Stackdriver Group via Deployment Manager. I used the --preview command line argument first to make sure that I got the deployment configuration right.</p><pre>$ gcloud deployment-manager deployments create apachegroup --config stackdriver_groups.yaml --preview</pre><pre>The fingerprint of the deployment is nm2-shmY2Oj2tDQfCc4__g==<br>Waiting for create [operation-1538010937030-576d0138ff573-f3a4b3a2-a3ed8f5c]...done.<br>Create operation operation-1538010937030-576d0138ff573-f3a4b3a2-a3ed8f5c completed successfully.<br>NAME                      TYPE                                     STATE       ERRORS  INTENT<br>create-apache-group       gcp-types/monitoring-v3:projects.groups  IN_PREVIEW  []      CREATE_OR_ACQUIRE<br>create-apache-prod-group  gcp-types/monitoring-v3:projects.groups  IN_PREVIEW  []      CREATE_OR_ACQUIRE<br>create-apache-qa-group    gcp-types/monitoring-v3:projects.groups  IN_PREVIEW  []      CREATE_OR_ACQUIRE</pre><p>The config did not generate any errors and so, I submitted the command to create the deployment.</p><pre>$ gcloud deployment-manager deployments create apachegroup --config stackdriver_groups.yaml</pre><pre>Create operation operation-1537807543251-576a0b8593038-2a7c079e-38f54092 completed successfully.<br>NAME                      TYPE                                     STATE      ERRORS  INTENT<br>create-apache-group       gcp-types/monitoring-v3:projects.groups  COMPLETED  []<br>create-apache-prod-group  gcp-types/monitoring-v3:projects.groups  COMPLETED  []<br>create-apache-qa-group    gcp-types/monitoring-v3:projects.groups  COMPLETED  []</pre><p>Once the Groups were created, I used the Stackdriver Monitoring console to verify that the Apache, qa and prod Groups had been successfully created. Notice that the prod and qa subgroups appear under the Apache group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NN1Jqz-TR8L2_U9y" /></figure><p>This concludes part 1 of the series. Read more about Stackdriver Monitoring Automation in the other posts in the series and references below.</p><ul><li><a href="https://medium.com/@charles.baer/stackdriver-automation-part-2-alerting-policies-9f42068603c4">Stackdriver Automation Part 2: Stackdriver Alerting Policies</a></li><li><a href="https://medium.com/@charles.baer/stackdriver-automation-part-3-uptime-checks-476b8507f59c">Stackdriver Automation Part 3: Stackdriver Uptime Checks</a></li></ul><p>References:</p><ul><li><a href="https://cloud.google.com/monitoring/docs/">Stackdriver Monitoring docs</a></li><li><a href="https://cloud.google.com/monitoring/api/ref_v3/rest/">Stackdriver Monitoring API docs</a></li><li><a href="https://cloud.google.com/monitoring/api/v3/filters">Stackdriver Monitoring filters</a></li><li><a href="https://cloud.google.com/monitoring/api/metrics_gcp">Stackdriver Monitoring metrics</a></li><li><a href="https://cloud.google.com/deployment-manager/docs/">Google Cloud Deployment Manager</a></li><li><a href="https://github.com/GoogleCloudPlatform/deploymentmanager-samples">Deployment Manager Samples</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e51f0aa9b03" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stackdriver-monitoring-automation-part-1-stackdriver-groups-8e51f0aa9b03">Stackdriver Monitoring Automation Part 1: Stackdriver Groups</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Stackdriver Logging and Monitoring Usage]]></title>
            <link>https://medium.com/google-cloud/understanding-stackdriver-logging-and-monitoring-usage-93a28c362c0e?source=rss-91a2d71c8e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/93a28c362c0e</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Charles]]></dc:creator>
            <pubDate>Thu, 16 Aug 2018 19:22:07 GMT</pubDate>
            <atom:updated>2018-12-06T18:41:23.469Z</atom:updated>
            <content:encoded><![CDATA[<p>Stackdriver <a href="https://cloudplatform.googleblog.com/2018/03/announcing-new-Stackdriver-pricing-visibility-for-less.html">announced</a> new pricing in June meant to simplify the pricing structure. Moving to a usage-based pricing model brought Stackdriver in line with other consumption-based cloud products which makes it easier to reason about the cost of using the product.</p><p>I recently went through the GCP bill analysis exercise to understand current and anticipated spend for my projects and was pleased at how easily I was able to find the details that I needed. Though my bill was extremely low, I wanted to know how Stackdriver contributed to the overall bill.</p><p>If you want to dig into your GCP bill and understand your Stackdriver spend, there are a number of different tools that help surface the relevant information. Here are the steps in the analysis that I used to analyze the Stackdriver costs.</p><h3>1. Understand Stackdriver pricing</h3><p>The Stackdriver pricing <a href="https://cloud.google.com/stackdriver/pricing_v2#table">site</a> provides the specifics of the Logging and Monitoring costs and provides a baseline for understanding what drives costs in Stackdriver. There is a great <a href="https://cloud.google.com/logging/docs/export/#overview">diagram</a> called “Life of a Log” that also helps to illustrate Stackdriver Logs ingestion and export.</p><h3>2. GCP Bill Analysis</h3><p>The first place I started was at the billing report for my GCP billing account. The Billing section in the GCP console has a Reports tab that allows you to analyze your bill by viewing your charges over time on a graph. The UI offers a useful range of filters to narrow the results by time, project, products, SKUs and labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/0*2ITqpGDjnjvNo0w-" /></figure><p>In my case, I wanted to understand the Stackdriver charges over time. First, I looked through the previous month and then found a specific week to review. I selected the date range option for the specific week and then removed 1 project from the projects filter that I didn’t want to include in the total.</p><p>I wanted to see my Stackdriver charges linked to this billing account across all projects and so I left the rest of the projects selected. Finally, I selected the Stackdriver Logging and Stackdriver Monitoring under the Products filter. The resulting graph illustrates the charges by project. I noticed that all of my cost was coming from a specific project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9fWWo-PetwJvo7DxWBpvIw.png" /></figure><p>Next, I wanted to see the mix of costs between the Logging and Monitoring products. I changed the Group filter to Product and I saw that Stackdriver Logging was actually the product making up the cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iNFqsZS6nbBgh_lW" /></figure><p>Now that I understood the costs, I looked into which projects and logs were generating log volume.</p><h3>3. Review Stackdriver Logs Ingestion</h3><p>Stackdriver Logging provides detailed list of logs, current log volume and projected monthly volume. This makes it easy to see which logs have the highest volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G23ZqpRG7hcJuHZB" /></figure><p>Additionally, each log entry has a deep link into Stackdriver Monitoring which displays a graph of the volume of the specific log over time. I selected “View detailed usage in Metrics Explorer” for the Kubernetes Cluster metrics and opened the Metrics Explorer dashboard in Stackdriver Monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9JPPDewZt7Nd9FeX" /></figure><p>In the Metrics Explorer, I looked at the volume of logs being ingested over the specific timeframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_VNfzp4P2YFRX8bU" /></figure><p>This analysis helped to answer my question about which logs were generating the most volume and therefore contributing to the overall GCP bill.</p><h3>4. Review Stackdriver Monitoring usage.</h3><p>My project didn’t have significant Stackdriver Monitoring charges though I did find it useful to look at the usage information. In the Stackdriver Monitoring UI, I selected my project name from the drop-down and then the Account Settings option. I then selected the Usage details which displayed the overall usage details for my Stackdriver account.</p><p>The metrics ingested box includes both the current overall metric volume ingested and the end of month projection based on the current ingestion rate. By selecting the project name, the specific types of metrics and their associated volumes are ingested. Both custom metrics and agent metrics incur cost when usage exceeds the monthly free ingestion quota (currently 150MB).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CMIM9dsl5fSCm8Yt2FlJTQ.png" /></figure><p>To explore the metrics ingestion volumes over time, I selected the drop-down next to the project and selected the “View in Metrics Explorer” link which opened the Stackdriver Monitoring Metrics Explorer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rsjbd5HoheKFhE9JtK8chw.png" /></figure><p>In the Metrics Explorer, I could see the volume of metrics being ingested over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jcScheVjxC7ZrKj2ixTYPg.png" /></figure><p>This analysis helped to answer my question about which metrics were generating the most volume and therefore contributing to the overall GCP bill.</p><h3>Conclusion</h3><p>The recent changes to the Stackdriver pricing simplifies understanding the Stackdriver cost on the GCP bill. Stackdriver Logging and Monitoring both provide useful tools to analyze their respective usage. See the references below for additional information.</p><h3>References</h3><ul><li><a href="https://cloudplatform.googleblog.com/2018/03/announcing-new-Stackdriver-pricing-visibility-for-less.html">Blog post announcing new Stackdriver pricing</a></li><li><a href="https://cloud.google.com/stackdriver/pricing_v2">Stackdriver Pricing</a></li><li><a href="https://console.cloud.google.com/billing">Google Cloud Billing</a></li><li><a href="https://console.cloud.google.com/logs/usage">Google Cloud Logging Logs Ingestion</a></li><li><a href="https://app.google.stackdriver.com/settings/accounts/">Stackdriver Monitoring Account Settings</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93a28c362c0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/understanding-stackdriver-logging-and-monitoring-usage-93a28c362c0e">Understanding Stackdriver Logging and Monitoring Usage</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>