<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adam Roberts on Medium]]></title>
        <description><![CDATA[Stories by Adam Roberts on Medium]]></description>
        <link>https://medium.com/@apr_1985?source=rss-3f9b824ebe37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8VHiK-RbBL5Wsz_OoRun5Q.png</url>
            <title>Stories by Adam Roberts on Medium</title>
            <link>https://medium.com/@apr_1985?source=rss-3f9b824ebe37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:07:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@apr_1985/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sending AWS Lambda Logs to an Observability Service Like Datadog or New Relic]]></title>
            <link>https://medium.com/better-programming/sending-aws-lambda-logs-to-an-observability-service-like-datadog-or-new-relic-e75ba8d9e461?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/e75ba8d9e461</guid>
            <category><![CDATA[kinesis-firehose]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[lambda]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Wed, 24 May 2023 17:40:59 GMT</pubDate>
            <atom:updated>2023-05-24T17:40:59.514Z</atom:updated>
            <content:encoded><![CDATA[<h4>Adding Terraform too</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*TJtbivJGkhosLfDM5SvUCg.png" /><figcaption>Lambda logo</figcaption></figure><p>AWS Lambda is a fantastic service for so many different use cases and requirements. A key issue I have with them, though, is logging.</p><p>At the time of writing, there is no way to directly send Lambda logs directly to an observability service such as Datadog or New Relic.</p><p>Currently, Cloudwatch is the only choice for Lambda logging, and while it is another great service and can do everything that you need, a lot of companies will be using a centralised observability service and so having Lambda logs out in their own sink doesn’t make a lot of sense.</p><h3>So, What Can I Do?</h3><p>Fortunately, AWS Cloudwatch provides <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html">Log Subscriptions Filters</a>, which allow logs to be sent to a Kinesis Data Stream, another Lambda, or a Kinesis Data Firehose.</p><p>Up until recently, I used Kinesis Data Stream Filters combined with Logging Destinations (as seen in my article <a href="https://awstip.com/cross-account-logging-in-aws-c49653e7550e">Cross Account Logging in AWS</a>) to send all my logs through a Logstash instance which forwarded onto my observability service.</p><p>This worked great and did the job for several years, and in the main, I totally forgot about log shipping. However, as more stuff moved serverless, I got a bit fed up maintaining infrastructure to forward logs and also required downtime to update Logstash versions roll out new AMIs, security patching, etc., so I looked for a new serverless way to do this.</p><h4>Enter the Firehose</h4><p>I knew I didn’t want to batch send logs with Lambda (especially as the circular pattern of Lambda log shipping with a Lambda melted my brain), so I looked into Firehose and found it comes “bundled” with the config to send logs to most major SaaS observability services. Here’s some examples:</p><figure><img alt="Amazon opensearch serverless, amazon opensearch service, amazon redshift, amazon s3, coralogix, datadog, dynatrace" src="https://cdn-images-1.medium.com/max/553/1*YZ6Dhl49TS9Q8bOC7RbpkQ.png" /><figcaption>Just some of the observability services natively supported by Firehose</figcaption></figure><p>Firehose was very easy to set up in the console, and I had it all configured in a couple of minutes (all the URLs to send to are preconfigured when you choose your provider), and most providers, I am sure, have docs to help with this too (e.g., <a href="https://docs.newrelic.com/docs/logs/forward-logs/stream-logs-using-kinesis-data-firehose/">New Relic</a>, <a href="https://www.datadoghq.com/blog/stream-logs-with-kinesis-firehose-and-datadog/">Datadog</a>).</p><p>After a couple more minutes following the provider docs to create IAM roles and bits and I had a working Firehose.</p><p>The AWS docs could then be followed to point the Cloudwatch Log group at the Firehose using a Subscription Filter.</p><p>Following the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample">Subscription filters with Amazon Kinesis Data Firehose</a> docs from step 6 will get the subscription filter setup and send data through the Firehose.</p><p>As I already have most of this setup for my previous logging solution, it was rather simple just to change a couple of ARNs to point to the new Firehose.</p><p>The last step was to move the pipelines that I had in Logstash to the Observability platforms ingest rules. For me, this was mainly Grok patterns to split the log lines into meaningful data that could be searched and grouped on, and also putting the logs into relevant partitions for the service that Lambda was part of.</p><h3>Job Done Then?</h3><p>Well, in theory, yes… But of course, while doing stuff in the console is fine, the thought of repeating the adding of the log subscription to my triple digits number of Lambdas brought me out in a cold sweat (as did the thought of ever having to make a config change), so it was time to backport into Terraform.</p><h4>Terraforming</h4><p>As with most Terraform jobs, we need to go through everything we did in the console and convert that into Terraform resources.</p><p>Let us start with the main one, the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream">Firehose Delivery Stream</a>.</p><p>In my example, I am sending it to New Relic, but the same config also works for other providers (e.g., Datadog). But do check the Terraform docs in case there are specific instructions for your service, e.g., <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream#splunk-destination">Splunk</a>.</p><pre>resource &quot;aws_kinesis_firehose_delivery_stream&quot; &quot;service_to_observability_firehose&quot; {<br>  name        = &quot;${var.prefix}_to_${replace(var.service, &quot; &quot;, &quot;_&quot;)}_firehose&quot;<br>  destination = &quot;http_endpoint&quot;<br><br>  s3_configuration {<br>    role_arn           = aws_iam_role.firehose_to_s3.arn<br>    bucket_arn         = aws_s3_bucket.firehose_drops.arn<br>    buffer_size        = 10<br>    buffer_interval    = 400<br>    compression_format = &quot;GZIP&quot;<br>  }<br><br>  http_endpoint_configuration {<br>    url                = var.observability_api_url<br>    name               = var.service<br>    access_key         = var.observability_license_key<br>    buffering_size     = 1 # Recommended value from New Relic<br>    buffering_interval = 60<br>    role_arn           = aws_iam_role.firehose_to_s3.arn<br>    s3_backup_mode     = &quot;FailedDataOnly&quot;<br><br>    request_configuration {<br>      content_encoding = &quot;GZIP&quot; # Must be set to GZIP to work with NR<br><br>      dynamic &quot;common_attributes&quot; {<br>        for_each = var.common_attributes<br>        content {<br>          name  = common_attributes.value[&quot;name&quot;]<br>          value = common_attributes.value[&quot;value&quot;]<br>        }<br>      }<br>    }<br>  }<br><br>  tags = {<br>    &quot;name&quot; = &quot;${var.prefix}_to_${replace(var.service, &quot; &quot;, &quot;_&quot;)}_firehose&quot;<br>  }<br>}</pre><p>To get the observability_api_url you can use the AWS console, go to manually create a stream for your observability services, and you will find the correct URL there (or check your provider&#39;s docs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*APuXcp3Q-nz7EjJsKV-Fig.png" /><figcaption>Example of finding the API to send logs to</figcaption></figure><p>Interestingly, the name in the http_endpoint_configuration needs to be the same as HTTP endpoint name in the console (i.e., New Relic in this case). Which isn’t very obvious from the TF docs.</p><p>As can be seen from the code, other resources need to be created, too, namely the S3 bucket and an IAM role for the Firehose to write to S3 (which were probably created manually earlier).</p><p>These are standard resources that I will assume you have Terraformed before, and the config covered in the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample">AWS docs</a>, so I won&#39;t go into details on how to create these other than to say the Assume Role Policy should look something like this:</p><pre>data &quot;aws_iam_policy_document&quot; &quot;firehose_to_s3_assume_policy&quot; {<br>  statement {<br>    effect  = &quot;Allow&quot;<br>    actions = [&quot;sts:AssumeRole&quot;]<br><br>    principals {<br>      type        = &quot;Service&quot;<br>      identifiers = [&quot;firehose.amazonaws.com&quot;]<br>    }<br>  }<br>}</pre><h4>Subscription filters</h4><p>Now that the Firehose is set up and ready (you can use the test function on the console to check it is still working), it needs some Cloudwatch Log Groups to send data to it.</p><p>The Terraform for the subscription filters is pretty simple and, for me, forms part of my Lambda module so that every Lambda I create automatically gets the log shipping applied.</p><pre>resource &quot;aws_cloudwatch_log_subscription_filter&quot; &quot;kinesis_log_subscription&quot; {<br>  name            = &quot;${var.lambda_name}_subscription&quot;<br>  log_group_name  = aws_cloudwatch_log_group.lambda_log_group.name<br>  role_arn        = var.cw_to_firehose_role_arn<br>  filter_pattern  = &quot;&quot;<br>  destination_arn = var.firehose_arn<br>}</pre><p>The role ARN is the role that was created earlier in the AWS docs to allow the Principal {“Service”: “logs.amazonaws.com”} to be allowed to put records on the Firehose.</p><p>I created this role so that any logs can write to the Firehose. This means the role can be created once and passed into a Lambda module as a var (as above) rather than having a role for every Lambda created (which may be more appropriate in shared accounts or high-security scenarios).</p><p>The destination_arn is the ARN of the Firehose.</p><p>Again, I pass this in as a variable as I like to keep the Firehose separate from the Lambdas. As it is infrastructure, I don’t expect to change often, so planning it on every Terraform run doesn&#39;t make sense.</p><h3>Conclusion</h3><p>This is a pretty simple to set up serverless solution which will allow you to have all your logs in a single place. Most Observability platforms allow you to enrich or parse the data on ingest to the platform, which means that the Groking and partitioning (and drops) can be done at the platform&#39;s edge too, so there really isn’t much of a downside to this pattern.</p><p>Getting the Terraform right is key to maintainability, and having layered modules and states will ensure all your Lambdas get their logs shipped with minimal effort and makes config changes quick and simple to roll out.<br>Even moving from one Observability platform to another could be done simply by updating where the Firehose points to (so long as the resource name doesn’t change), which is really about as flexible as you can get.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e75ba8d9e461" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/sending-aws-lambda-logs-to-an-observability-service-like-datadog-or-new-relic-e75ba8d9e461">Sending AWS Lambda Logs to an Observability Service Like Datadog or New Relic</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing Grafana Dashboards With Terraform]]></title>
            <link>https://medium.com/better-programming/managing-grafana-dashboards-with-terraform-ad49ff6bb552?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad49ff6bb552</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[grafana]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Wed, 01 Feb 2023 12:38:59 GMT</pubDate>
            <atom:updated>2023-02-16T15:16:36.648Z</atom:updated>
            <content:encoded><![CDATA[<h4>We’ve all done it — deleted a graph from a dashboard, realised we still need it but have forgotten the query. Use Terraform to go back in time and save yourself the headache</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fohvAc3UfU-IZBzNmoJchg.png" /><figcaption>Grafana logo</figcaption></figure><p>Grafana is a great visualisation tool. It is powerful, flexible, and generally easy to use.</p><p>However, manually administering the dashboards has some cons, for example, accidental deletions of graphs, people “testing” changes and forgetting to clean them up, and general DR capabilities.</p><p>Join the Config as Code revolution and store your dashboards as code for ultimate peace of mind.</p><h3>Terraform</h3><p>Terraform is probably the most used solution for As Code, and the Grafana project has a Terraform provider to simplify managing Grafana. Here’s a <a href="https://registry.terraform.io/providers/grafana/grafana/">list of those providers</a>.</p><p>This won&#39;t be a full Terraform tutorial and will assume some knowledge of Terraform.</p><h3>Let’s Get Started</h3><h4>Authentication</h4><p>In Terraform, we need to add the Grafana provider to give us access to the Grafana resources. Here’s how to do that:</p><pre>// Update versions as appropriate for time of reading<br>terraform {<br>  required_version = &quot;&gt;= 1.3.0&quot;<br>  required_providers {<br>    grafana = {<br>      source  = &quot;grafana/grafana&quot;<br>      version = &quot;1.34.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;grafana&quot; {<br>  url  = &quot;http://grafana.example.com/&quot;<br>  auth = var.grafana_auth<br>}</pre><p>In this example, the url is the URL of your Grafana instance and auth is an acceptable way to<a href="https://registry.terraform.io/providers/grafana/grafana/latest/docs#authentication"> authenticate with Grafana</a>.</p><p>As you can see, we need some Terraform credentials to authenticate Grafana.</p><p>You could use your username and password in a basic auth combination, e.g., username:password, but a better practice is to use an API key.</p><p>Recent versions of Grafana API keys have been replaced with <a href="https://grafana.com/docs/grafana/latest/administration/service-accounts/">Service Accounts</a>, so we need to set one up to use Terraform.</p><blockquote>There is a bit of a chicken and egg dilemma here. Ideally, you would use <a href="https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/service_account">Terraform to manage your Grafana Service Accounts</a>. but we need to manually create one to use Terraform. Once Terraform is up and running, you could replace the manually created account with one managed by Terraform.</blockquote><p>In Grafana, click on the settings (cog) icon from the LHM and choose Service Accounts from the menu ribbon.</p><p>Choose Add service account, and call it whatever you want, e.g., Terraform and give it Editor permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*QubzA1IEBOq8iuTlsTx-AA.png" /><figcaption>Creating Service Account in Grafana UI</figcaption></figure><p>Now, choose Add service account token and set the expiry date as appropriate for your security policies.</p><p>Copy the token and store it inline with your security policies too. This can now be added to the Terraform provider auth section (or in a var) we started above.</p><blockquote>The token can be added to an Environment Variable called GRAFANA_AUTH and the auth argument be omitted from the provider for a more secure, conscious, and flexible approach</blockquote><p>Now you can set up the Terraform state as required and run the terraform init</p><h3>The Dashboard</h3><p>Dashboards in Grafana are stored internally as JSON, and it is this JSON that Terraform uses to represent the dashboards as code.</p><p>For me, the best way to start looking after a dashboard in Terraform is to create it in Grafana manually, export the JSON, and import it into Terraform.</p><p>Here, I have manually created a basic dashboard with a couple of panels in Grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*d8YUTiDe008q0jteTFN2cg.png" /><figcaption>Basic Dashboard</figcaption></figure><p>To export the JSON, click on the dashboard settings (cog) icon near the top right and then choose JSON Model from the menu.</p><p>From here, copy all the JSON into a file in your Terraform directory and name it as you choose.</p><h3>Terraform Resource</h3><p>In the Terraform code, we can now add our dashboard resource:</p><pre>resource &quot;grafana_dashboard&quot; &quot;terraform_maintained&quot; {<br>  config_json = file(&quot;tf_dashboard.json&quot;)<br>}</pre><p>Now my folder structure looks like the following:</p><pre>.<br>├── dashboard.tf<br>├── providers.tf<br>├── terraform.tfstate<br>└── tf_dashboard.json</pre><p>If we try and do a Terraform apply here, an error will be thrown with a version mismatch. It is an odd error as it says it cannot create the dashboard because it already exists.</p><pre>Error: status: 412, body: {&quot;message&quot;:&quot;The dashboard has been changed by someone else&quot;,&quot;status&quot;:&quot;version-mismatch&quot;}</pre><p>As stated earlier, we need to <a href="https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/dashboard#import">import</a> the dashboard that’s been manually created into Terraform (see the docs on how to import if using Grafana organizations).</p><pre>terraform import grafana_dashboard.terraform_maintained GC797v0Vz</pre><p>Once imported, the Terraform will apply.</p><h3>Drift Detection and Correction</h3><p>Making a manual change to the dashboard will result in Terraform drift (where Terraform shows a change in the configuration).</p><p>Let&#39;s say someone was testing a change on the dashboard and forgot to remove it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*uL7cXTJwLB5WEYqruyQlzw.png" /><figcaption>Change to dashboard</figcaption></figure><p>Running a Terraform plan will show the delta between what is stored in the JSON file and what is live (I won&#39;t show this as it is quite long).</p><p>Running Terraform apply again will set the Dashboard back to how it is defined in the stored JSON before the changes.</p><p>We can even delete the whole dashboard. Running Terraform apply will restore it for us.</p><p>This same method can be used if you mess something up, want to restore it, and it can also be used to move from one Grafana instance to another (by updating where the provider points to).</p><h3>Updating the JSON</h3><p>Of course, there will be times when changes do need to be made to the dashboard. These can either be done in the JSON file and rolled out with Terraform or made manually. After that, the JSON can be re-exported and updated in the Terraform location.</p><h3>Conclusion</h3><p>Maintaining dashboards in Terraform is a simple task. One day, it will dig you out of a deep hole, so there is no reason not to do it :).</p><p>I would look at storing as much of your Grafana config in Terraform for the same drift and DR reasons, so check out the full provider docs to see what can be done.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad49ff6bb552" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/managing-grafana-dashboards-with-terraform-ad49ff6bb552">Managing Grafana Dashboards With Terraform</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pushgateway: Getting Prometheus metrics from Ephemeral sources]]></title>
            <link>https://blog.devops.dev/pushgateway-getting-prometheus-metrics-from-ephemeral-sources-70c97747267e?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/70c97747267e</guid>
            <category><![CDATA[pushgateway]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Mon, 30 Jan 2023 12:06:47 GMT</pubDate>
            <atom:updated>2023-02-07T18:11:28.495Z</atom:updated>
            <content:encoded><![CDATA[<h4>Sometimes there are situations where Prometheus’ pull method is not suitable for getting metrics and a push method is more desirable. This is where Pushgateway comes in.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/1*QR6pqHxgV6NQCUgfMRg_YQ.png" /></figure><p>We all know how good Prometheus is for gathering and aggregating metrics from your estate. But what about if you have something that does not natively expose Prometheus metrics e.g.</p><ul><li>Third party tooling</li><li>Legacy code which cannot be instrumented</li><li>Tooling that does not run constantly</li><li>Ephemeral jobs like scripts or CI Runs?</li></ul><p>Often in these situations a custom exporter that runs alongside the application is a good way forward. But there are times when metrics are better pushed than pulled.</p><h4>What is Pushgateway?</h4><p><a href="https://github.com/prometheus/pushgateway">Pushgateway</a> is a standalone service which can have jobs and metrics posted to it (which can be done through the Prometheus libraries or http), and exposes them for scraping by a Prometheus server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*s6Lr-_TfO7rlXF3KCR77zw.png" /></figure><h4>Give me an example</h4><p>I use Pushgateway for monitoring the costs of my AWS accounts.</p><p>I want a metric that can be scraped by Prometheus regularly so that trend graphs look smooth. But Cost Explorer is charged each time it is queried and the account costs are only updated once per day so running a traditional exporter would be expensive just to report the same stat over and over.<br>Pushgateway fits my needs here as I can send the cost updates once per day via a Jenkins job but still have an endpoint that Prometheus can scrape at all times.</p><p>As the Pushgateway code is included in the Prometheus clients it can be done in Python quite easily.</p><blockquote>The actual script gathers much more cost data and for more accounts</blockquote><pre>from dateutil.relativedelta import relativedelta<br>from datetime import date, datetime<br>from prometheus_client import CollectorRegistry, Gauge, push_to_gateway<br>import boto3<br>import logging<br><br>client = boto3.client(&#39;ce&#39;, region_name=&#39;eu-west-1&#39;)<br>registry = CollectorRegistry()<br><br>#############################################<br># Required Date objects<br>#############################################<br>today = date.today()<br>yesterday = today - relativedelta(days=1)<br>tomorrow = today + relativedelta(days=1)<br><br>#############################################<br># Amount spent yesterday<br>#############################################<br>logging.info(&quot;Gathering yesterdays spending&quot;)<br>yesterdays_spend = Gauge(<br>    &quot;YesterdaysSpending&quot;,<br>    &#39;Help text&#39;,<br>    [&#39;period&#39;, &#39;account&#39;],<br>    registry=registry<br>)<br>response = client.get_cost_and_usage(<br>    TimePeriod={<br>        &quot;Start&quot;: str(yesterday),<br>        &#39;End&#39;: str(today)<br>    },<br>    Granularity=&#39;DAILY&#39;,<br>    Metrics=[&quot;BlendedCost&quot;]<br>)<br>yesterdays_spend.labels(<br>    period=&#39;yesterday&#39;,<br>    account=args.aws_account<br>).set(response[&#39;ResultsByTime&#39;][0][&#39;Total&#39;][&#39;BlendedCost&#39;][&#39;Amount&#39;])<br><br>#############################################<br># Push to Gateway<br>#############################################<br>logging.info(&quot;Pushing to pushgateway&quot;)<br>push_to_gateway(&quot;https://pushgateway.domain.systems&quot;, job=&#39;aws-billing-aws-something-np&#39;, registry=registry)</pre><p>The stats are sent to the Pushgateway in the last line, tied to the job specified (if you don’t specify a job when sending to Pushgateway the metrics will be listed under the pushgateway job).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ5o5UFacVtQmnZ5QnfhAA.png" /><figcaption>Metric Sent to Pushgateway</figcaption></figure><p>Prometheus is configured to scrape the Pushgateway same as any other target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*nEjcnmtFxo9qes1qsOBbbA.png" /><figcaption>Prometheus targeting the Pushgateway</figcaption></figure><p>The information can then be shown in Grafana (or tooling of your choice) to show the latests costs and see the trends using different graph types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*73RyFZHJZv_GYt1-rYDmBA.png" /><figcaption>Stat in Grafana</figcaption></figure><p>Alerts can also be setup in <a href="https://prometheus.io/docs/alerting/latest/alertmanager/">Alertmanger</a> (again or tooling of your choice) if there is an unusually high spend or trend.</p><p>Another usage could be for tracking the outputs of a script, for example a Lambda that cleans up deactivated AD users from a system might report back how many users were cleaned on the last run.<br>Or a batch processing script could send the time taken to process the latest batch.</p><h4>Gotchas</h4><p>Pushgateway never forgets. Once a stat has been added to Pushgateway it is there until explicitly deleted through the API. This means that Prometheus will scrape that stat even if it is no longer required or relevant e.g. that login Lambda has been retired for a Kube micro service the lambda stats will still be on the pushgateway.</p><p>If the service sending to Pushgateway is down the stats on Pushgateway will still be scraped. With a normal Prometheus metrics page if the service is down often the metrics page will also be down which means that Prometheus won’t gather any stats and can also report that the service is unreachable. Using Pushgateway, Prometheus will carry on scrapping the stats without any knowledge that they are totally wrong leading to misleading and incorrect data.</p><p>A Pushgateway server can only be setup once in the Prometheus configuration so all jobs sent to that Pushgateway will be scraped with the same intervals and settings. <br>To get round this you could setup multiple Pushgateways depending on the require scrape config.</p><p>The timestamp used by the Prometheus ingestion will be the timestamp when Prometheus scrapes rather than the time the metric was pushed to the Gateway. This seems odd at first, but if it used the timestamp of the push then Prometheus would be scraping the same metric with the same timestamp over and over until it was updated which would be inconsistent with how Prometheus scrapes exporters.</p><h4>Conclusion</h4><p>The Pushgateway is a really handy service to complete your Prometheus monitoring where you don’t want to or can’t run an exporter for a service. It is easy to use and built into the Prometheus clients. So long as your are aware of the gotchas and guard around them then it can really fill the gaps in your coverage.</p><p>Further Reading</p><ul><li><a href="https://github.com/prometheus/pushgateway">https://github.com/prometheus/pushgateway</a></li><li><a href="https://prometheus.io/docs/practices/pushing/">https://prometheus.io/docs/practices/pushing/</a></li><li><a href="https://prometheus.io/">https://prometheus.io/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70c97747267e" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/pushgateway-getting-prometheus-metrics-from-ephemeral-sources-70c97747267e">Pushgateway: Getting Prometheus metrics from Ephemeral sources</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resilient Vault with ASG, KMS Auto-Unseal and S3 Storage]]></title>
            <link>https://awstip.com/resilient-vault-with-asg-kms-auto-unseal-and-s3-storage-dcb07b4a8cab?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcb07b4a8cab</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[devsecops]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Wed, 21 Dec 2022 16:01:51 GMT</pubDate>
            <atom:updated>2022-12-21T17:57:54.367Z</atom:updated>
            <content:encoded><![CDATA[<h4>Keeping Vault available but easy to manage can be achieved with AWS toolings</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4mioAx9gs6BeYrSO3EonQ.jpeg" /></figure><p>I am a big fan of Hashicorp Vault (I even <a href="https://medium.com/p/e83577958e5b">run it on my dev box</a>). <br>But for bigger installations more thought needs to be put into availability and recovery. Using AWS tooling we can achieve this goal easily even on a single node cluster.</p><p>A major piece of modern architecture is a cattle over pets model. If our Vault node becomes unreachable, restarts unexpectedly or is terminated we don’t want to have to manually intervene to get the service back online. We want to rely on AWS tooling to do this for us. So the idea here is to have an immutable instance of Vault which if it is killed for any reason will bring itself back up and online in a fully working state but with minimal maintenance overheads.</p><blockquote>Note: this setup will involve downtime if there is an issue with Vault or the EC2 instance. For super high uptime installations please look at clustering Vault with a high availability backend.</blockquote><h4>1. The Auto Scaling Group</h4><p>As the plan is to only run a single node cluster (in most companies I have worked a few minutes of MTTR is acceptable), using an auto scaling group allows the Vault to be “self-healing”, in that if it goes down for any reason a new instance will be spun up and the User Data scripts run again.</p><p>This keeps the availability high for unexpected issues, and also allows upgrades to take place at the operators command by either minting a new AMI (see <a href="https://www.packer.io/">Packer</a>) and updating the Launch configuration/template or updating the User Data to pull in different versions. <br>A prebuilt AMI will likely load faster than configuring via User Data so bear this in mind when decided how to setup Vault.</p><p>Using an ASG also allows Vault to run on SPOT so that if there are budget restrictions, or running a dev Vault system this can be done in the cheapest way and a new instance will be spun up again after a SPOT termination.</p><p>Of course if you have a Kubernetes cluster available then it <strong>might</strong> be more appropriate to run an immutable Vault on there …. Just don’t store the secrets you need to build the cluster on the Vault ;) <br>I tend to keep Vault outside of the Kube cluster purely from a separation of concerns POV.</p><h4>2. S3 Storage Backend</h4><p>The common backend Vault uses is the filesystem (or Raft) for storing data. This wont give us the simple immutability that we are looking for as we would lose the data for our Vault if the instance was terminated.</p><p>We could use DynamoDB, and this would be the choice for clustered scenarios, but for a single node, S3 is much easier to configure with practically zero maintenance required.</p><p>To setup the S3 backend you will need to add the <strong>storage “s3”</strong> stanza to your HCL config file.</p><p>In its simplest form that could look like this.</p><pre>storage &quot;s3&quot; {<br>  access_key = &quot;abcd1234&quot;<br>  secret_key = &quot;defg5678&quot;<br>  bucket     = &quot;my-bucket&quot;<br>}</pre><p>If you are using an IAM on your Vault node you can omit the access_key and secret_key e.g.</p><pre>storage &quot;s3&quot; {<br>  bucket     = &quot;my-bucket&quot;<br>  region     = &quot;eu-west-1&quot;<br>  path       = &quot;vault/data/&quot;<br>}</pre><p>For a full range of options for the backend see <a href="https://developer.hashicorp.com/vault/docs/configuration/storage/s3">the docs</a></p><h4>3. KMS Auto-Unseal</h4><p>A headache of a default setup Vault is that on reboot you need to enter in a set number of unseal keys to be able to bring Vault back online. While this is super secure, in most environments this is overkill and doesn’t fit the cattle model we are looking for.</p><p>Vault allows itself to be automatically unsealed using AWS KMS. <br>Once you have created the KMS key you will need to give your Vault instance IAM permissions on that key.</p><p>An example Terraform policy would be</p><pre>data &quot;aws_iam_policy_document&quot; &quot;vault-kms-unseal&quot; {<br>  statement {<br>    sid       = &quot;VaultKMSUnseal&quot;<br>    effect    = &quot;Allow&quot;<br>    resources = [aws_kms_key.vault.arn]<br><br>    actions = [<br>      &quot;kms:Encrypt&quot;,<br>      &quot;kms:Decrypt&quot;,<br>      &quot;kms:DescribeKey&quot;,<br>    ]<br>  }<br>}</pre><p>and then you will need to set the <strong>seal &quot;awskms&quot;</strong> stanza in your config file.</p><pre>seal &quot;awskms&quot; {<br>    region = &quot;eu-west-1&quot;<br>    kms_key_id = &quot;1a111b22-a123-b123-c123-333e9ca99999&quot;<br>}</pre><h4>4. Cloudwatch</h4><p>As we are using AWS services, might as well log to AWS too.</p><p>Using the AWS <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html">Cloudwatch Agent</a> logs can be sent from the Vault node to Cloudwatch.</p><p>A <a href="https://developer.hashicorp.com/vault/docs/audit/file">file audit device</a> will need to be configured to allow Vault to log to a path that the Cloudwatch agent will collect from.</p><pre>vault audit enable file file_path=/var/log/vault_audit.log</pre><h4>Conclusion</h4><p>Combining these three AWS tools we can run a single node Vault cluster in a way that it can be treated like cattle and will withstand termination but still remain secure and usable.</p><blockquote>Detailed Configuration: I haven’t gone into any details about how things should be setup as there are too many variables to consider e.g. Public or Private subnets, run in a container or install via yum, Amazon Linux or Debian, Packer or Userdata etc etc. The idea was to give an overview architecture and let you choose how to implement it.</blockquote><blockquote>Security: for the same reasons as above I have not given any security configurations as your company might have totally different policies to how I would do it. So please ensure you secure S3 buckets, Security Groups etc inline with best practice and your policies.</blockquote><p>References:</p><ul><li><a href="https://developer.hashicorp.com/vault/tutorials/auto-unseal/autounseal-aws-kms#step-1-provision-the-cloud-resources">https://developer.hashicorp.com/vault/tutorials/auto-unseal/autounseal-aws-kms#step-1-provision-the-cloud-resources</a></li><li><a href="https://developer.hashicorp.com/vault/docs/configuration/storage/s3">https://developer.hashicorp.com/vault/docs/configuration/storage/s3</a></li><li><a href="https://developer.hashicorp.com/vault/docs/audit/file">https://developer.hashicorp.com/vault/docs/audit/file</a></li><li><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html">https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html</a></li><li><a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html">https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html</a></li><li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html">https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html</a></li><li><a href="https://www.packer.io/">https://www.packer.io/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcb07b4a8cab" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/resilient-vault-with-asg-kms-auto-unseal-and-s3-storage-dcb07b4a8cab">Resilient Vault with ASG, KMS Auto-Unseal and S3 Storage</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 reasons I run Vault on my dev box]]></title>
            <link>https://medium.com/@apr_1985/5-reasons-i-run-vault-on-my-dev-box-e83577958e5b?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/e83577958e5b</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[vault]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Tue, 13 Dec 2022 15:30:23 GMT</pubDate>
            <atom:updated>2022-12-13T16:06:32.867Z</atom:updated>
            <content:encoded><![CDATA[<h4>Keeping all your API tokens and passwords safe on your dev box can be tricky. Vault to the rescue</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z4mioAx9gs6BeYrSO3EonQ.jpeg" /><figcaption><a href="https://www.hashicorp.com/products/vault">https://www.hashicorp.com/products/vault</a></figcaption></figure><p>Vault is an enterprise grade secrets management tool from Hashicorp. But that doesn’t mean it can’t be used for protecting secrets on a dev box too :)</p><h4>1. I can access all my API tokens in scripts without ever seeing them</h4><p>Keeping my API and access tokens in my Vault container allows me to run scripts and use the tokens without loading them into memory or copy and pasting them around the place (leaving them in shell history etc).</p><p>For example to grab a file from Artifactory:</p><pre>curl \<br>-H &quot;X-JFrog-Art-Api:$(vault kv get -mount=kv -field=token af_api)&quot; \<br>-O &quot;https://artifactory.my.company/artifactory/repo/test.txt&quot;</pre><p>Or running a Script:</p><pre>python run_a_script.py \<br>--username=$(vault kv get -mount=kv -field=username serviceaccount) \<br>--password=$(vault kv get -mount=kv -field=password serviceaccount) </pre><p>There are also SDKs for most languages for getting credentials inside the script if you know that it will only be run on your dev box.</p><h4>2. Getting short term creds for Databases</h4><p>While I have administrator access to databases most of the time I really do not require that level of access, in fact most of the time readonly is sufficient. I don’t want to be managing all these credentials and again I don’t want valid credentials hanging around in my shell or env vars.</p><p>Using Vault I can get short lived credentials for the databases that I have access to so I have no worries about leaving credentials being visible as they will have expired before they can do any damage</p><pre>❯ vault read database/creds/postgresRO<br>Key                Value<br>---                -----<br>lease_id           database/creds/postgresRO/vwYp1xYyUKJnImyHdjZiJOlK<br>lease_duration     1h<br>lease_renewable    true<br>password           AQhW6V-G8gvyesjRWhfq<br>username           v-root-postgres-ZuPRlDkAA4PXGgCtMpXU-1670941588</pre><p>If a credential does escape before it expires it can easily be revoked too</p><pre>vault lease revoke database/creds/readonly/vwYp1xYyUKJnImyHdjZiJOlK</pre><h4>3. Protecting SSH keys on my box</h4><p>I have a bunch of keys that are used to SSH on to various boxes in the estate. These keys can be encrypted using the the Transit secrets engine so that they do not need storing in plaintext on the box but can be loaded into SSH agent when required.</p><p>Setup the Transit engine and create a key</p><pre>vault secrets enable transit<br>vault write -f transit/keys/sshkeys</pre><p>To encrypt an ssh key with a Transit key</p><pre>vault write -field=ciphertext \<br>transit/encrypt/sshkeys \<br>plaintext=$(cat ssh.key |base64) &gt;&gt; encodedssh.pem</pre><p>Then you can delete the unencrypted file. <br>To use the key you can then run</p><pre>ssh-add - &lt;&lt;&lt; $(vault write -field=plaintext transit/decrypt/sshkeys ciphertext=@encodedssh.pem | base64 -d)<br>ssh ec2-user@1.2.3.4</pre><h4>4. Sealing the Vault when required</h4><p>When I finish for the day, if I need to let someone have access to my computer (tech support or some pair programming), if I am using public wifi or if I think there is a compromise on my box I can seal the Vault and no one can get to my tokens etc.</p><pre>vault operator seal</pre><h4>5. It’s easy, so why not?</h4><p>Setting up the Vault is just a small docker-compose file</p><pre>version: &#39;2&#39;<br>services:<br>  vault:<br>    image: vault<br>    container_name: vault<br>    ports:<br>      - &quot;8200:8200&quot;<br>    restart: always<br>    volumes:<br>      - ./volumes/logs:/vault/logs<br>      - ./volumes/file:/vault/file<br>      - ./volumes/config:/vault/config<br>    cap_add:<br>      - IPC_LOCK<br>    entrypoint: vault server -config=/vault/config/vault.jsonya</pre><p>and config file</p><pre>{<br>  &quot;backend&quot;: {<br>    &quot;file&quot;: {<br>      &quot;path&quot;: &quot;/vault/file&quot;<br>    }<br>  },<br>  &quot;listener&quot;: {<br>    &quot;tcp&quot;:{<br>      &quot;address&quot;: &quot;0.0.0.0:8200&quot;,<br>      &quot;tls_disable&quot;: 1<br>    }<br>  },<br>  &quot;ui&quot;: true<br>}</pre><blockquote>Credit to <a href="https://blog.ruanbekker.com/blog/2019/05/06/setup-hashicorp-vault-server-on-docker-and-cli-guide/">https://blog.ruanbekker.com/blog/2019/05/06/setup-hashicorp-vault-server-on-docker-and-cli-guide/</a><br>Check out the page for full container install and basic Vault usage guide</blockquote><p>and you are away safely storing secrets and credentials with easy access for your scripts.</p><p>There is plenty more that Vault can do. E.g. if you are using Static IAM credentials you can use Vault to automatically rotate these for you and even (like the database) provide short lived credentials to certain IAM roles so that you do need to be in full power mode all the time.<br>If you haven’t already it is a tool well worth checking out.</p><p><em>Note: Lots of these commands can be quite long so invest some time in making some alias’ for them and you’ll find it much easier to adopt</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e83577958e5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using GitHub Actions OIDC to run Terraform in AWS]]></title>
            <link>https://awstip.com/using-github-actions-oidc-to-run-terraform-in-aws-31ba395518cb?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/31ba395518cb</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[oidc]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Wed, 07 Dec 2022 15:13:07 GMT</pubDate>
            <atom:updated>2022-12-09T11:39:30.012Z</atom:updated>
            <content:encoded><![CDATA[<h4>Forget long lived AWS credentials, use OIDC to secure your credentials in under 100 lines of Terraform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z_N4ySOLaRQ6Sl9cFmoWAA.png" /><figcaption>OIDC Flow</figcaption></figure><p>Running Terraform against AWS is a prime use case for GitHub Actions (who wants to run Terraform on their own box), but storing long lived AWS credentials in the GitHub secret store isn’t really great practice. Be honest how often are you rotating those credentials across your repos…</p><p>Using OIDC allows you to request short lived tokens from AWS when you need them.</p><p>I wont go into the ins and outs of OIDC and what it is or how it works, there are far more qualified people out there who have written far better articles (including <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect">GitHub</a> and <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html">AWS</a> themselves) but I can show you how to get up and running with it using Terraform for the heavy lifting.</p><h4>The Terraform</h4><p>Let us get straight into the Terraform and go from there.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7754d017a6980d10c91bb5aef5c912c/href">https://medium.com/media/a7754d017a6980d10c91bb5aef5c912c/href</a></iframe><blockquote>Note: this is sample code, it could be made a lot cleaner and use more up-to-date TF syntax such as<strong> data “aws_iam_policy_document”</strong> but I feel this is easier to read for examples</blockquote><p>As you can see the required code to get up and running is quite minimal.<br>You create and OIDC provider using the GitHub Cert fingerprint, attach a role to the provider which is limited to the repo(s) that you want to allow access to you AWS account and you attach whatever permissions to the role that is required to get the job done.</p><h4>The GitHub Workflow</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f94a0186d67886367381955eb09605e6/href">https://medium.com/media/f94a0186d67886367381955eb09605e6/href</a></iframe><p>The parts of the workflow to note here are the permissions <strong>id-token: write</strong> which is required to allow GHA to require the token. And the call to the action <strong>aws-actions/configure-aws-credentials</strong> as this is what will get the token and assume the role created in the Terraform earlier.</p><p>Once the OIDC Terraform written above is applied the workflow is good to go and so long as you granted the role the permissions that the job Terraform requires then it should now be able to plan (and apply) in your AWS account.</p><p>That is much better than long lived credentials ;)</p><blockquote>The way I have this setup is to allow the OIDC role to assume other roles in the account for specific tasks. This means that I don’t need to have an Uber powerful IAM role that can do anything that any job requires but have lots of smaller IAM roles that individual workflows can assume after getting the OIDC role.</blockquote><h4>Further Reading</h4><p>The GitHub offical guide to setting up OIDC with AWS (note they miss the step to attach S3 permissions to the role)</p><p><a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services">Configuring OpenID Connect in Amazon Web Services - GitHub Docs</a></p><p>Security Hardening Guide for OIDC which details how to secure the relationship further, but also gives useful advice on the <strong>Conditions</strong> for the assume policy e.g. being able to wildcard a bunch of repos (hopefully your org has a naming convention).</p><p><a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect">About security hardening with OpenID Connect - GitHub Docs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31ba395518cb" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/using-github-actions-oidc-to-run-terraform-in-aws-31ba395518cb">Using GitHub Actions OIDC to run Terraform in AWS</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test Results on GitHub Actions Summary Page]]></title>
            <link>https://blog.devops.dev/test-results-on-github-actions-summary-page-e60864ed1c79?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/e60864ed1c79</guid>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 16:06:02 GMT</pubDate>
            <atom:updated>2023-02-21T18:30:53.001Z</atom:updated>
            <content:encoded><![CDATA[<h4>Fed up of having to dive deep into the Actions pages and logs to find linting and test errors. Here is a quick way to show them on the summary page</h4><p>Just a quick article on how to surface test results (or indeed anything you want) to the summary page of an Actions run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pEHALTZXa-JFqNqG_zH7Yg.png" /></figure><p>GitHub Actions has a feature called <a href="https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#environment-files">Environment Files</a>. This (at the time of writing) is the current way to set environment variables and output parameters which are accessible to other steps.<br>It is also how you can set <a href="https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-job-summary">Job Summaries</a>.</p><p>You can send any markdown that you want into the $GITHUB_STEP_SUMMARY environment file and it will be rended to the summary page at the end of the job.</p><p>For example to render the above my workflow looks like this</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ff31a146476d497b1a5c9b6b260f492/href">https://medium.com/media/3ff31a146476d497b1a5c9b6b260f492/href</a></iframe><p>Yes it is a little messy at the moment but I don’t really mind as it is a simple enough workflow not to worry.</p><p>If you are an Actions writer then you can create Job Summaries using the core.summary utility in <a href="https://www.npmjs.com/package/@actions/core">@actions/core</a> npm package.</p><p>If you use these already feel free to share how in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e60864ed1c79" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/test-results-on-github-actions-summary-page-e60864ed1c79">Test Results on GitHub Actions Summary Page</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Parameters and Secrets Lambda Extension]]></title>
            <link>https://awstip.com/aws-parameters-and-secrets-lambda-extension-806d457bb4de?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/806d457bb4de</guid>
            <category><![CDATA[aws-parameter-store]]></category>
            <category><![CDATA[aws-secrets-manager]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Mon, 31 Oct 2022 11:23:37 GMT</pubDate>
            <atom:updated>2022-11-01T16:53:39.302Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to get Parameters and Secrets without requiring a full SDK built into your Lambda</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*TJtbivJGkhosLfDM5SvUCg.png" /></figure><p>One of the tricky parts of Lambda is managing Parameters and Secrets. You need to add a full SDK e.g. Boto3 just to get some parameters which bloats the Function and also depending on your Function can increase costs where the parameters are got over and over again leading to increase API costs.</p><p>AWS has released a new extension (layer) that aims to remove both these issues. <br><a href="https://aws.amazon.com/about-aws/whats-new/2022/10/aws-parameters-secrets-lambda-extension/">The AWS Parameters and Secrets Lambda Extension</a> uses conventional GET requests to make calls to parameter store or secrets manager so you no longer need the SDK and can also cache them so that future requests do not cost and are quicker.</p><h3>How to use it</h3><h4>Adding the Layer</h4><p>First thing you need to do is add the layer to you Lambda.<br>This can be done through the UI by searching for version 2 (at time of writing) of the AWS-Parameters-and-Secrets-Lambda-Extension and adding it there.</p><p>Or can be done in Terraform by adding the ARN (found <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html#ps-integration-lambda-extensions-add">here</a> for your region).<br>e.g.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70d380ff0fa4edb8a56f4679199d22cd/href">https://medium.com/media/70d380ff0fa4edb8a56f4679199d22cd/href</a></iframe><h4>IAM Permissions</h4><p>Assuming you already have an SDK you are looking to remove then you should already have the correct IAM permissions to get parameters or secrets.</p><p>Otherwise you will need to add some permissions to the Lambdas execution role that allows it to access the required parameters or secrets.<br>e.g. the full IAM to access parameters under the /lambda/ path</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd6e7f9d8d2dfadd860f6dee257dd935/href">https://medium.com/media/cd6e7f9d8d2dfadd860f6dee257dd935/href</a></iframe><p>You will need to modify this as appropriate for Secrets Manager or other paths etc.</p><h4>Making a call in the Lambda</h4><p>So with the layer available and the correct IAM in place you can now make the calls just using standard HTTP GET requests.</p><p>The path you need to make the requests to is</p><pre># Parameters<a href="http://localhost:2773/systemsmanager/parameters/get?name="><br>http://localhost:2773/systemsmanager/parameters/get?name=</a>%2Fmy%2Fparameter</pre><pre># Secrets<br><a href="http://localhost:2773/systemsmanager/parameters/get?name=">http://localhost:2773/</a>/secretsmanager/get?secretId=<em>secretId</em>&amp;versionId=<em>versionId</em></pre><pre># Port is configurable with env var PARAMETERS_SECRETS_EXTENSION_HTTP_PORT</pre><blockquote>Note: The parameter path needs to be URL encoded for slashes etc</blockquote><p>To the request you will also need to add a header of the AWS Session Token which can be got from the Environment</p><pre>&#39;X-Aws-Parameters-Secrets-Token&#39;:os.environ.get(&#39;AWS_SESSION_TOKEN&#39;)</pre><p>Putting it all together (In Python):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae648228cb8e54cbb3c15ac2bd99dcc6/href">https://medium.com/media/ae648228cb8e54cbb3c15ac2bd99dcc6/href</a></iframe><p>Which will produce the results:</p><pre>START RequestId: 7a288f36-8a21-44d8-a66c-3cb877096d0e Version: $LATEST</pre><pre>[AWS Parameters and Secrets Lambda Extension] 2022/10/31 10:43:09 INFO ready to serve traffic</pre><pre>[INFO] 2022-10-31T10:43:09.591Z 7a288f36-8a21-44d8-a66c-3cb877096d0e Received Param: svc-accountname</pre><pre>END RequestId: 7a288f36-8a21-44d8-a66c-3cb877096d0e<br>REPORT RequestId: 7a288f36-8a21-44d8-a66c-3cb877096d0e Duration: 10.38 ms Billed Duration: 11 ms Memory Size: 256 MB Max Memory Used: 65 MB</pre><p>As you can see this would be very simple to wrap into a Python function and if you support several Lambdas then that Python function could be included into a module added to all Lambdas.<br>This would save building in the SDK into any Lambdas.</p><blockquote>Note: If you see an error similar to<strong><em> not authorized to perform: ssm:GetParameter </em></strong>then you need to look at your IAM policy.</blockquote><h3>Conclusion</h3><p>There is more to the Extension including caching of parameters, getting parameter versions etc so please do read through the documentation to get all the details. <br>But if you are only building an AWS SDK in for parameters or secrets then this is an easy win to make your Functions smaller, cheaper and more performant.</p><p><strong>Parameter store docs</strong>: <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html#ps-integration-lambda-extensions-add">https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html#ps-integration-lambda-extensions-add</a></p><p><strong>Secrets Manager docs</strong>: <a href="https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html">https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=806d457bb4de" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-parameters-and-secrets-lambda-extension-806d457bb4de">AWS Parameters and Secrets Lambda Extension</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Software Reliability Incident Management]]></title>
            <link>https://medium.com/@apr_1985/software-reliability-incident-management-cba4bd6f9936?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/cba4bd6f9936</guid>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 15:39:41 GMT</pubDate>
            <atom:updated>2022-10-25T15:46:04.991Z</atom:updated>
            <content:encoded><![CDATA[<h4>Incidents can happen at any time, and they may not even be you or your company at fault but you need to be prepared for handling them quickly and efficiently.</h4><blockquote>Note: I use the word incident rather than outage as there are other issues that can require immediate attention like increased latency or errors in page loads etc.</blockquote><p>Incidents happen. There is no contesting that, a bad config change, an untested edge case, a failed disk etc can all cause a bad experience for an applications users.<br>The difference is how a team or company reacts when incidents do happen.</p><p>I am going to go through some tips or questions to ask yourself to help prepare for incident prevention, response and what to do afterwards.</p><p>Please skim the article and its headings first and then read into anything that interests you 👍</p><h3>Monitoring and Observability is key</h3><p>The key to successful incident prevention or successful mitigation, is having decent monitoring and observability throughout the product and stack…. without going over the top.</p><p>It is worth the upfront costs to get good monitoring, centralised logging and tracing etc built into the products and stacks early as that cost will be paid several times over when an incident that could cost the company’s reputation or money is prevented or mitigated quickly. Paying this cost also highlights where product improvements can be made to proactively stop incidents from occurring.</p><h4>Reframe what is an incident</h4><p>It is all too easy to implement the quick win alerts like “high CPU” or “low disk space”. These <strong>metrics</strong> are important and necessary, but the <strong>alerts</strong> are more often than not are setup badly with thresholds just pulled out of the air and so alerts fire all over the place and flap about which leads to alerts being ignored and pager fatigue. It also maybe that these alerts don’t actually correlate to a user facing issue and don’t need acting on immediately.</p><p>The more important monitoring for alerting is what the customer experience currently is. What is the latency of the application transitions? Are errors being thrown to the users? Is a page moving all over the place on load leading to users clicking the wrong button (I really hate that).<br>These are the things that cause users to complain and so when these start to take a turn for the worst this is what needs alerting and jumping on.</p><blockquote>Think of it like this…. A customer doesn’t care that a database is down, but they do care that they cannot sign in.</blockquote><p>As mentioned earlier the “high CPU” or “low disk space” metrics are still required as it is these in combination with application logs and traces that will narrow down <strong>why</strong> an application is having an issue.</p><pre>Alert: /login valid responses &lt; 95% for more than 2 minutes<br>Metric: Database current CPU 99%<br>Logs: Timed out connecting to database</pre><p>This helps with quick and effective incident response.</p><p><strong>Note</strong>: You may want “informational” alerts on some of your metrics e.g. if it takes two week to procure a disk drive or get sign off to bump and instance size you probably want to know when it is getting close to full.</p><p>If you haven’t already, make time to read the Site Reliability Engineering book from Google<br><a href="https://sre.google/sre-book/table-of-contents/">https://sre.google/sre-book/table-of-contents/</a></p><h4>Monitoring is a constant process</h4><p>Monitoring is too often done once and then left. Thresholds are plucked out of the air at the time the alert is setup, and those numbers are forever gospel and Ops teams die on them. <br>Make re-evaluating the thresholds a regular activity. When a new feature is proposed how will it affect the thresholds, when a senior member of the team leaves do the thresholds need adjusting to take pressure off the rest of the team? Is a big client about to onboard and stability is required? Then tighten the thresholds.</p><p>Remember though tighter thresholds will result in a more expensive process either in peoples time responding to alerts or in reduced feature delivery to keep changes less risky.<br>Too loose thresholds and the users may not get the experience they want.</p><h4>Don’t be better than you need to be</h4><p>Ask a product manager how quick the page needs to respond and they will say AS QUICK AS POSSIBLE. So, you work hard optimising everything and make the page super responsive. Users get used to this level of snappyness. Then you introduce a new feature and the responsiveness of the page gets worse. Because the page was so optimised in the first place there is nothing you can do now and users are not happy as the page is now slower.</p><p>In reality the customer would have been more than happy with a page 3 or 4 times slower in the first place and that would have been their expectations. Then when the new feature was introduced wiggle room could have been found or optimisations could have been made.</p><p>Do not over engineer yourself into a corner.</p><h4>Know your risky times</h4><p>There are times when incidents are more likely to occur. During or after a deploy, peak load times, or when both the senior engineers are on holiday (and so neither of them is slyly rebooting that server every night).<br>Knowing when these periods are and what is happening can help shape your response.<br>Deployment Markers, Progressive rollouts, good quality tested runbooks can all help to mitigate the risks posed and mean that you aren’t forced into doing out of hours deploys</p><h3>During the Incident</h3><h4>Don’t fix the root problem… yet</h4><p>This sounds controversial at first but the goal of incident response is to get the service back to normal so users are happy again. If you spend another hour searching through logs, and doing diagnostics and testing this and that, when a simple server reboot will get the service back, then that is another hour of lost revenue or reputation.<br>Get the service restored then start diagnosis.</p><p>Do the diagnosis as a team in working hours. Hopefully your incident happened in working hours and the whole team is on it any way. But if it does happen out of hours then you don’t want the on-call engineer or the L1/L2 team doing diagnosis as they are either really sleep or not experts in the service and so may miss something critical.<br>This also helps to keep run books snappy and allows a wider range of people to be able to use them if less domain knowledge is required.</p><h4>Ensure runbooks are stored in multiple locations and are up to date</h4><p>Just a quick note to ensure runbooks are always available. It is no good keeping your runbook on how to restore the Wiki on the Wiki (yes I have seen this).</p><h4>Use Generic Mitigations and Automation</h4><p>Generic Mitigations are easy processes that can fix a bunch of issues and are especially useful when an L1/L2 team are looking after on-call. These mitigations can be if you see A or B or C then reboot the server. If you see X or Y or Z then clear all the caches, if you see nodes overwhelmed for any reason just add more nodes.<br>Making the during incident process as simple as possible will keep your users happier (and on-call engineers better rested), even if that comes at the expense of an easier diagnosis afterwards.</p><p>Generic Mitigations can often be packaged at Playbooks or Jenkins jobs (or whatever), meaning that they can be run by L1/L2 teams without having to give them direct access to production services which raises the security bar too.</p><p>To go further (which you should) Generic Mitigations can also be automated as a first point of call, and should be where possible.<br>If an alert goes off and some automation can run to replace/reboot the node and then update a channel on what it did and why then this can save a person needing to be involved and often react quicker than a person can which might lead to not having to even put out a public communication.<br>A full post incident process should still be carried out and it should not be relied on as a normal procedure, but is worth the investment.</p><p>Read more about generic mitigation in this great article <a href="https://www.oreilly.com/content/generic-mitigations/">https://www.oreilly.com/content/generic-mitigations/</a></p><h4>Using “Hacks” to keep your services available</h4><p>While not technically during the incident, it is worth mentioning that “hacks” should not be looked down on <strong>if they are short term measures</strong>.<br>For example a new feature is rolled out and it is noticed that with production loads the memory usage creeps up to exhaustion over a 6-hour period. The feature cannot be rolled back as too many customers are now using it. If the service can be safely automatically rebooted every couple of hours to keep the memory at bay, then this is a good thing and allows the team time to investigate and develop a proper fix rather than rushing something out that may make the situation worse.</p><h4>Communication</h4><p>The hardest part of any incident as get this wrong and you either have angry users on Twitter raging that your service is down and you aren’t doing anything, or you communicate early and then end up fixing the issue 30 seconds later and users would never have noticed.</p><p>Make this part of your incident response process to have conditions on when and who needs to be notified and how often updates should be published. It is also helpful to have signed off statements that are used for public communication if this is required (especially if the incident is out of hours and PR people are in bed).<br>This communication should be done by the incident leader and they are responsible for keeping the incident log of events up to date which should form the basis of a post-mortem afterwards.</p><p>As a company it is worth investing in tooling like StatusPage (no affiliation). For single product companies you can have each component of your product listed and have granular updates like GitHub does <a href="https://www.githubstatus.com/">https://www.githubstatus.com /</a><br>For larger companies you could have each of your products listed and updated during an incident.<br>Even if this page is internal facing only is it super helpful to know the state of other teams in the company especially if you depend on them<br>e.g. if my product relies on the authentication team and they are having an issue it is good for me to be notified of this so I can prepare mitigations.<br>As it can be updated manually or through APIs it is very flexible and good for getting metrics from too.</p><h3>After the Incident</h3><h4>Diagnosis</h4><p>As mentioned previously it is best to do the diagnosis work after the incident so that no one is panicking and time can be taken to ensure that the correct conclusions are drawn especially if several teams are involved.</p><p>This is where you will see the investment in central logging, tracing and monitoring pay off as you will be able to go back to the point when the incident happened and correlate what was going on round the stack without having to logon to various servers and grep logs and old skool antics like that</p><h4>Blameless Post-Mortem</h4><p>Key word here <strong>Blameless</strong> no one set out to make the application break (and if they did then that is a whole other article) and so bun throwing doesn’t help anyone and must <strong>not</strong> be tolerated. <br>Everyone needs to work as a team to discover why the incident happened. How was it not caught in review, or testing, or staging? Is it a process issue? A technical issue? a documentation issue? Often incidents happen due to many little issues rather than one big one and so getting to the bottom of things takes time.</p><p>As part of the post-mortem (P-M), a log of what happened during the incident should be reviewed. During the incident all steps and changes made must be recorded so that they can be unwound in the P-M. This allows any code to be updated with changes made (e.g. IaC) and also to ensure that runbooks are updated if they are wrong or don’t exist and also to check if there are any improvements that can be made to the response process.</p><p>P-Ms can be long drawn-out affairs if lots of teams are involved, or relatively quick if automation caught the problem and resolved it but the root cause should always been found and a remediation put in place.</p><p>All actions that result from the P-M should be recorded and it is up to the Incident Manager (whoever plays this role in the company), to follow up and ensure that all actions are followed through.<br>Actions could include application changes, runbook updates, process changes etc</p><p>Post-Mortems should be recorded centrally and preferably shared with the wider company and possibly publicly (you could even write a blog post about it :D ), it certainly should go to any teams that rely on your service internally to ensure they understand what happened and know you are doing something about it.</p><p>There are lots of great P-M templates out there e.g. this one from <a href="https://www.atlassian.com/incident-management/postmortem/templates">Atlassian</a> so have a look at a few and adopt or adapt one that works for your requirements.</p><h4>Don’t be afraid to try new things</h4><p>After an incident it can be all too tempting to tighten your belts and make minimal changes to anything. But really it is the right opportunity to explore new things.<br>Have you had a couple of successive incidents that could have been caught during testing? Don’t sack your QA team but sit with them and see what could be done better and improved. Maybe they have been trying to get time to use a new framework and not been allowed yet or maybe if they got a preview environment per PR they could do new things.<br>Did an incident happen because something was missed in code review? Maybe now is a good time to look into review practices to see if anything can be done to help, maybe better static analysis, allowing more time per review, gamification, <a href="https://github.com/chralp/heyburrito">HeyBurrito</a> slack app… whatever works.</p><h3>A little note on reward and shaping your team</h3><h4>Do not reward Heroes</h4><p>All too often it is easy to let a “Hero” develop in the team. The person that fixes all the issues without following the processes and then “forgets” to update tickets or runbooks, BUT THEY GET STUFF DONE. <br>Or the “Hero” that is quietly rebooting the server every evening rather than fixing it properly.<br>Or the “Hero” that neglects package updates until there is a security outbreak and then swoops in working all weekend to patch stuff.</p><p>These are not “Heroes” they are a detriment to your team, yet they are all too often rewarded. I have seen where someone neglected package updates for a year and then got a reward for fixing the most security issues in a product. I’ve also seen where someone refused to test their code, pushed a broken change and was then rewarded for “working all weekend to fix the production issue”.</p><p>A real Hero ensures they spread their knowledge throughout the team and works as part of it. They ensure they are following processes and calling out issues. They ensure their tickets are up-to-date and review PRs in a timely manner.<br>It is the team with no security issues or large production outages that should be rewarded but are all too often overlooked.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cba4bd6f9936" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS Parameter Store in Jenkins Pipeline]]></title>
            <link>https://medium.com/@apr_1985/using-aws-parameter-store-in-jenkins-pipeline-89f0b3bc2736?source=rss-3f9b824ebe37------2</link>
            <guid isPermaLink="false">https://medium.com/p/89f0b3bc2736</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[jenkins-pipeline]]></category>
            <category><![CDATA[aws-parameter-store]]></category>
            <dc:creator><![CDATA[Adam Roberts]]></dc:creator>
            <pubDate>Thu, 13 Oct 2022 10:14:24 GMT</pubDate>
            <atom:updated>2022-10-13T10:41:12.742Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometimes you need to be able to use Dynamic parameters or Secrets in you Jenkins pipelines which isn’t trivial with Jenkins Credential Store so AWS Parameter can help.</p><h4>What is Parameter Store</h4><p>AWS Parameter Store is part of the AWS Systems Manager group of tools. <br>It allows you to store parameters as either plain text or a secret string within your AWS Account.<br>These parameters can be namespaced by a path.<br>e.g.</p><pre>/dev/jenkins/name<br>/dev/jenkins/token</pre><p>Secrets can be added and updated in Parameter Store through various APIs and other tooling like Terraform in a dynamic fashion which makes it a very flexible offering and far cheaper than Secrets Manager.</p><h3>Using Parameters from Parameter Store</h3><h4>IAM Policy</h4><p>First thing that you need is for your Jenkins to have an IAM policy and role that allows it to access the parameters.</p><p>This is a minimum example but adapt it to work with your security requirements.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;ssm:*&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:ssm:&lt;region&gt;:&lt;account&gt;:parameter/*&quot;<br>        }<br>    ]<br>}</pre><h4>Plugins</h4><p>Next step is to install the <a href="https://plugins.jenkins.io/aws-parameter-store/">aws-parameter-store</a> plugin which provides the required step to be able to grab the parameters in a pipeline.</p><p>If you do not already have it installed it would be advisable to install the <a href="https://plugins.jenkins.io/mask-passwords/">mask-passwords</a> plugin too to stop any secrets being displayed in the Jenkins output</p><h4>Code</h4><p>Now that the Controller has IAM permissions to get secrets and the plugin to add the step is available we can grab parameters from the Parameter Store</p><pre>withAWSParameterStore(naming: &#39;basename&#39;, path: &#39;/dev/jenkins/&#39;, regionName: &#39;eu-west-1&#39;) {<br>  sh(&quot;make USER=${NAME} TOKEN=${TOKEN} release&quot;)<br>}</pre><p>The <strong>withAWSParameterStore</strong> step wraps any nested commands allowing them to use the parameters according to the arguments passed in.<br>For this example it makes any parameter under the path <strong>/dev/jenkins/</strong> available and the <strong><em>basename</em></strong> says that just the last element on the path forms the variable name.<br>e.g. <strong>/dev/jenkins/name</strong> is accessed through <strong>${NAME}</strong></p><p>As well as <strong>basename</strong> you can also specify <strong>relative or absolute </strong>the descriptions for which can be found in the plugin <a href="https://plugins.jenkins.io/aws-parameter-store/">documentation</a></p><h4>Masking the Output</h4><p>As some of these values are sensitive they should not be sent to the Jenkins log in clear text. Using the <a href="https://plugins.jenkins.io/mask-passwords/">mask-passwords</a> plugin obfuscates the text.</p><pre>withAWSParameterStore(naming: &#39;basename&#39;, path: &#39;/dev/jenkins/&#39;, regionName: &#39;eu-west-1&#39;) {<br>  wrap([$class: &#39;MaskPasswordsBuildWrapper&#39;, varPasswordPairs: [[password: TOKEN, var: &#39;SECRET&#39;]]]) {<br>    sh(&quot;make USER=${NAME} TOKEN=${TOKEN} release&quot;)<br>  }<br>}</pre><p>Wrapping the nested commands with the <strong>MaskPasswordsBuildWrapper</strong> will stop any of those commands from printing the given list of variables to the console</p><h4>Jenkins not in AWS?</h4><p>If Jenkins is not in AWS you can still use this technique. But rather than assigning a Role to Jenkins you will need to create an IAM user with the correct policy attached and then store the credentials in the Jenkins Credential Store using the <a href="https://plugins.jenkins.io/aws-credentials">AWS Credentials Plugin</a>.</p><p>Then you can pass in the Credentials ID to the <strong>withAWSParameterStore </strong>call.</p><pre>withAWSParameterStore(credentialsId: &#39;AWS_USER&#39;, naming: &#39;basename&#39;, path: &#39;/dev/jenkins/&#39;, regionName: &#39;eu-west-1&#39;)</pre><h3>Bonus</h3><p>However the main limitation is that most secrets get into Jenkins by being manually added in by administrators. This means that it is a manual job to rotate and update credentials when they need changing.</p><p>You can pull Parameters out in you JCasC files too.<br>Install the <a href="https://plugins.jenkins.io/configuration-as-code-secret-ssm/">configuration-as-code-secret-ssm</a> Secret Source extension plugin.</p><p>As above ensure your Jenkins has the correct IAM policy attached to get parameters.</p><p>Set the environment variable to the prefix required</p><pre>---<br>credentials:<br>  system:<br>    domainCredentials:<br>      - credentials:<br>        - string:<br>            scope: GLOBAL<br>            id: opsgenie-api-key<br>            secret: &quot;ABVD12342jfj&quot;<br>            description: OpsGenie API Access Key<br>        - usernamePassword:<br>            scope: GLOBAL<br>            id: &quot;artifactory-api-user-token&quot;<br>            username: &quot;admin&quot;<br>            password: &quot;apikey123&quot;<br>            description: &quot;Service account user and token for artifactory&quot;</pre><p>but of course you do not want to be hard coding in our secret values and committing them you want to template them out and use AWS Parameter Store to house them.</p><p>To do this a new plugin needs to be installed called <a href="https://plugins.jenkins.io/configuration-as-code-secret-ssm/">configuration-as-code-secret-ssm</a> this is the Secret Source extension plugin that does the work.<br>Once installed you will need to set an environment variable with the prefix for your parameters. In the case of the examples above it would be <strong>/product1/</strong></p><pre>CASC_SSM_PREFIX=/product1/</pre><p>Once this is set and Jenkins has AWS permissions to read the required parameters (see the plugin help <a href="https://plugins.jenkins.io/configuration-as-code-secret-ssm/">configuration-as-code-secret-ssm</a>), you can template out the variables in your JCasC</p><pre>---<br>credentials:<br>  system:<br>    domainCredentials:<br>      - credentials:<br>        - string:<br>            scope: GLOBAL<br>            id: opsgenie-api-key<br>            secret: &quot;${opsgenie_api_key}&quot;<br>            description: OpsGenie API Access Key<br>        - usernamePassword:<br>            scope: GLOBAL<br>            id: &quot;artifactory-api-user-token&quot;<br>            username: ${service-account-name}<br>            password: ${service-account-artifactory-token}<br>            description: &quot;Service account user and token for artifactory&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*3srPd-VPqhD_UI7YlJ4KIQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89f0b3bc2736" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>