<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis Vernon on Medium]]></title>
        <description><![CDATA[Stories by Louis Vernon on Medium]]></description>
        <link>https://medium.com/@louisvernon?source=rss-51f348a29c92------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Rf8dSIhqV48ISxNw</url>
            <title>Stories by Louis Vernon on Medium</title>
            <link>https://medium.com/@louisvernon?source=rss-51f348a29c92------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:17:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@louisvernon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mapping Kubernetes Service Accounts to GCP IAMs using Workload Identity]]></title>
            <link>https://medium.com/google-cloud/mapping-kubernetes-service-accounts-to-gcp-iams-using-workload-identity-b53496d543e0?source=rss-51f348a29c92------2</link>
            <guid isPermaLink="false">https://medium.com/p/b53496d543e0</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[secrets]]></category>
            <dc:creator><![CDATA[Louis Vernon]]></dc:creator>
            <pubDate>Tue, 18 Jun 2019 16:17:54 GMT</pubDate>
            <atom:updated>2020-06-09T21:21:39.394Z</atom:updated>
            <content:encoded><![CDATA[<h4>GCP Credential Management on GKE just got a whole lot easier</h4><p>We love Google Kubernetes Engine (GKE) but until recently we did not have a great story around how to give our services running on Kubernetes the desired Google Cloud Platform (GCP) permissions.</p><p>We explored a bunch of tooling around GCP service account credential management - leveraging Terraform, Vault, Kubernetes Secrets and KMS encrypted credentials along the way - but nothing gave us the clean, scalable, short-lived token solution we were looking for.</p><p>What we really wanted (and asked for many times) was a way for Kubernetes service accounts to act as GCP service accounts — something Google has now finally made possible via <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity"><strong>Workload Identities.</strong></a></p><p>Let’s walk through an example of how this works.</p><h4>Setup</h4><ul><li>We have a GKE cluster running under a service account: <br><strong>gke-cluster-sa@louis-testing-project.iam.gserviceaccount.com</strong></li><li>We have a bucket <strong>gs://workload-identity-test</strong></li><li>We have a different service account which can read the bucket:<br><strong>bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com</strong></li></ul><p><strong>Objective:</strong> A specific pod running on our GKE cluster can list objects in our bucket without ever touching a GCP service account key or mounting any secrets.</p><h4>How Things Worked Before</h4><p>Let’s try to list the bucket contents without enabling Workload Identity.</p><pre>kubectl run -i -tty test1 -image gcr.io/cloud-builders/gsutil ls gs://workload-identity-test</pre><pre>&gt; AccessDeniedException: 403 gke-cluster-sa@louis-testing-project.iam.gserviceaccount.com does not have storage.objects.list access to workload-identity-test.</pre><p>So what happened here? The gsutil CLI running inside the pod hit the node’s metadata endpoint which returned a token for the service account running the node -<strong> gke-cluster-sa</strong>. This service account doesn’t have permission to read the bucket so we get a 403.</p><h4>Setting Up Workload Identity</h4><p>The first thing we need to do is enable Workload Identity on the GKE cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*7rIyx6N1hDEX48U9" /></figure><p>The Identity Namespace, which is statically defined in the Cluster Edit UI, maps the Kubernetes service account name to a virtual GCP service account handle used for Identity &amp; Access Management (IAM) binding (more on this below).</p><h4>Enable the Metadata Server</h4><p>The Metadata Server runs on each GKE node and changes the behavior of the metadata endpoint. This also secures the metadata endpoint removing the need for <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata">metadata concealment</a>. Let’s enable Metadata Server on our default node pool.</p><pre>gcloud beta container node-pools update default-pool \<br> — cluster=workload-identity-test \<br> — workload-metadata-from-node=GKE_METADATA_SERVER \<br> — zone=us-central1-b</pre><p>With the Metadata Server running on our nodes let’s see what happens of we try to list our bucket contents now.</p><pre>kubectl run -i — tty test2 — image gcr.io/cloud-builders/gsutil ls gs://workload-identity-test</pre><pre>&gt; ServiceException: 401 Invalid Credentials</pre><p>We’re getting somewhere. With the metadata endpoint hardened we can no longer retrieve tokens for the service account running the node and so we now get a 401 error instead of 403.</p><h4>Create and Bind a Kubernetes Service Account</h4><p>Let’s create a specially annotated Kubernetes service account — <em>witest</em></p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  annotations:<br>    iam.gke.io/gcp-service-account: bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com<br>  name: witest<br>  namespace: default</pre><p>Here we’ve defined a service account <em>witest</em>, in the <em>default</em> namespace and we’ve provided an annotation that defines the GCP service account we want to map to.</p><p>Next we need to bind our derived virtual GCP service account handle to the real GCP service account we want to act as.</p><pre>gcloud iam service-accounts add-iam-policy-binding \<br>  --role roles/iam.workloadIdentityUser \<br>  --member &quot;serviceAccount:louis-testing-     project.svc.id.goog[default/witest]&quot; \<br>  bucket-lister-sa@louis-testing-project.iam.gserviceaccount.com</pre><p><strong>serviceAccount:louis-testing-project.svc.id.goog[default/witest]</strong> is derived from the Identity Namespace defined when enabling Workload Identity along with the namespace and name of the Kubernetes service account we are binding.</p><h4>Confirm it Works</h4><p>Let’s try and list our bucket’s contents after configuring everything.</p><pre>kubectl run --serviceaccount=witest -i --tty test3 --image gcr.io/cloud-builders/gsutil ls gs://workload-identity-test</pre><pre>&gt; gs://workload-identity-test/success/</pre><p>Superb, this works beautifully. We are able to list the bucket contents with nary a long-lived token in sight. This is a huge feature which makes our credential management and security story much nicer — thanks Google!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b53496d543e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/mapping-kubernetes-service-accounts-to-gcp-iams-using-workload-identity-b53496d543e0">Mapping Kubernetes Service Accounts to GCP IAMs using Workload Identity</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Kubernetes Scaling via Envoy Metrics]]></title>
            <link>https://medium.com/descarteslabs-team/custom-kubernetes-scaling-via-envoy-metrics-110d0bac720?source=rss-51f348a29c92------2</link>
            <guid isPermaLink="false">https://medium.com/p/110d0bac720</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[envoy-proxy]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[hpa]]></category>
            <dc:creator><![CDATA[Louis Vernon]]></dc:creator>
            <pubDate>Mon, 20 May 2019 16:01:07 GMT</pubDate>
            <atom:updated>2019-09-13T05:46:29.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*NHyURPTHjgQF_s-d0MPisQ.png" /><figcaption>A selection of data products returned from our Raster service</figcaption></figure><p>The <strong>Descartes Labs Platform</strong> runs on Kubernetes and scales from hundreds to tens of thousands of cores in response to customer traffic.</p><p>Much of this load comes from our <strong>Tasks</strong> service, which allows users to scale out analytics models with high throughput access to petabytes of geospatial data. The bulk of the heavy lifting behind the retrieval, transformation, and delivery of this data is handled by our <strong>Raster</strong> service.</p><h4>Scaling Challenges</h4><p><strong>Raster</strong> can be called directly from our Python client or via RESTful API and response times can vary significantly depending on the nature of the request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*azIzWrKIl6zIZo9TKm8zdg.png" /><figcaption><em>Raster</em> request latencies at a random moment in time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*hIX7J6U8ei3Wd-j5" /><figcaption>CPU utilization was not a great indicator of load</figcaption></figure><p>Our original approach to scaling <strong>Raster</strong> used a standard horizontal pod autoscaler (HPA) that tracked CPU utilization per pod. Unfortunately, variation in compute characteristics (requests could be I/O- or CPU-bound) made CPU utilization a poor indicator, and we needed a low threshold to stay ahead of the load.</p><p>Variation in both the nature and duration of requests meant that scaling based on request rate was also not ideal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*WDQclQMw9KpCSkYccPDLHw.png" /><figcaption>At Descartes Labs we are using Managed Istio (1.0.6) on GKE</figcaption></figure><p>We have been using <strong>Istio</strong> for a long time and took note of <a href="https://github.com/stefanprodan/istio-hpa">Istio metrics-based autoscaling</a>, but these higher level metrics (i.e., labeled request counts, duration, rate) were not an obvious fit for our service.</p><h4>Tapping into Envoy Metrics</h4><p>Fortunately for us, <strong>Envoy</strong>, the sidecar proxy used by <strong>Istio</strong>, allows us to directly measure current saturation of our <strong>Raster </strong>service. From the <a href="https://www.envoyproxy.io/docs/envoy/latest/operations/admin"><em>Envoy</em> docs</a>:</p><blockquote><strong><em>upstream_rq_active</em></strong> — Gauge — Total active requests</blockquote><p>By summing the <em>upstream_rq_active </em>across all <em>Raster</em> pods we get an effective measure of how many requests are currently being handled by our service.</p><p>To allow Kubernetes to scale on this metric we installed the <a href="https://github.com/zalando-incubator/kube-metrics-adapter">Zalando kube-metric-adapter</a> as packaged by <a href="https://github.com/stefanprodan/istio-hpa"><em>Stefan Prodan</em></a>. We could technically configure the metric adapter to scrape this metric directly from <strong>Envoy </strong>using the JSON stats endpoint, but it made more sense to let our existing <strong>Prometheus</strong><em> </em>infrastructure handle scraping and aggregation.</p><p><strong>Update:</strong> After upgrading to Istio 1.1.x we found that the specific Envoy metric we leverage is no longer exposed by default. To configure Envoy to provide the metric you must now add the following to the pod template of the deployment you are scaling:</p><pre>template:<br>    metadata:<br>      annotations:<br>        sidecar.istio.io/statsInclusionPrefixes: cluster.inbound,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc</pre><h4>Prometheus Configuration</h4><p>In <strong>Prometheus</strong> this metric is surfaced as: envoy_cluster_upstream_rq_active<br> (<em>this metric is also exposed via </em><em>envoy_cluster_upstream_cx_active</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MMxCD8J-eenoqyx-NWAvBw.png" /><figcaption>Number of active connections to <strong>Raster</strong> as reported by <strong>Envoy&#39;s</strong> <strong><em>upstream_rq_active </em></strong>metric</figcaption></figure><p>Unfortunately the <strong>Istio</strong> bundled <strong>Prometheus</strong> configuration scrapes but drops this number. To retain it you must modify or remove the following lines from the <strong>Prometheus</strong> config.</p><pre>- source_labels: [ cluster_name ]<br>        regex: &#39;(outbound|inbound|prometheus_stats).*&#39;<br>        action: drop</pre><h4>Implementing Our Custom Autoscaler</h4><p>With the number of active requests per pod being tracked by <strong>Prometheus</strong> we could then implement our custom HPA:</p><pre>apiVersion: autoscaling/v2beta1<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: raster-release<br>  namespace: raster<br>  annotations:<br>    metric-config.object.raster-release-rq-active.prometheus/per-replica: &quot;true&quot;<br>    metric-config.object.raster-release-rq-active.prometheus/query:|      sum(max_over_time(envoy_cluster_upstream_rq_active{app=&quot;raster&quot;,cluster_name=&quot;inbound|8000|http|raster-release.raster.svc.cluster.local&quot;, namespace=&quot;raster&quot;,stage=&quot;release&quot;}[1m]))<br>spec:<br>  maxReplicas: 1500<br>  minReplicas: 12<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: raster-release<br>  metrics:<br>    - type: Object<br>      object:<br>        metricName: raster-release-rq-active<br>        target:<br>          apiVersion: v1<br>          kind: Pod<br>          name: raster-release # needed for schema consistency<br>        targetValue: 2</pre><p>The critical part of this HPA config is the annotation block where we:</p><ul><li>Provide a consistent label for the metric: raster-release-rq-active</li><li>Tell the metrics adapter to normalize the metric with respect to the number of pods: per-replica: “true&quot;</li><li>Provide a PromQL query that returns the sum of the max number of active requests per <strong>Raster</strong> pod over the last minute.</li></ul><p><strong>Raster</strong> can handle four concurrent requests per pod, so we set the <em>targetValue</em> to two active requests per pod.</p><p>In our small-scale testing we found that simply taking the sum of envoy_cluster_upstream_rq_active yielded accurate numbers, but when we tested with production traffic (yay for <strong>Istio</strong> traffic mirroring!) and large numbers of pods, we needed to use a window of at least one minute to get consistent numbers.</p><h4>Does It Work?</h4><p>As shown below, our <em>rq_active</em> HPA roughly halved the requested resources. Even accounting for the delay introduced between <strong>Prometheus</strong> scraping <strong>Envoy</strong> and the metrics adapter querying <strong>Prometheus</strong>, we still get more responsive scaling than using CPU utilization, resulting in a lower 503 rate overall. We saw these trends continue once we applied our custom HPA in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OzK1fDNOGo41RzJMQiJZMw.png" /><figcaption>Requested cores for raster-release (using CPU scaling) vs raster-mirror (using rq_activ)</figcaption></figure><h4>Final Thoughts</h4><ul><li><strong>Istio</strong> and <strong>Envoy</strong> made collecting telemetry and safely testing with production traffic (via mirroring) simple.</li><li>We’re now rolling this methodology out to multiple services throughout our stack.</li><li>How often do you get to improve quality of service <strong><em>and </em></strong>reduce costs in the process?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=110d0bac720" width="1" height="1" alt=""><hr><p><a href="https://medium.com/descarteslabs-team/custom-kubernetes-scaling-via-envoy-metrics-110d0bac720">Custom Kubernetes Scaling via Envoy Metrics</a> was originally published in <a href="https://medium.com/descarteslabs-team">descarteslabs-meditations</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Descartes Labs at Spinnaker Summit 2018]]></title>
            <link>https://medium.com/descarteslabs-team/descartes-labs-at-spinnaker-summit-2018-8e3d85a03e0a?source=rss-51f348a29c92------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e3d85a03e0a</guid>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[spinnaker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technical]]></category>
            <dc:creator><![CDATA[Louis Vernon]]></dc:creator>
            <pubDate>Tue, 23 Oct 2018 16:58:52 GMT</pubDate>
            <atom:updated>2018-11-08T18:44:20.077Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rFGA9IoZyNYwaf4P" /><figcaption>Deployment Pipelines for a Descartes Labs <em>Service</em></figcaption></figure><h3>Takeaways from Spinnaker Summit 2018</h3><p>At Descartes Labs we’ve been using Spinnaker to deploy our services since the Kubernetes V2 provider launched earlier this year. As a company still new to Spinnaker, we were excited to attend <a href="https://www.spinnakersummit.com/">Spinnaker Summit</a> in Seattle two weeks ago where we had the chance to see how others are using Spinnaker with Kubernetes and meet some of the awesome community around Spinnaker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zc2RJ7hy_OlR67VQGXBmpw.png" /><figcaption>Descartes Labs deployment triggers and Spinnaker cluster integrations</figcaption></figure><p>I took the opportunity to talk about our experiences <a href="https://cdn2.hubspot.net/hubfs/482737/form-uploads/36c155de-7beb-4fd5-822c-0fad4226ac81/61d1b5a6-fbc2-4483-8d0f-7cf075d5089b-upload_your_presentation-GettingFamiliarWithKubernetesV2.pdf?utm_source=hs_email&amp;utm_medium=email&amp;utm_content=51577051"><em>Getting Familiar with Spinnaker and the Kubernetes V2 Provider (on GKE)</em></a><em>, </em>where<em> </em>I detailed some of our deployment pipelines. After showing how we use a replica Kubernetes cluster for blackbox tested canaries I demoed how we propagate to production based on <a href="https://istio.io/">Istio</a> generated metrics. I then outlined pipelines for high velocity deployments from Google’s Container Registry using the V2 PubSub trigger and linked to a <a href="https://gist.github.com/louisvernon/ca9f8b6aac913fa620fbb8f59f16581e">Halyard customization script</a> which configures those integrations for us automatically.</p><p>Edit: <a href="https://www.youtube.com/watch?v=aNg4LmvgfFg">You can watch my Spinnaker Summit talk here.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MygfZ_ln0kEttkUm" /><figcaption>Istio Workload Dashboard showing our latest deployment canary might have some issues</figcaption></figure><p>Big takeaways for Descartes Labs:</p><ul><li>Automated canary analysis is mainstream — the level of adoption of this feature (only introduced in April 2018) was amazing. Descartes Labs is using manual judgements to approve stages of our pipelines so this is a bandwagon we can’t wait to jump on.</li><li>Automated pipeline generation is tough but tractable — lots of approaches to generating pipelines from code were discussed, both in talks and offline. Managed Pipeline Templates (currently undergoing a re-write) and large third party efforts reinforce what a complex problem this is, but offline chats germinated solid ideas about how we can tackle this more effectively.</li><li>Istio is <em>so</em> hot right now — as one of the only companies at Spinnaker Summit using Istio in production I became an accidental Istio evangelist with lots of interest in our performance numbers, metrics dashboards, and excitement around the dynamic service graph and service monitoring capabilities furnished by <a href="https://cloud.google.com/service-monitoring/">Google’s Stackdriver Service Monitoring</a>. It’s clear Istio has a big part to play in the future of the Kubernetes V2 provider.</li></ul><p>I wish I could have seen more talks, but these were some of my highlights:</p><p><a href="https://docs.google.com/presentation/d/1bhhbgtvO5VVPCP0kytaTDa-hroX9bwY4xnXfQmZrK-0/edit"><strong>Spinnaker, Borg and Annealing</strong></a> — Pim van Pelt from Google led a discussion panel detailing a <em>Kubernetes Bridge </em>that allowed Spinnaker to manage Borg infrastructure through Kubernetes custom resource definitions. This facilitated discussion around Google’s <em>Annealing</em> tool which continuously drives internal Google infrastructure towards a declarative DAG encoded representation of desired state. At the end of the presentation Pim asked us to complete <a href="https://goo.gl/SjXLFV">this survey</a> if we are interested in seeing this open sourced (yes please!).</p><p><a href="https://cdn2.hubspot.net/hubfs/482737/form-uploads/36c155de-7beb-4fd5-822c-0fad4226ac81/27d0f6e3-3370-4fbf-8d97-219801867582-upload_your_presentation-Spin-Summit_-Declarative-Delivery.pdf?utm_source=hs_email&amp;utm_medium=email&amp;utm_content=51577051"><strong>Declarative Spinnaker</strong></a> — The <em>Council of Robs </em>discussed why we want declarative infrastructure management (aka <em>Annealing</em>, see above) and what this would look like in Spinnaker.</p><p><strong>Gluing It All Together</strong> — Oleg Atamanenko from Kublr walked the audience through a Kubernetes V2 deployment strategy leveraging Istio <a href="https://istio.io/docs/reference/config/istio.networking.v1alpha3/#VirtualService">VirtualServices</a> for high granularity canaries/blue-green deployments while using Istio metrics for automated canary analysis.</p><p><strong>Schibsted</strong> — I caught two great talks by Schibsted folks. Over three thousand deployment pipelines, impressive pipeline templating infrastructure, dynamic evaluation of pipelines as artifacts with full provenance and reproducibility of pipeline execution... these folks are doing a great job of highlighting the scalability and extensibility of Spinnaker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*9HP6lRvOXuW3NmAv.png" /></figure><p>It was great to see what a strong and considerate community Spinnaker is fostering. I really hope to make it back next year.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e3d85a03e0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/descarteslabs-team/descartes-labs-at-spinnaker-summit-2018-8e3d85a03e0a">Descartes Labs at Spinnaker Summit 2018</a> was originally published in <a href="https://medium.com/descarteslabs-team">descarteslabs-meditations</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>