<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Capsaicin on Medium]]></title>
        <description><![CDATA[Stories by Capsaicin on Medium]]></description>
        <link>https://medium.com/@kbambridge?source=rss-4dbc578c312------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DyL7raYUPP1Txhb4g0s2Tw.jpeg</url>
            <title>Stories by Capsaicin on Medium</title>
            <link>https://medium.com/@kbambridge?source=rss-4dbc578c312------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:09:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kbambridge/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Setting up Istio 1.0 in Production]]></title>
            <link>https://medium.com/@kbambridge/setting-up-istio-1-0-in-production-105207e7e742?source=rss-4dbc578c312------2</link>
            <guid isPermaLink="false">https://medium.com/p/105207e7e742</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[istio]]></category>
            <dc:creator><![CDATA[Capsaicin]]></dc:creator>
            <pubDate>Mon, 22 Oct 2018 16:01:02 GMT</pubDate>
            <atom:updated>2018-10-22T16:01:02.268Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello! At Ads on Top we were in need of a monitoring solution for our cluster that could send alerts to a slack channel when our cluster was experiencing failures, so we decided on setting up an Istio service mesh. In this post I’d like to give a short introduction to the benefits of a service mesh, why we decided it was the solution for us, and what our setup looks like.</p><h3>Motivation</h3><ul><li>We wanted a way to have alerting on success rates in our kubernetes cluster</li><li>Small startup with &lt;5 engineers, and couldn’t afford Sumo Logic or data dog</li><li>Ingress code was in large YAML files and hard to follow</li><li>Canary deploys for rolling out new versions of our services</li></ul><h3>Service Mesh</h3><p>There are two main components of a service mesh, I list out their main features here. This is by no means a comprehensive description of all that a service mesh is for, but it’s just a primer for those who are unfamiliar. A couple resources I found helpful were <a href="https://blog.buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/">Bouyant’s blog post</a> and this <a href="https://thenewstack.io/service-mesh-and-the-promise-of-istio/">New Stack article</a>.</p><h4>Control Plane</h4><ul><li>Routing</li><li>Traffic shifting</li><li>Timeouts + other L7 configuration</li><li>Canary Deploys</li></ul><h4>Data Plane</h4><ul><li>Telemetry gathered by mesh proxies</li><li>Provides success rate information</li><li>Introspection into L7 traffic</li></ul><p>I won’t go into the specifics of the Istio architecture here, but they explain it well in their <a href="https://istio.io/docs/concepts/what-is-istio/#architecture">docs</a>. To give you a basic idea, the control plane is determined by Pilot pods, which are responsible for storing policies in the Mixer pods who are the “enforcers” of the policy. Whenever an Envoy sidecar wants to make a request, it talks to the mixer first so it knows where to make the request. The Envoy proxy and Mixer represents the Data Plane, or where the request data is piped through. The pilot is the control plane.</p><h3>Setting it up</h3><p>There multiple ways to install Istio, I used <a href="https://github.com/helm/helm">helm</a> because of the ability to leverage community packages and the built in versioning. Install it in istio-system namespace. Full helm installation instructions <a href="https://istio.io/docs/setup/kubernetes/helm-install/">here</a>. Then running</p><ul><li>kubectl label istio-injection=enabled</li></ul><p>Will label the kubernetes namespace resource:</p><pre>$ kubectl get namespace default -o yaml</pre><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  creationTimestamp: 2016-12-17T22:44:11Z<br>  labels:<br>    istio-injection: enabled<br>  name: default<br>...</pre><p>Istio reads the labels from the namespaces to know where to perform the injection, now the pods will restart with a sidecar called istio-proxy!</p><pre>$ kubectl describe pod &lt;pod-in-default-namespace&gt; | grep istio-proxy -C2</pre><pre>...<br>containers:<br>...<br>  <strong>istio-proxy</strong><em>:</em><br>    Container ID:  docker://7f4df0885d1163734dff3285d7c1d72a53153fc38f4e923098c04fe3159c9def<br>    Image:         docker.io/istio/proxyv2:1.0.1</pre><p>So to recap we now have all of our pods in the default namespace wrapped in a proxy that can enforce traffic rules, configure network parameters like timeouts, and make telemetry information available for collection. Now, however you will quickly find out your cluster cannot access anything outside of itself! This is because Istio is locked down by default. Since our cluster is able to talk to a multitude of external services I decided to only include the IP ranges for the nodes in the kubernetes cluster which was 100.64.0.0/10 and can be configured in the helm chart values <a href="https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/values.yaml#L155">here</a>. For more details on controlling egress traffic see the <a href="https://istio.io/docs/tasks/traffic-management/egress/">istio docs</a>.</p><h4>Gateways</h4><p>Now we have traffic between pods and traffic going outside our cluster working. What about incoming traffic? For that Istio uses a resource called a Gateway. Gateways basically represent the potential entry points from outside the cluster. They define how the istio-ingressgatewaydeployment in the istio-system namespace routes traffic into the cluster.</p><p>The trick here is for each port, you need a Gateway entry for all the hosts that can access that port. For us that meant setting up a server entry for ports 80, 443, and 8080.</p><p>Here is what our gateway file looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d9d164f4ce915c45f9561aefc949b68/href">https://medium.com/media/0d9d164f4ce915c45f9561aefc949b68/href</a></iframe><p>Note the TLS entry at the bottom of the YAML file, we’ll get to that later.</p><h4>Ingress Routing</h4><p>Migrating from nginx-ingress to Istio’s concept of a VirtualService allowed us to properly modularize our ingress. Istio’s <a href="https://istio.io/docs/reference/config/istio.networking.v1alpha3/#HTTPMatchRequest">HTTPMatchRequest</a> rules make it easy to define specific enough rules so you don’t have collisions between files. In the VirtualService definition one can specify a gateway to route traffic from, but it is not required. VirtualServices can also be used for a particular service to do things like canary deployments with header-based routing.</p><p>Here’s an example of our backend <a href="https://istio.io/docs/reference/config/istio.networking.v1alpha3/#VirtualService">VirtualService</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fae4499ed127a142ed9f9203c29c5a75/href">https://medium.com/media/fae4499ed127a142ed9f9203c29c5a75/href</a></iframe><p>What this does is take all hosts coming in through the web gateway and applies these routing rules. For the above example, it takes any requests going to /login* and routes them to the service web-proxy-v2, and applies the corresponding CORS policy to the destination. Other configuration can be added here like http timeouts. For full list of configurations check out the <a href="https://istio.io/docs/reference/config/istio.networking.v1alpha3/#VirtualService">API Reference</a>.</p><h4>Certificates</h4><p>Remember that TLS entry in the gateway above? Those certs are setup by a library called cert-manager. A short explanation of <a href="https://github.com/jetstack/cert-manager">cert-manager</a> is it’s a job that automatically renews certificates using LetsEncrypt and installs them in a configurable location in a k8s cluster. For more information on how to set up TLS with Istio see <a href="https://istio.io/docs/tasks/traffic-management/secure-ingress/#configure-a-tls-ingress-gateway">the docs</a>. Here is the <a href="https://medium.com/@prune998/istio-0-8-0-envoy-cert-manager-lets-encrypt-for-tls-d26bee634541">guide</a> I used to set up cert-manager with Route53. As always here is the <a href="https://github.com/helm/charts/tree/master/stable/cert-manager">helm chart</a>.</p><h3>Monitoring</h3><p>Istio comes with a few addons that made our monitoring setup really simple. It comes with:</p><ul><li>Prometheus deployment with config that automatically collects numerous metrics like status codes, latency, request size, etc. from the mesh</li><li>Grafana dashboard with templates that visualize those prometheus metrics</li><li>Servicegraph for visualizing which services are talking to one another</li><li>Jaeger for distributed tracing (this requires some additional setup to fully work)</li><li>and more!</li></ul><p>Enabling the desired addons is as simple as setting some configuration in the helm chart!</p><p>Here are a couple screenshots of Istio provisioned dashboards:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e-zFl-r3nE6DVhaP" /></figure><p><em>Mesh dashboard: Global view of the cluster’s success rates and latency</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*whxdOuzRYzmHktbg" /></figure><p><em>Service workload dashboard: information about traffic coming in and out of a particular service</em></p><p>Metrics from Istio are great to visualize the network activity in the cluster, but not to monitor the cluster itself. For that something like that <a href="https://github.com/kubernetes/kube-state-metrics">kube-state-metrics</a> is useful. Again, there is a helm chart for that! Check out: <a href="https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus">kube-prometheus</a>.</p><p>I launched this in a separate monitoring namespace, but it came with another Grafana instance. To solve the problem of needing to access two Grafana instances, I instead downloaded the dashboards from the kube-prometheus grafana instance, disabled it, and then re-uploaded the dashboards to a separate folder in the Istio grafana instance. This required adding the prometheus instance in the monitoring namespace to the data sources, and then selecting that datasource for the imported dashboards. Another option was using <a href="https://prometheus.io/docs/prometheus/latest/federation/">Prometheus Federation</a> to mix the data together, but this was unnecessary since they were separate dashboards so they could use separate data sources.</p><h3>Gotchas</h3><p>As with anything new, there will be issues. Here are some the issues I ran into while installing and configuring Istio:</p><ul><li>Configuring timeouts. Our requests still cap out at 10s (this still causes our success rates to drop in some edge cases, I haven’t been able to successfully increase the timeout 😞)</li><li>Running a service mesh is not free! After installing Istio our cpu usage went up mostly because of all the telemetry info that was being collected. Each request also has an extra hop to the mixer so that will increase overall latency a bit. <a href="https://github.com/istio/istio/issues/7879#issuecomment-431408635">Here</a> is a discussion on the current performance expectations.</li><li>There can be some issues with Custom Resource Definitions and helm while trying to install and upgrade. If it fails the first time, I recommend deleting the Istio crds or trying the --no-crd-hook flag for helm install .</li></ul><h3>Things I’m looking forward to</h3><ul><li>Spinnaker integration with Istio. There has been a little <a href="https://community.spinnaker.io/t/kubernetes-service-mesh-adoption-istio/66">discussion</a>. I’m no Spinnaker expert, but it would probably involve <a href="https://www.spinnaker.io/guides/developer/new-stage/">writing a new stage</a>.</li><li>All the kinks surrounding mTLS and perf issues being fixed. I see many commits coming in regarding auth.</li><li>A new proposal for <a href="https://docs.google.com/document/d/1ut33e7kxq9SrlxYdl-HmQgCwdDTfu2DBQCjZhUV0Lo8/edit#heading=h.qex63c29z2to">Multi-Cluster Service Mesh</a></li></ul><p>I think the Istio team has done a great job creating an abstraction that makes it easy to deploy a feature-rich service mesh which enables the use of best practices learned from companies operating at massive scale. I’m excited to continue following it and see where this project goes!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=105207e7e742" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>