<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jcordoba on Medium]]></title>
        <description><![CDATA[Stories by Jcordoba on Medium]]></description>
        <link>https://medium.com/@jcordoba_225?source=rss-3b4d52a18d7a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*cZ1w1_9RmsStD5IT</url>
            <title>Stories by Jcordoba on Medium</title>
            <link>https://medium.com/@jcordoba_225?source=rss-3b4d52a18d7a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:27:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jcordoba_225/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Enhanced mesh troubleshooting: How Kiali can help you diagnose proxy issues]]></title>
            <link>https://medium.com/kialiproject/enhanced-mesh-troubleshooting-how-kiali-can-help-you-diagnose-proxy-issues-1f4e65104c40?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f4e65104c40</guid>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <category><![CDATA[service-mesh]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 20:56:33 GMT</pubDate>
            <atom:updated>2026-11-17T20:56:33.396Z</atom:updated>
            <content:encoded><![CDATA[<p>Keeping your service mesh healthy just got easier. Kiali’s new v2.34 release improves troubleshooting, going beyond traffic visualization to help surface and resolve Envoy memory bottlenecks, before they impact your applications.</p><p>High memory on the istio-proxy can be an issue in production meshes. Until limits are breached, high istio-proxy memory is easy to miss without the right dashboards. Kiali adds a workload-level memory indicator that warns when a proxy starts pushing its memory limits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-IFQHYmvmFQd3aGr_ryww.png" /><figcaption>Envoy indicator in the workload details</figcaption></figure><p>From this indicator, you can directly jump to the dedicated <strong>Envoy Overview</strong> page. Here, Kiali highlights potential memory bottlenecks and connects related performance trends. This gives you the needed context to prioritize next steps: scope config, adjust limits, or scale traffic, while the proxy is still healthy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*ITHxVxrHJFAhO8HbIg_hhA.png" /><figcaption>Envoy tooltip with likely cause of Degraded state</figcaption></figure><h3>Inside the Envoy overview</h3><p>The <strong>Envoy Overview</strong> tab provides a new dashboard with a summary of its configuration, and a banner with a threshold-based status. When the memory is high, it tries to answer a practical question: <em>Why is this proxy consuming so much memory?</em> The classification falls into one of these:</p><ul><li><strong>Large Configurations:</strong> Large dynamic configurations (numerous clusters, listeners, or routes), which can be optimized using Istio’s <a href="https://istio.io/latest/docs/ops/configuration/mesh/configuration-scoping/">Configuration Scoping</a>.</li><li><strong>Heavy Active Traffic:</strong> The proxy is simply handling a high volume of live requests.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLT7RIS1Y72Xm0TwT3TRug.png" /><figcaption>Envoy overview tab</figcaption></figure><p>Memory Status is classified as high when peak istio-proxy memory, in the selected time range, exceeds the warning threshold for the most pressured pod. Kiali prefers the container working set metrics; if that is missing, it uses the Envoy allocated memory metric. Limits are per pod (sidecar annotations and/or container_spec_memory_limit_bytes).</p><p>When a proxy memory limit is set (for example on waypoints), the warning threshold is 70% of that limit. If no limit is set then Kiali uses fixed defaults by proxy type (sidecar vs gateway/waypoint).</p><p>The warning only means memory is above threshold. Configuration vs traffic labels are separate hints from cluster count, traffic or connections; they do not change the memory calculation.</p><h4>Large Envoy configuration</h4><p>If memory is above threshold and active clusters exceed the large-config limit (100 for sidecars, 150 for gateways and waypoints), it is a sign of a large Envoy config footprint, which often correlates with higher proxy memory. Kiali labels this likely configuration when traffic is not busy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*PatK3RNP2zDEnfTlLTYmtQ.png" /><figcaption>Large number of active clusters can cause the high memory due to large configurations</figcaption></figure><h4>Active traffic</h4><p>If memory is above threshold and at least one busy signal is present: about ≥ 100 HTTP req/s, ≥ 10 active connections (when Kiali can measure them), or ≥ 256 KiB/s TCP bytes (typical for waypoints). If the cluster footprint is already large, any busy signal shifts the label toward traffic; with a small footprint, busy traffic is the main reason to blame load instead of configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bM8ykEZFDYcCanxXxDbH0A.png" /><figcaption>High traffic rate can cause the high memory</figcaption></figure><h4>When the cause is unclear</h4><p>Memory is high but the footprint is not large and no busy threshold is met, or memory metrics are missing.</p><p>These labels serve as a guidance, not a definitive root cause. Thresholds are currently not configurable (<em>If you think this is interesting, don’t hesitate to </em><a href="https://github.com/kiali/kiali/issues/10455"><em>upvote this issue</em></a>!).</p><h4>The Memory vs Connections chart</h4><p>The Envoy Overview chart shows istio-proxy working set and Envoy allocated memory. When a proxy memory limit is set, the chart also draws the memory limit and a 70% warning line. The right axis shows active connections over the same time range so you can correlate memory with load.</p><h3>Support for Ambient Mesh and Waypoint Proxies</h3><p>What about Ambient? Ambient workloads do not include an Envoy indicator because they do not use sidecar proxies. However, Kiali provides visibility for <strong>Waypoint proxies</strong>, which can handle heavy loads and require accurate metrics for sizing within the mesh.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u873fXggHyyEuFe9Ve8esg.png" /><figcaption>Envoy overview tab for a Waypoint proxy</figcaption></figure><p>The main difference is that Waypoints rely on L4 metrics for traffic, operating under different thresholds due to higher expected loads.</p><ul><li><strong>Sidecars and Gateways:</strong> Prioritize HTTP req/s. If HTTP traffic is zero and TCP bytes are present, byte rates are used.</li><li><strong>Waypoints:</strong> Always classify traffic using TCP byte rates (TrafficIsByteRate = true). In Ambient architecture, the waypoint acts at L4, and Envoy downstream HTTP stats are often absent, so Istio TCP metrics (and HTTP metrics when applicable) fill the gap.</li></ul><p>These memory diagnostic features are available starting in <strong>Kiali v2.34</strong>.</p><h3>Deep-Dive Investigations</h3><p>Beyond this improvement in Envoy proxy status visibility, Kiali v2.34 includes other enhancements focused on helping resolve mesh issues.</p><p>When you need to investigate the root cause of an issue, Kiali can retain up to 7 days of historical health metrics for workloads, applications, services, and namespaces (when health metrics storage is enabled). From the diagnostic overview, you can easily replay graph history or dive straight into related logs, traces, and metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDL-s2TJ3StMt625ZxsVlw.png" /><figcaption>Health status ribbon</figcaption></figure><p>When the AI assistant is enabled, you can also run prompt-based analyses to understand system behavior during specific historical windows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*nZNEfvOU4kT4hIccqo_g3A.png" /><figcaption>One click created prompt for investigation</figcaption></figure><blockquote>With these enhanced observability capabilities, Kiali helps you catch proxy performance drops early and gives you the context needed to resolve service mesh issues quickly.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f4e65104c40" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/enhanced-mesh-troubleshooting-how-kiali-can-help-you-diagnose-proxy-issues-1f4e65104c40">Enhanced mesh troubleshooting: How Kiali can help you diagnose proxy issues</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Closing the Gap: Bringing Sidecar-Level Tracing to Istio Ambient and Kiali]]></title>
            <link>https://medium.com/kialiproject/closing-the-gap-bringing-sidecar-level-tracing-to-istio-ambient-and-kiali-817b1f9bc61d?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/817b1f9bc61d</guid>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[distributed-tracing]]></category>
            <category><![CDATA[ambient-mesh]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Tue, 19 May 2026 14:54:33 GMT</pubDate>
            <atom:updated>2026-05-19T14:54:33.208Z</atom:updated>
            <content:encoded><![CDATA[<p>Istio Ambient has been steadily evolving across recent releases. This evolution hasn’t always been linear; behaviors have shifted, and certain features only recently reached stability. Although Ambient became <strong>GA in version 1.24</strong>, we can now confidently state that <strong>tracing parity</strong> between Ambient and Sidecar modes is a reality, specifically starting with <strong>Istio 1.29 and Kiali 2.25 in regards to tracing.</strong></p><h4>The Role of Waypoint Proxies</h4><p>First, it is important to clarify that L7 telemetry - both (L7) metrics and traces - in Ambient mode is provided by <strong>Waypoint proxies</strong>. Unlike the sidecar model, where the proxy resides within the same pod as the application, Waypoints are shared, decoupled resources. They act as the “gatekeepers” for a namespace or service, processing all Layer 7 logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*4WWaAPAILIGWEKXc84NuXg.png" /><figcaption>The waypoint handling the traffic — View from Kiali’s traffic graph</figcaption></figure><h4>The Shift in Identity: From Service to Waypoint</h4><p>Because Waypoint proxies are external to the workload, identifying the “owner” of a trace has been an architectural challenge. In Istio 1.27, it was the waypoint proxy that created the trace using the waypoint name as the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1vjXx7YMswamJeUZNaxiA.png" /><figcaption>Service traces with Istio 1.27</figcaption></figure><p>In <strong>Istio 1.28</strong>, a change was introduced to attribute the trace directly to the destination service instead of the Waypoint. This behavior remained in <strong>Istio 1.29</strong>.</p><blockquote>Note that this shift was exclusive to the OpenTelemetry (OTel) provider; traditional formats like Zipkin continued to use the waypoint identity.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UlNPJerxQSN_ImrP-Cfqag.png" /><figcaption>Service traces with Istio 1.29</figcaption></figure><p>However, <strong>Istio 1.30</strong> refined this by treating the Waypoint as a <strong>“first-class citizen.”</strong> The Waypoint now preserves the architectural reality of the hop without obfuscating the destination. The operation name of the span correctly reflects the redirected service traffic, while the infrastructure metadata points to the Waypoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j-GxHqshQ3R7fPgav-huow.png" /><figcaption>Service traces with Istio 1.30</figcaption></figure><h4>The Missing Link: Workload Visibility</h4><p>While service-to-service tracing was functional, there was a significant gap: <strong>Workload-level granularity.</strong> In sidecar mode, we rely on the <em>node_id</em> attribute to identify specific pods. In Ambient, this was the missing piece. If a service routed traffic to multiple versions of a workload (e.g., v1, v2, and v3), we could see the service hop, but we lost the ability to pinpoint which specific workload was processing the request — losing important information for debugging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVDxjPddtS3pteLR8pweIw.png" /><figcaption>Incomplete traceability without the workloads information</figcaption></figure><p>This gap was closed in <strong>Istio 1.29</strong>. By introducing explicit attributes such as <em>istio.source_workload</em> and <em>istio.destination_workload</em>, and leveraging the Waypoint’s privileged “middleman” position, we now have a complete view of both the origin and the final destination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*n7AocdrkaaeoWe6uaYgJSg.png" /><figcaption>Span attributes from a trace</figcaption></figure><h4>Kiali: The Unified Observability Plane</h4><p>How does this look in <strong>Kiali</strong>? Thanks to these new attributes, Kiali’s <strong>Trace Overlay</strong> feature is now more powerful than ever with Istio Ambient Mesh. When selecting a trace, Kiali highlights the entire request path, including both services and the specific workloads involved, completing the view for a trace in the versioned app graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHfliATmJffsLAm9_q47Mg.png" /><figcaption>Kiali trace overlay</figcaption></figure><p>One of the Kiali’s missions is to facilitate troubleshooting. In the <strong>Workload Details</strong> view, it correlates data automatically so you don’t have to manually hunt for traces hidden under a Waypoint’s name. Kiali identifies the associated Waypoint for a workload, queries the tracing provider, and filters the results for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w98mgbXCI01hCk3Elj2mtQ.png" /><figcaption>Service traces in Kiali’s Service details page</figcaption></figure><p>While Kiali by default looks for traces using the service name, you can fine-tune this behavior. If your environment requires it, based on the Istio version and Ambient mesh, you can configure how Kiali maps these identities, using the Kiali configuration <em>external_services.tracing.use_waypoint_name=true</em>, which is set to false by default.</p><p>Beyond visualization, this telemetry enrichment opens the door to even more advanced capabilities. With consistent tracing and metrics across the mesh, Kiali’s AI-driven features (like the c<a href="https://kiali.io/docs/ai/kiali-chatbot/">hatbot</a>) can now more accurately assist in root cause analysis. By having all the pieces of the puzzle in place, the assistant can proactively identify bottlenecks or misconfiguration in your Ambient mesh, turning raw observability into automated intelligence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzxKmAK7pPXHY9Dbra9eOg.png" /><figcaption>Kiali chatbot providing a trace overview based on the current context</figcaption></figure><blockquote>As Istio continues to mature, Kiali ensures that moving to Ambient doesn’t mean losing control. It completes the puzzle, allowing you to focus on your services while Kiali handles the intricate mapping of the mesh.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=817b1f9bc61d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/closing-the-gap-bringing-sidecar-level-tracing-to-istio-ambient-and-kiali-817b1f9bc61d">Closing the Gap: Bringing Sidecar-Level Tracing to Istio Ambient and Kiali</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing OpenShift Service Mesh 3 with Kiali and OpenShift monitoring]]></title>
            <link>https://medium.com/kialiproject/installing-openshift-service-mesh-3-with-kiali-and-openshift-monitoring-584cbddb8c24?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/584cbddb8c24</guid>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Tue, 27 May 2025 15:09:00 GMT</pubDate>
            <atom:updated>2025-10-17T14:36:36.839Z</atom:updated>
            <content:encoded><![CDATA[<p>This article covers the installation of OpenShift Service Mesh 3 (based in the Istio upstream Sail operator), Kiali and OpenShift monitoring for metrics.</p><p>The following versions are used:</p><ul><li>Red Hat OpenShift Service Mesh 3 Operator: 3.0.1</li><li>Red Hat Kiali: 2.4.4</li><li>OpenShift 4.18</li></ul><p>Prerequisites: Have installed the mentioned operators:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*Oqx28akB3kYmgzm2N1SS6g.png" /><figcaption>Service Mesh 3 and Kiali operators</figcaption></figure><h3>Table of contents</h3><ul><li><a href="#6144">Istio Installation</a></li><li><a href="#f230">Enabling OpenShift monitoring</a></li><li><a href="#1888">Install a sample application and add it to the Mesh</a></li><li><a href="#714d">Setup the user monitoring resources</a></li><li><a href="#8a72">Install and setup a Kiali CR</a></li></ul><h3>Istio Installation</h3><p>Create the namespace for Istio and the Istio CNI:</p><pre>oc create ns istio-system<br>oc create ns istio-cni</pre><p>Create an Istio CR (The default values are used in this example):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*KkrCCG51DQfMHIKNja5syw.png" /><figcaption>Istio CR</figcaption></figure><p>Create an Istio CNI resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A6kroq0NtqNUFN3rs3HEMA.png" /><figcaption>Creating an Istio CNI</figcaption></figure><p>It will be created in the <em>istio-cni</em> namespace by default:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*a67uO7EkgMotzXsD6OFJ_w.png" /><figcaption>Istio CNI CR</figcaption></figure><h3>Enabling OpenShift monitoring</h3><p>The following oc apply command will create the ConfigMap below, enabling User Workload Monitoring:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: cluster-monitoring-config<br>  namespace: openshift-monitoring<br>data:<br>  config.yaml: |<br>    enableUserWorkload: true</pre><p>Apply the config map:</p><pre>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/cluster-monitoring-cm.yaml</pre><p>Verify that the <em>prometheus-operator</em>, <em>prometheus-user-workload</em>, and <em>thanos-ruler-user-workload</em> pods are running:</p><pre>oc -n openshift-user-workload-monitoring get pod</pre><p>Check the <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/monitoring/configuring-user-workload-monitoring#granting-users-permission-to-configure-monitoring-for-user-defined-projects_preparing-to-configure-the-monitoring-stack-uwm">documentation reference</a> for more information.</p><h3>Install a sample application and add it to the Mesh</h3><p>The next step is to create a sample application and include it into the Mesh. For this, the bookinfo sample application can be created following the <a href="https://github.com/istio-ecosystem/sail-operator/blob/main/docs/addons/addons.md#deploy-gateway-and-bookinfo">Sail Operator steps</a>. Make sure the revision label matches the Istio resource:</p><pre>kubectl label namespace bookinfo istio.io/rev=default</pre><p>To generate metrics, create an OpenShift route and use it to access the bookinfo app from the browser:</p><pre>oc expose svc/productpage -n bookinfo<br>oc get route productpage -n bookinfo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2eV_6IcHIVKqKk-CVqBfw.png" /><figcaption>Bookinfo demo application</figcaption></figure><h3>Setup the user Monitoring Resources</h3><p>The following oc apply command will create the ServiceMonitor below, installing it in the istio-system namespace:</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: istiod-monitor<br>  namespace: istio-system<br>spec:<br>  endpoints:<br>  - interval: 30s<br>    port: http-monitoring<br>    scheme: http<br>  selector:<br>    matchLabels:<br>      istio: pilot</pre><p>Apply the resource:</p><pre>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/istio-service-monitor.yaml</pre><p>With OpenShift monitoring, each mesh namespace being monitored needs a PodMonitor. Each PodMonitor can have the same definition, but needs a unique name. For this example we have two namespaces generating metrics: istio-system and bookinfo. Use the oc applycommands below to add this PodMonitor definition:</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: PodMonitor<br>metadata:<br>  name: istio-proxies-monitor<br>  namespace: istio-system<br>spec:<br>  selector:<br>    matchExpressions:<br>      - key: istio-prometheus-ignore<br>        operator: DoesNotExist<br>  podMetricsEndpoints:<br>    - path: /stats/prometheus<br>      interval: 30s<br>      relabelings:<br>        - action: keep<br>          sourceLabels: [&quot;__meta_kubernetes_pod_container_name&quot;]<br>          regex: &quot;istio-proxy&quot;<br>        - action: keep<br>          sourceLabels: [&quot;__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape&quot;]<br>        - action: replace<br>          regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})<br>          replacement: &#39;[$2]:$1&#39;<br>          sourceLabels: [&quot;__meta_kubernetes_pod_annotation_prometheus_io_port&quot;, &quot;__meta_kubernetes_pod_ip&quot;]<br>          targetLabel: &quot;__address__&quot;<br>        - action: replace<br>          regex: (\d+);((([0-9]+?)(\.|$)){4})<br>          replacement: &#39;$2:$1&#39;<br>          sourceLabels: [&quot;__meta_kubernetes_pod_annotation_prometheus_io_port&quot;, &quot;__meta_kubernetes_pod_ip&quot;]<br>          targetLabel: &quot;__address__&quot;<br>        - action: labeldrop<br>          regex: &quot;__meta_kubernetes_pod_label_(.+)&quot;<br>        - sourceLabels: [&quot;__meta_kubernetes_namespace&quot;]<br>          action: replace<br>          targetLabel: namespace<br>        - sourceLabels: [&quot;__meta_kubernetes_pod_name&quot;]<br>          action: replace<br>          targetLabel: pod_name</pre><p>Each PodMonitor needs a unique name and namespace, as shown below:</p><pre>metadata:<br>  name: bookinfo-proxies-monitor<br>  namespace: bookinfo</pre><p>Apply the changes:</p><pre>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/istio-pod-monitor.yaml<br>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/bookinfo-pod-monitor.yaml</pre><p>It is possible to customize the labels from the pod monitor. For example, adding a Mesh ID:</p><pre>   - action: replace<br>     replacement: &quot;my-mesh&quot;<br>     targetLabel: mesh_id</pre><p>Then, the query done to Prometheus from Kiali can be scoped by modifying the Kiali CR:</p><pre>spec:<br>  external_services:<br>    prometheus:<br>      ...<br>      query_scope:<br>        mesh_id: &quot;my-mesh&quot;</pre><p>The next oc apply will create the Istio telemetry resource:</p><pre>apiVersion: telemetry.istio.io/v1<br>kind: Telemetry<br>metadata:<br>  name: enable-prometheus-metrics<br>spec:<br>  metrics:<br>    - providers:<br>        - name: prometheus</pre><p>Apply the yaml:</p><pre>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/istio-telemetry.yaml</pre><p>Now, it is time to generate traffic and verify the metrics in <em>Observe &gt; Metrics</em> using <em>istio_requests_total</em> to verify the HTTP traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HR8Lk4mUNGI8USiYuGoUrg.png" /></figure><p>Check the <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/monitoring/configuring-user-workload-monitoring#granting-users-permission-to-configure-monitoring-for-user-defined-projects_preparing-to-configure-the-monitoring-stack-uwm">documentation reference</a> for further information.</p><h3>Install and setup a Kiali CR</h3><p>Next we will create the following Kiali CR. It includes the specific Prometheus configuration:</p><pre>apiVersion: kiali.io/v1alpha1<br>kind: Kiali<br>metadata:<br>  name: kiali<br>  namespace: istio-system<br>spec:<br>  external_services:<br>    prometheus:<br>      auth:<br>        ca_file: /kiali-cabundle/service-ca.crt<br>        insecure_skip_verify: false<br>        type: bearer<br>        use_kiali_token: true<br>      thanos_proxy:<br>        enabled: true<br>      url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091</pre><p>The CR can be created via the console, but if you prefer, here is an oc apply command</p><pre>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/kiali-CR.yaml</pre><p>And next, create the custom Role Binding:</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: kiali-monitoring-rbac<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: cluster-monitoring-view<br>subjects:<br>- kind: ServiceAccount<br>  name: kiali-service-account<br>  namespace: istio-system</pre><p>Apply the yaml:</p><pre>oc apply -f https://raw.githubusercontent.com/josunect/yamls/refs/heads/main/openshiftMonitoring/kiali-RB.yaml</pre><p>Once applied and some traffic has been generated, the traffic graph should be visible in Kiali:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SdcLaerlcIXPjz5gAHMwoQ.png" /><figcaption>Kiali metrics with OpenShift monitoring</figcaption></figure><blockquote>Following these simple steps should be easy to have an OpenShift integrated solution with metrics, Service Mesh and Kiali.</blockquote><p>What’s next? While tracing is not a required dependency from Kiali as Prometheus, you might want to <a href="https://medium.com/kialiproject/installing-openshift-service-mesh-3-tp1-with-kiali-and-grafana-tempo-6b76881ceaef">setup Kiali with Grafana Tempo and Service Mesh 3</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=584cbddb8c24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/installing-openshift-service-mesh-3-with-kiali-and-openshift-monitoring-584cbddb8c24">Installing OpenShift Service Mesh 3 with Kiali and OpenShift monitoring</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Troubleshooting Istio Ambient with Kiali V2]]></title>
            <link>https://medium.com/kialiproject/troubleshooting-istio-ambient-with-kiali-v2-ae30baef5653?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae30baef5653</guid>
            <category><![CDATA[istio-ambient]]></category>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 12:58:15 GMT</pubDate>
            <atom:updated>2025-04-23T12:58:15.804Z</atom:updated>
            <content:encoded><![CDATA[<p>The GA release of Istio Ambient has been long awaited, but it can sometimes be challenging to determine whether an application is properly included in the mesh, and what processing capabilities are available to it. This is because Istio Ambient splits the data plane into two main components: the ztunnel, which handles Layer 4 traffic, and the waypoint proxy, which operates at Layer 7.</p><p><a href="https://kiali.io/news/release-notes/">Kiali version 2</a> introduces new features to help troubleshoot Ambient Mesh applications. In this article, we’ll walk through how to diagnose and troubleshoot your application when it’s running within the Ambient Mesh.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*9cy686oxFCvq4Pe_YRY2PA.png" /><figcaption>Kiali and Ambient Mesh</figcaption></figure><h3>When you’re a Developer…</h3><p>First things first: Kiali first detects that Istio is installed in Ambient mode — otherwise, none of the Ambient features will work. Kiali will check if there is a ztunnelDaemonSet in any accessible namespace.</p><p>If so, the Kiali Overview page will identify namespaces labeled with the Istio Ambient label istio.io/dataplane-mode=ambient. Namespaces with this label will have traffic redirected through ztunnel, automatically providing your applications with Layer 4 processing, including mTLS, L4 traffic policies, and L4 observability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*GOiY31OsEeTcId3ciXvENQ.png" /><figcaption>Ambient Mesh namespace</figcaption></figure><p>Navigating to Kiali’s Traffic Graph, and selecting desired namespaces, Kiali will show the traffic flow from your application. Ambient provides Layer 4 telemetry, TCP level, automatically with ztunnel. Without waypoints you should expect to only see the TCP traffic. To see Layer 7 HTTP telemetry, you must introduce waypoints. With Ambient, having both Layer 4 and Layer 7 telemetry makes the graph more complete, but also more complicated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJW5H2UCUYtUUMIS4iiRWw.png" /><figcaption>Ambient Mesh graph</figcaption></figure><p>Kiali lets you isolate the Ambient telemetry via the Traffic dropdown. You can simplify the graph by selecting traffic reported only by ztunnel, or only by waypoints. Note that the non-Ambient options are also available, letting you isolate on protocol.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYm3h06jW7uunhwv6oPMNg.png" /><figcaption>ztunnel Telemetry</figcaption></figure><p>To verify secure connections you can enable the “Security” display option. Edges displaying a lock icon are mTLS secured, which is automatically provided by ztunnel. By clicking on an edge you can view information about the mTLS connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*q8G-aqCI5EipaN1SHcOipg.png" /><figcaption>mTLS connection</figcaption></figure><p>When you include waypoint telemetry you may see double edges for HTTP requests. This is because it is showing both the L4 and L7, ztunnel and waypoint, telemetry. You can isolate your graph to show just the “waypoint” traffic, which provides a traffic graph more similar to what you may be familiar with using sidecars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w81kwNmkalh3NnaPeFn6CQ.png" /><figcaption>Limit the traffic to just the Waypoint telemetry</figcaption></figure><p>Waypoint proxy nodes are hidden by default, because they’re not part of the application — and they make the graph harder to understand. But if you need to know where they are deployed and what application traffic is being routed through a waypoint, you can enable the display option in the graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nmsr_B9P1zjg7Xqm_cniUg.png" /><figcaption>Waypoint proxies in the Kiali graph</figcaption></figure><p>One of the first signs of failure in an application is when a graph edge turns yellow or red, indicating some 4xx or 5xx error in the requests — <a href="https://kiali.io/docs/configuration/health/">the tolerance</a> can be configured in Kiali by overriding the defaults. You can right-click a failing node and select “Details” from the context-menu.</p><p>The workload detail page provides you with a lot of information, and you can verify if the application is correctly enrolled in Ambient. Hover over the Ambient badge to check if it includes the L4 and L7 badges (When it is enrolled in a waypoint proxy, L7 will be added).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*GyLDrYnooBkrHN22-NshQA.png" /><figcaption>Application details</figcaption></figure><p>The waypoint proxy is also linked, in case you need to check any details.</p><p>The workload pods can give us information about the protocol the workload is using — <a href="https://istio.io/latest/docs/ambient/architecture/hbone/">HBONE in the case of Ambient</a>; otherwise it will be TCP. This step helps verify whether the application is enrolled in Ambient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*WZAKNVNXPSynY5_9j5xjkw.png" /><figcaption>Protocol used by the pod</figcaption></figure><p>The next step when troubleshooting an application is to check the application logs. If there are no clear clues in there about what might be happening, it is very useful to check the proxy logs — at least to make sure there are no errors on that side.</p><p>With sidecars, it was easy because the proxy was deployed in the same pod as the application. With Ambient, you need to check the ztunnel pods logs for the relevant lines related to your application — If you have the necessary permissions — and do the same with the waypoint proxy.</p><p>Kiali makes this easier by consolidating all this information when available. You can choose whether to hide/show the proxy logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQpHnl2-A47vOGLMH_oA2Q.png" /><figcaption>Application logs with the Ztunnel and the Waypoint</figcaption></figure><p>The same applies to traces. Istio Ambient will offer trace processing at L7 — but only when the application is enrolled in a waypoint proxy and Istio is configured accordingly. The traces will be created using the waypoint proxy name.</p><p>Kiali will integrate all the traces related to your application in the “traces” tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSQzl1LhsC1hkoyVD_N_rg.png" /><figcaption>Application traces with Istio Ambient</figcaption></figure><p>So, there are a few things to consider when viewing the traces for your application. Can’t see any trace? You can check <a href="https://kiali.io/docs/faq/distributed-tracing/">this FAQ</a>.</p><p>If you think the problem might be in the waypoint proxy, click the waypoint proxy link to see the waypoint workload details. Here, verify the proxy status, the <a href="https://istio.io/latest/docs/ambient/usage/waypoint/#waypoint-traffic-types">waypoint traffic</a> type, and the services or the workloads that are enrolled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vN7tRvwpg_XZJNtDmA0UA.png" /><figcaption>Waypoint enrolled services</figcaption></figure><h3>When you’re an Administrator…</h3><p>Features like the Mesh graph, or access to the ztunnel deployment namespace, may require elevated privileges. In this section we will go through these additional features accessible by admins, and useful for troubleshooting applications with Istio Ambient mesh.</p><p>With permission to access the Istio control plane, you are able to access the Kiali Mesh page. This provides a revision-based overview of the Mesh components and their status. It is also possible to show waypoints and gateways. Whereas the Traffic Graph provides a topology of your application traffic, the Mesh page provides a topology of your mesh infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SHvz6JifGBwPZfcmWUJ2KA.png" /><figcaption>Kiali Mesh page</figcaption></figure><p>Select any component to get insight into metrics and configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*isVBlp3ZQV7AbVwtXrnVxw.png" /><figcaption>ztunnel component from the Mesh page</figcaption></figure><p>From the side-panel, click on the ztunnel workload link to navigate to the ztunnel workload details, it will include a specific Ztunneltab.</p><p>Under this tab Kiali provides in the UI, similar information that you can get using istioctl ztunnel-config :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*miMZaaQmVHr3UspWOuVzlA.png" /><figcaption>ztunnel metrics</figcaption></figure><p>The Ztunneltab is in addition to the standard workload detail tabs, so you also have quick access to things like the ztunnellogs.</p><blockquote>Hopefully this helps you when troubleshooting issues for applications running in Istio Ambient. You can easily recreate a testing environment <a href="https://medium.com/kialiproject/from-0-to-cloud-native-dev-environment-in-less-than-5-minutes-istio-ambient-kiali-e8e0ba7a39cb">following these steps</a>. And, (as always!), we are open to feedback.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae30baef5653" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/troubleshooting-istio-ambient-with-kiali-v2-ae30baef5653">Troubleshooting Istio Ambient with Kiali V2</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 0 to Cloud-Native Dev environment in less than 5 Minutes: Istio Ambient & Kiali]]></title>
            <link>https://medium.com/kialiproject/from-0-to-cloud-native-dev-environment-in-less-than-5-minutes-istio-ambient-kiali-e8e0ba7a39cb?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8e0ba7a39cb</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[istio-ambient]]></category>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[service-mesh]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Sat, 08 Feb 2025 02:56:42 GMT</pubDate>
            <atom:updated>2025-02-08T02:56:42.822Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometimes, having a complete and working example of an application in a service mesh on Kubernetes is not so easy to start from scratch. With these simple steps, you will have <strong>Istio Ambient</strong> running with <strong>Kiali</strong> and a test application with traffic for development with Minikube in less than 5 minutes.</p><h4>Prerequisites</h4><ul><li><a href="https://minikube.sigs.k8s.io/docs/start/">Minikube</a></li><li><a href="https://kubernetes.io/docs/tasks/tools/">kubectl</a></li></ul><h4>Getting Started</h4><p>First, we will download some Kiali hack scripts that will make the setup tasks very easy:</p><pre>curl -L -o install-istio.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-istio-via-istioctl.sh<br>curl -L -o download-istio.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/download-istio.sh<br>curl -L -o install-bookinfo.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-bookinfo-demo.sh</pre><p>And change the permissions to make them executable:</p><pre>chmod +x download-istio.sh <br>chmod +x install-istio.sh<br>chmod +x install-bookinfo.sh</pre><p>We will first start minikube:</p><pre>minikube start</pre><p>Then, we will download Istio latest version:</p><pre>./download-istio.sh</pre><p>We will need to check what was the Istio folder downloaded, and then, with that version, setup the <em>ISTIO_DIR</em> variable that we will use later. In this case, we have downloaded <em>istio-1.24.2</em>:</p><pre>ISTIO_DIR=&quot;$(pwd)/istio-1.24.2&quot;</pre><p>Then, we will run the Istio install script, that will download the latest stable version of Istio and install it with Prometheus, Grafana and Jaeger in the <em>istio-system</em> namespace:</p><pre>./install-istio.sh -c kubectl -cp ambient -id ${ISTIO_DIR}</pre><p>Then, we will install Kiali and the Bookinfo application with a traffic generator:</p><pre>kubectl apply -f ${ISTIO_DIR}/samples/addons/kiali.yaml<br>./install-bookinfo.sh -c kubectl -ai false -tg -w true --mongo --mysql -id ${ISTIO_DIR}</pre><p>Finally, we will do a port-forward to the Kiali service to access outside the cluster:</p><pre>kubectl port-forward svc/kiali 20001:20001 -n istio-system</pre><h4>And… Getting Finished!</h4><p>Now, we can access Kiali from the browser in <a href="http://localhost:20001/kiali/">http://localhost:20001/kiali</a> and see the traffic graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HcrKNnymRqH35kpeSJXNhA.png" /><figcaption>Kiali and Istio Ambient</figcaption></figure><p><em>Note: It might need some minutes for all the pods to start and generate traffic</em></p><h4>Furthermore</h4><p>The installation doesn’t have to be Istio Ambient. There are many options for the scripts, with the <em>-cp</em> option, you can select the profile in the Istio installation script, and <em>-h</em> will display a help menu with all the available options.</p><p>With these very simple steps we have an example running to discover all the features that Kiali has to offer on an Istio Ambient Mesh. If you want to know more, you can see all these features in <a href="https://kiali.io/docs/features/ambient/">kiali.io</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8e0ba7a39cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/from-0-to-cloud-native-dev-environment-in-less-than-5-minutes-istio-ambient-kiali-e8e0ba7a39cb">From 0 to Cloud-Native Dev environment in less than 5 Minutes: Istio Ambient &amp; Kiali</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing OpenShift Service Mesh 3 with Kiali and Grafana Tempo]]></title>
            <link>https://medium.com/kialiproject/installing-openshift-service-mesh-3-tp1-with-kiali-and-grafana-tempo-6b76881ceaef?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b76881ceaef</guid>
            <category><![CDATA[grafana-tempo]]></category>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[red-hat-openshift]]></category>
            <category><![CDATA[sail-operator]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Fri, 27 Sep 2024 13:59:41 GMT</pubDate>
            <atom:updated>2026-10-16T14:08:53.001Z</atom:updated>
            <content:encoded><![CDATA[<p>This article outlines the steps to install the new OpenShift Service Mesh 3 operator (based in the Istio upstream Sail operator), with Kiali as UI and Grafana Tempo for distributed traces, in an OpenShift cluster.</p><p>Versions used:</p><ul><li>Red Hat Tempo Operator: 0.22.0–3</li><li>Red Hat OpenTelemetry: 0.158.0–3</li><li>Red Hat Kiali: 2.27.5</li><li>Red Hat OpenShift Service Mesh 3 Operator: 3.4.3</li></ul><p>Prerequisites</p><ul><li>Tempo Operator Red Hat built from OLM installed</li><li>OpenTelemetry Red Hat built Operator from OLM installed</li><li>Red Hat OpenShift Service Mesh 3 Operator installed</li><li>Red Hat Kiali Operator installed</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*-yS68L3PAcuKLdW5pMV0Yg.png" /><figcaption>Installed operators</figcaption></figure><h4>Table of contents</h4><ul><li><a href="#d9b0">Grafana Tempo</a></li><li><a href="#5341">OpenTelemetry</a></li><li><a href="#7a56">Install Istio with OpenShift Service Mesh 3 Operator</a></li><li><a href="#f092">Install Kiali</a></li><li><a href="#9614">Troubleshooting</a></li></ul><h4>Grafana Tempo</h4><p>Create the Tempo namespace:</p><pre>oc create ns tempo</pre><p>Tempo needs storage, and for this, SeaWeedFS will be used. The SeaWeedFS resource will be created first:</p><pre>oc apply -f saeweedfs.yaml -n tempo</pre><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: tempo<br>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: seaweedfs-pv-claim<br>  namespace: tempo<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 256Mi<br>---<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: tempostack-dev-seaweedfs<br>  namespace: tempo<br>stringData:<br>  access_key_id: seaweedfs<br>  access_key_secret: seaweedfs123<br>  bucket: tempo-data<br>  endpoint: http://seaweedfs.tempo.svc.cluster.local:8333<br>type: Opaque<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    app: seaweedfs<br>  name: seaweedfs<br>  namespace: tempo<br>spec:<br>  selector:<br>    matchLabels:<br>      app: seaweedfs<br>  strategy:<br>    type: Recreate<br>  template:<br>    metadata:<br>      labels:<br>        app: seaweedfs<br>    spec:<br>      containers:<br>        - args:<br>            - mini<br>            - -dir=/data<br>            - -admin.port=12646<br>            - -master.telemetry=false<br>          env:<br>            - name: AWS_ACCESS_KEY_ID<br>              value: seaweedfs<br>            - name: AWS_SECRET_ACCESS_KEY<br>              value: seaweedfs123<br>            - name: S3_BUCKET<br>              value: tempo-data<br>          image: ghcr.io/chrislusf/seaweedfs:4.47<br>          livenessProbe:<br>            initialDelaySeconds: 10<br>            periodSeconds: 5<br>            tcpSocket:<br>              port: s3<br>          name: seaweedfs<br>          ports:<br>            - containerPort: 8333<br>              name: s3<br>          readinessProbe:<br>            initialDelaySeconds: 10<br>            periodSeconds: 5<br>            tcpSocket:<br>              port: s3<br>          volumeMounts:<br>            - mountPath: /data<br>              name: storage<br>      volumes:<br>        - name: storage<br>          persistentVolumeClaim:<br>            claimName: seaweedfs-pv-claim<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: seaweedfs<br>  namespace: tempo<br>spec:<br>  ports:<br>    - name: s3<br>      port: 8333<br>      protocol: TCP<br>      targetPort: s3<br>  selector:<br>    app: seaweedfs<br>  type: ClusterIP</pre><p>Create a TempoStack CR from the OpenShift console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h_1gYL_Ep4AeUJEXKgtuzw.png" /><figcaption>TempoStack CR Creation</figcaption></figure><pre>apiVersion: tempo.grafana.com/v1alpha1<br>kind: TempoStack<br>metadata:<br>  name: sample<br>  namespace: tempo<br>spec:<br>  networkPolicy:<br>    enabled: true<br>  observability:<br>    tracing:<br>      jaeger_agent_endpoint: &#39;localhost:6831&#39;<br>  resources:<br>    total:<br>      limits:<br>        cpu: 2000m<br>        memory: 2Gi<br>  tenants:<br>    authentication:<br>      - tenantId: &quot;f78bf974-aee2-47e8-8bb4-9ba41a41824a&quot;<br>        tenantName: default<br>    mode: openshift<br>  managementState: Managed<br>  template:<br>    gateway:<br>      enabled: true<br>    queryFrontend:<br>      jaegerQuery:<br>        enabled: true<br>  storage:<br>    secret:<br>      name: tempostack-dev-seaweedfs<br>      type: s3<br>  storageSize: 1Gi</pre><p>What has been modified:</p><ul><li>Configured the storage secret previously created</li><li>Created 1 tenant (default). The tenantId is required</li></ul><p>With this configuration, we can access to the Jaeger query console with:</p><pre>https://tempo-sample-gateway-tempo.apps-crc.testing/api/traces/v1/default/search </pre><p>We will create a ClusterRole and a ClusterRoleBinding to allow read access to the <em>north</em> tenant to all the authenticated users:</p><pre>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: tempostack-traces-reader<br>rules:<br>  - apiGroups:<br>      - &#39;tempo.grafana.com&#39;<br>    resources:<br>      - default<br>    resourceNames:<br>      - traces<br>    verbs:<br>      - &#39;get&#39;<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: tempostack-traces-reader<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: tempostack-traces-reader<br>subjects:<br>  - kind: Group<br>    apiGroup: rbac.authorization.k8s.io<br>    name: system:authenticated<br>EOF</pre><h4>Create an OpenTelemetry CR</h4><p>Create the Istio control plane namespace first:</p><pre>oc create ns istio-system</pre><p>Create the OpenTelemetry collector in the istio namespace from the OpenShift console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OCB5-YI_b1V3yH7JV_urQ.png" /><figcaption>OpenTelemetry collector CR</figcaption></figure><p>With the following setup:</p><pre>kind: OpenTelemetryCollector<br>apiVersion: opentelemetry.io/v1beta1<br>metadata:<br> name: otel<br> namespace: istio-system<br>spec:<br> config:<br>   exporters:<br>     otlp:<br>       endpoint: tempo-sample-gateway.tempo.svc.cluster.local:8090<br>       tls:<br>         insecure: false<br>         ca_file: &quot;/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt&quot;<br>       auth:<br>         authenticator: bearertokenauth<br>       headers:<br>         X-Scope-OrgID: &quot;default&quot;<br>   extensions:<br>     bearertokenauth:<br>       filename: &quot;/var/run/secrets/kubernetes.io/serviceaccount/token&quot;     <br>   receivers:<br>     otlp:<br>       protocols:<br>         grpc: {}<br>         http: {}<br>   service:<br>     extensions: [bearertokenauth]<br>     pipelines:<br>       traces:<br>         exporters:<br>           - otlp<br>         receivers:<br>           - otlp</pre><p>The header “X-Scope-OrgID” specify the name of the Tenant.</p><p>Now, we will create a cluster role with write permissions in the <em>Default</em> tenant and associate it to the OTel collector service account:</p><pre>kubectl apply -f - &lt;&lt;EOF<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: tempostack-traces-write<br>rules:<br>  - apiGroups:<br>      - &#39;tempo.grafana.com&#39;<br>    resources:<br>      - default<br>    resourceNames:<br>      - traces<br>    verbs:<br>      - &#39;create&#39;<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: tempostack-traces<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: tempostack-traces-write<br>subjects:<br>  - kind: ServiceAccount<br>    name: otel-collector<br>    namespace: istio-system<br>EOF</pre><h4>Install Istio with OpenShift Service Mesh 3 Operator</h4><p>Create an Istio resource with the OpenShift profile:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*nk8d1-7TYfMED3REr0UOWQ.png" /><figcaption>Creating an Istio CR with OpenShift Service Mesh 3</figcaption></figure><p>With the following configuration:</p><pre>kind: Istio<br>apiVersion: sailoperator.io/v1<br>metadata:<br>  name: default<br>spec:<br>  namespace: istio-system<br>  updateStrategy:<br>    inactiveRevisionDeletionGracePeriodSeconds: 30<br>    type: InPlace<br>  values:<br>    meshConfig:<br>      enableTracing: true<br>      extensionProviders:<br>        - name: otel-tracing<br>          opentelemetry:<br>            port: 4317<br>            service: otel-collector.istio-system.svc.cluster.local<br>  version: v1.30.5</pre><p>Create an Istio Telemetry resource:</p><pre>apiVersion: telemetry.istio.io/v1<br>kind: Telemetry<br>metadata:<br>  name: otel-demo<br>  namespace: istio-system<br>spec:<br>  tracing:<br>  - providers:<br>    - name: otel-tracing<br>    randomSamplingPercentage: 100</pre><p>Create an Istio CNI with the Istio profile:</p><pre>kubectl create ns istio-cni</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cspd0ZUNSutqKEOfEFDFEQ.png" /><figcaption>Create Istio CNI CR</figcaption></figure><p>Install the Prometheus (required) and Grafana (optional) add-ons in the Istio namespace:</p><pre>oc apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/prometheus.yaml -n istio-system<br>oc apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/grafana.yaml -n istio-system</pre><p>Instead of an addon, Prometheus can be installed with <a href="https://medium.com/p/584cbddb8c24">OpenShift monitoring</a>.</p><p><strong>Note</strong>: If we want to enable the Kiali and Grafana integration, a route should be created for the Grafana service.</p><p>In order to generate some traces, we will install the bookinfo demo application with traffic generation, and include it in the mesh. For this, we will follow the <a href="https://github.com/istio-ecosystem/sail-operator/blob/main/docs/addons/addons.adoc#deploy-gateway-and-bookinfo">Sail Operator instructions</a>.</p><p>Note the injection label is the following for this setup:</p><pre>kubectl label namespace bookinfo istio.io/rev=default</pre><p>We can also create an OpenShift route and send traffic in the browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fXyoiEKy9MMuhGRV0mCKeQ.png" /><figcaption>Bookinfo page</figcaption></figure><h4>Install Kiali</h4><p>Create a CR using with the Kiali Operator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sdkpVm9omyrlQtHyHED-CA.png" /></figure><p>With the Kiali configuration:</p><pre>kind: Kiali<br>apiVersion: kiali.io/v1alpha1<br>metadata:<br>  name: kiali<br>  namespace: istio-system<br>spec:<br>  version: default<br>  istio_namespace: istio-system<br>  deployment:<br>    logger:<br>      log_level: info<br>    view_only_mode: false<br>  external_services:<br>    grafana:<br>      enabled: true<br>      internal_url: &#39;http://grafana.istio-system:3000&#39;<br>      external_url: &#39;http://grafana-istio-system.apps-crc.testing&#39;<br>    tracing:<br>      enabled: true<br>      namespace_selector: true<br>      tempo_config:<br>        datasource_uid: a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00<br>        org_id: &#39;1&#39;<br>      auth:<br>        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt<br>        type: bearer<br>        use_kiali_token: true<br>      query_timeout: 15<br>      external_url: &#39;https://tempo-sample-gateway-tempo.apps-crc.testing/api/traces/v1/default/search&#39;<br>      internal_url: &#39;https://tempo-sample-gateway.tempo.svc.cluster.local:8080/api/traces/v1/default/tempo&#39;<br>      provider: tempo<br>      use_grpc: false  </pre><p>In order to correctly redirect Kiali to the right Grafana Tempo Datasource, the <em>datasource_uid</em> should be modified:</p><pre>spec:<br> external_services:<br>   tracing:<br>     tempo_config:<br>       org_id: &quot;1&quot;<br>       datasource_uid: &quot;a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00&quot;</pre><p>To redirect to the Jaeger UI in the Kiali <em>View in Tracing</em> link, the following setting can be updated (Takes the <em>Grafana</em> value by default):</p><pre>spec:<br> external_services:<br>   tracing:<br>     tempo_config:<br>       url_format: &quot;jaeger&quot;</pre><p>The <em>in_cluster_url</em> config should point to the gateway with the tenant name as part of the url, <a href="http://tempo-resource-gateway.tempo:8080/api/traces/v1/%7BtenantName%7D/tempo,"><em>http://tempo-resource-gateway.tempo:8080/api/traces/v1/{tenantName}/tempo</em>,</a> similar to:</p><pre>internal_url: &#39;http://tempo-sample-gateway.tempo:8080/api/traces/v1/default/tempo&#39;<br>provider: tempo</pre><p>Where <em>north</em> is the tenant name in this example config. It can also be used the Jaeger url (Without Tempo at the end):</p><pre>in_cluster_url: &#39;http://tempo-sample-gateway.tempo:8080/api/traces/v1/default&#39;<br>provider: jaeger</pre><p>When we access Kiali and select a workload, the Traces tab should be available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ka7H-iSmjqd88H_o-HQ4XQ.png" /><figcaption>Kiali Traces View</figcaption></figure><h4>Troubleshooting</h4><p>We can access to the Tempo Jaeger UI to validate if there are traces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HIksSLnE3R6v2RuPrAfmDg.png" /><figcaption>Tempo Jaeger UI</figcaption></figure><p>If there are no traces, we should check the Istio logs and look for any related issue. Are all the components healthy? Are there any issues with the URL configuration?</p><p>If there are traces, check the Kiali logs and see if there is any related issue. Is the Kiali configuration right? And the service URLs are correct?</p><p>If you reach to a <em>Certificate Signed by Unknown Authority</em> Error, you might be interested in <a href="https://medium.com/@yakovbeder/ossm-3-kiali-and-grafana-tempo-the-epic-quest-for-custom-certificates-mutual-trust-and-06004f2ab334">this article</a>.</p><p><em>Note: Edited to include the secure configuration and </em><a href="https://developers.redhat.com/articles/2023/08/01/how-deploy-new-grafana-tempo-operator-openshift"><em>Tempo multi-tenant</em></a><em> and remove TP Version</em></p><p><em>Note: Edited to update operator versions, use SeaweedFS and use just one tenant</em></p><p>After these steps we should have OpenShift Service Mesh 3 installed to generate traces, collected by the OpenTelemetry collector, managed in Tempo, and have Kiali to visualize our Mesh applications with the tracing information collected from Tempo.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b76881ceaef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/installing-openshift-service-mesh-3-tp1-with-kiali-and-grafana-tempo-6b76881ceaef">Installing OpenShift Service Mesh 3 with Kiali and Grafana Tempo</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Group Authorization with Kiali and Keycloak]]></title>
            <link>https://medium.com/kialiproject/group-authorization-with-kiali-and-keycloak-1bc4a3fed6c8?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bc4a3fed6c8</guid>
            <category><![CDATA[jwt-token]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Tue, 23 Jul 2024 13:24:07 GMT</pubDate>
            <atom:updated>2024-07-23T13:24:07.166Z</atom:updated>
            <content:encoded><![CDATA[<p>Kiali supports <a href="https://kiali.io/docs/configuration/authentication/">several authentication mechanisms</a>. In this post we will set up Kiali with the OpenID authentication strategy, using <a href="https://www.keycloak.org/">Keycloak</a> as the authentication provider. Keycloak will be configured to use groups for <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">Kubernetes RBAC Authorization</a>.</p><p>Kiali does not handle the authorization directly; instead, Kubernetes does it using the ID token passed by the authentication provider. The ID token contains the user identity: a list of properties that are encoded as a JWT (JSON Web Token). These properties are known as claims.</p><p>Kubernetes uses this information to authorize actions based on its policies. It reads different <a href="https://kubernetes.io/docs/reference/access-authn-authz/authorization/#request-attributes-used-in-authorization">claims</a> to provide the authorization based on different properties, and the <em>groups</em> claim is included.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*o4A7ei13O49kNxC2RNMz_Q.png" /><figcaption>The set of claims as a payload in a JWT token</figcaption></figure><p>The authentication provider needs to be configured properly in order to send the <em>groups</em> claim as part of the ID Token.</p><p><strong>Note</strong>: This post assumes that a realm and a client are already properly configured.</p><p>We will define a new <em>Client scope</em> to define the <em>groups</em> claim.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjNpy_gbJS4P0SfX5LoXkA.png" /><figcaption>Keycloak — Create client scope</figcaption></figure><p>The <em>Client scope</em> will include the following details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IxDU05hXVd6krl-M9X0BSw.png" /><figcaption>Keycloak — Groups client scope</figcaption></figure><p>We will add a new <em>Group Membership</em> mapper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-TJ0FdFvJYJmQr7eBXJA8Q.png" /><figcaption>Keycloak — Group Membership Mapper</figcaption></figure><p>Next, the g<em>roups Client scope</em> will be added to the <em>Client </em>used by Kiali. We will go to the <em>Clients</em> menu, click on the client, go to the <em>Client Scopes</em> tab and add the g<em>roups </em>client scope:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYRA1gDVwqij2XU_1IAMTg.png" /><figcaption>Keycloak — Client Scopes</figcaption></figure><p>We have the user Kiali added into the default group. Let’s create a <em>Role</em> and a <em>RoleBinding</em> now defining a <em>group</em> to allow access to the <em>bookinfo</em> namespace:</p><p><a href="https://raw.githubusercontent.com/josunect/yamls/main/roleBookinfo.yaml">https://raw.githubusercontent.com/josunect/yamls/main/roleBookinfo.yaml</a></p><p>Add the <em>roleBinding</em> to the group:</p><p><a href="https://raw.githubusercontent.com/josunect/yamls/main/roleBindingBookinfoGroup.yaml">https://raw.githubusercontent.com/josunect/yamls/main/roleBindingBookinfoGroup.yaml</a></p><p>After this change, the user <em>Kiali</em> should be able to login and see just the <em>bookinfo</em> namespace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XTtx8Aut8Uy9ZrcARMFGgw.png" /><figcaption>Kiali Traffic Graph for bookinfo namespace</figcaption></figure><p>In this post, we have learned how to implement<strong> group authorization</strong> for <strong>Kiali</strong> using <strong>Keycloak</strong> as authentication provider. This setup centralizes authentication and authorization, enhancing security and simplifying access management in the cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bc4a3fed6c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/group-authorization-with-kiali-and-keycloak-1bc4a3fed6c8">Group Authorization with Kiali and Keycloak</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Kiali and Grafana Tempo]]></title>
            <link>https://medium.com/kialiproject/integrating-kiali-and-grafana-tempo-9230821fb41a?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9230821fb41a</guid>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[grafana-tempo]]></category>
            <category><![CDATA[distributed-tracing]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Mon, 13 Nov 2023 19:07:59 GMT</pubDate>
            <atom:updated>2024-01-08T14:32:29.954Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Updated to reflect the latest changes in Kiali 1.78</blockquote><p>Kiali, serving as an Istio console, offers a means to visually represent Istio’s data and configuration. When Istio’s distributed tracing is activated, Kiali has the capability to present trace and span data using a bubble chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHyxFkAf27mgKDkCAjzGvA.png" /><figcaption>Bubble chart in Kiali showing Traces for a Workload</figcaption></figure><p>This optional feature historically integrates with the Jaeger distributed tracing tool. But with the release of Kiali version 1.75, <a href="https://grafana.com/oss/tempo/">Grafana Tempo</a> is now also supported as a distributed tracing back-end! The full set of features is available from Kiali version 1.78.</p><p>Kiali can both read and send traces. In its default configuration, Kiali reads the traces sent by Istio, from the Jaeger or Tempo API. Additionally, Kiali can be configured to send its own traces to the Trace ingester, for debugging purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*Oo1D11r8TWKxj-qKmWEKsQ.png" /><figcaption>Kiali, Istio and Grafana Tempo integration</figcaption></figure><p>Tempo has the capability to ingest traces in various formats, such as Jaeger or Zipkin. Kiali sends traces to Tempo in the OpenTelemetry format.</p><p>In order to display tracing data from Tempo, Kiali reads the traces from the Tempo API in OpenTelemetry format and internally transforms them into Jaeger traces.</p><h3>How to configure Tempo</h3><p>It is quite easy to configure Kiali with Grafana Tempo, just the following configuration is required (assuming the typical ports):</p><pre>external_services:<br>  ...<br>  tracing:<br>    enabled: true<br>    provider: &quot;tempo&quot;<br>    in_cluster_url: http://frontend-query.tempo:3200<br>    url: http://my-tempo-host:3200 <br>    use_grpc: true<br>    grpc_port: 9095<br>  ...<br>  grafana:<br>    in_cluster_url: http://grafana.istio-system:3000<br>    url: http://external-grafana-url</pre><p>Kiali links every trace and span with the proper URL for redirection to the Tracing tool. Because Grafana is the default Tempo UI, the Grafana URL needs to be set up also in the Kiali configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YRZfVyAzfEGd0yksVheBow.png" /><figcaption>Kiali redirects to the trace tool</figcaption></figure><p>For Kiali to appropriately redirect to the correct URLs, it’s necessary to establish a default <a href="https://grafana.com/docs/grafana/latest/datasources/tempo/">Tempo Datasource</a> in Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKqdxefCLlinbO0HUHkl6Q.png" /><figcaption>Istio trace in Grafana Tempo Datasource</figcaption></figure><p>gRPC is available for accessing the Tempo API. But it will still require the HTTP endpoint to be accessible.</p><p>The full set of features have been released. We would love to hear your feedback to keep improving the Grafana Tempo and Kiali integration!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9230821fb41a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/integrating-kiali-and-grafana-tempo-9230821fb41a">Integrating Kiali and Grafana Tempo</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup a multi-cluster environment with Istio and Kiali (Experimental support)]]></title>
            <link>https://medium.com/kialiproject/setup-a-multi-cluster-environment-with-istio-and-kiali-experimental-support-c2f64a9fba2?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2f64a9fba2</guid>
            <category><![CDATA[multi-cluster]]></category>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Mon, 12 Jun 2023 19:23:15 GMT</pubDate>
            <atom:updated>2023-06-26T13:31:43.285Z</atom:updated>
            <content:encoded><![CDATA[<p>Kiali v1.69 includes experimental support for multi-cluster, which allows to have Kiali with Istio Mesh installed across multiple Kubernetes clusters. It is now possible to visualize multiple workloads in the same graph from different clusters, as well as the workload’s health, the services, the applications and the Istio configurations, from the same Kiali instance.</p><p>Further details can be seen in the multi-cluster section in <a href="https://kiali.io/">Kiali.io</a>. In this article we will describe how to deploy a multi-cluster environment, install workloads distributed in different clusters, and how to set up Kiali to visualize the multi-cluster mesh from a single Kiali instance:</p><ul><li><a href="#e2a4">Requirements</a></li><li><a href="#3f0b">Installation</a></li><li><a href="#2590">Demo</a></li></ul><h3><strong>Requirements</strong></h3><p>The experimental support indicates limited support for multi-cluster. At this first stage, just the <a href="https://istio.io/latest/docs/setup/install/multicluster/">primary-remote</a> deployment model is supported. Also, Kiali will not be able to write (i.e. create/modify/delete) any resource in the remote clusters. Regarding authentication strategies, just anonymous authentication is supported. OpenID authentication is supported, but this support might depend on the provider. The purpose of this experimental release is to receive early feedback for the next iteration to GA status.</p><p>Istio supports different <a href="https://istio.io/latest/docs/ops/deployment/deployment-models/">deployment models</a>. The primary-remote model supports a single control plane and one or more data planes, across more than one cluster. A single Kiali will be deployed in the primary cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_hhYuJ9PaRHBNiQ_iopkJw.png" /><figcaption>Kiali in a primary-remote deployment model</figcaption></figure><p>Kiali requires a single, aggregated metrics endpoint that unifies the mesh metrics. The same is true for traces. The unified datastores should be accessible from the primary cluster. The mechanism for producing the unified store is outside the scope of Kiali. In this example, the metrics will be unified using Prometheus federation.</p><p>Kiali will identify the remote clusters by reading the Kiali remote cluster secrets that were created at Kiali installation time. The remote secrets consist of a service account, a role and a role binding in the remote cluster, and a secret with the remote service account credentials in a kubeconfig file that it is mounted in the Kiali pod. This file also indicates the name of the remote cluster.</p><p>There is a Kiali configuration item to define the cluster name. By default, the cluster name will be obtained from the Istio configuration. To avoid the auto-detection, as it may fail if istiod is not available when Kiali starts, it can be explicitly defined in the Kiali configuration. It is important both names match, so the remote cluster can be found in the telemetry queries.</p><p>Kiali will communicate with the remote clusters through the Kubernetes API server. The Istio CRDs don’t need to be installed in the remote clusters.</p><p>The supported authentication strategies are anonymous and OpenID (With the restrictions mentioned earlier). In this example we will use two minikube clusters with the anonymous authentication strategy.</p><h3><strong>How to install Kiali in a multi-cluster environment</strong></h3><p>The following environment will be installed:</p><ul><li>Primary cluster: east</li><li>Remote cluster: west</li><li>Minikube clusters</li><li>Anonymous authentication</li></ul><p>Requirements:</p><ul><li>Minikube (Tested in 1.30.1)</li><li>kubectl (Tested with 1.27)</li><li>istioctl (Tested in 17.2)</li></ul><p>Export the cluster names:</p><pre>export CTX_CLUSTER1=east<br>export CTX_CLUSTER2=west</pre><p>Export the Istio directory to install the addons:</p><pre>export ISTIO_DIR=~/bin/istio-1.17.2</pre><p>Create Minikube using the Kiali hack script. This script will create the Minikube profile and enable ingress, the image registry and metallb load balancer addon:</p><pre>curl -L -o k8s-minikube.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/k8s-minikube.sh<br>chmod +x k8s-minikube.sh<br><br>./k8s-minikube.sh start -lba &quot;20-30&quot; -mp $CTX_CLUSTER1<br>./k8s-minikube.sh start -lba &quot;40-50&quot; -mp $CTX_CLUSTER2</pre><p>Create a self-signed certificate to setup in both clusters to be able to communicate:</p><pre>mkdir -p /tmp/certs<br>pushd /tmp/certs<br><br>make -f ${ISTIO_SRC}/tools/certs/Makefile.selfsigned.mk root-ca<br><br>make -f ${ISTIO_SRC}/tools/certs/Makefile.selfsigned.mk cluster1-cacerts<br>make -f ${ISTIO_SRC}/tools/certs/Makefile.selfsigned.mk cluster2-cacerts<br><br>kubectl create namespace istio-system --context $CTX_CLUSTER1<br>kubectl create secret generic cacerts -n istio-system --context $CTX_CLUSTER1 \<br>      --from-file=cluster1/ca-cert.pem \<br>      --from-file=cluster1/ca-key.pem \<br>      --from-file=cluster1/root-cert.pem \<br>      --from-file=cluster1/cert-chain.pem<br><br>kubectl create namespace istio-system --context $CTX_CLUSTER2<br>kubectl create secret generic cacerts -n istio-system --context $CTX_CLUSTER2 \<br>      --from-file=cluster2/ca-cert.pem \<br>      --from-file=cluster2/ca-key.pem \<br>      --from-file=cluster2/root-cert.pem \<br>      --from-file=cluster2/cert-chain.pem<br><br>popd</pre><p>Setup the network for cluster east and install Istio and addons:</p><pre>cat &lt;&lt;EOF &gt; cluster1.yaml<br>apiVersion: install.istio.io/v1alpha1<br>kind: IstioOperator<br>spec:<br>  values:<br>    global:<br>      meshID: mesh1<br>      multiCluster:<br>        clusterName: east<br>      network: network1<br>EOF<br><br>istioctl install --skip-confirmation=true --set hub=gcr.io/istio-release --set values.meshConfig.enableAutoMtls=true --set values.gateways.istio-egressgateway.enabled=true --set values.gateways.istio-ingressgateway.enabled=true --set values.global.meshID=mesh1 --set values.global.multiCluster.clusterName=east --set values.global.network=network1 --set values.meshConfig.accessLogFile=/dev/stdout --set values.pilot.env.EXTERNAL_ISTIOD=true --context=&quot;${CTX_CLUSTER1}&quot; -f cluster1.yaml<br><br>kubectl apply -f ${ISTIO_DIR}/samples/addons/prometheus.yaml -n istio-system --context=&quot;${CTX_CLUSTER1}&quot;<br>kubectl apply -f ${ISTIO_DIR}/samples/addons/grafana.yaml -n istio-system --context=&quot;${CTX_CLUSTER1}&quot;<br>kubectl apply -f ${ISTIO_DIR}/samples/addons/jaeger.yaml -n istio-system --context=&quot;${CTX_CLUSTER1}&quot;</pre><p>Generate the east gateway, and expose Istiod and the services:</p><pre>${ISTIO_DIR}/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster ${CTX_CLUSTER1} --network network1 | istioctl --context=&quot;${CTX_CLUSTER1}&quot; install -y -f -<br><br>kubectl apply --context=${CTX_CLUSTER1} -n istio-system -f ${ISTIO_DIR}/samples/multicluster/expose-istiod.yaml<br>kubectl apply --context=${CTX_CLUSTER1} -n istio-system -f ${ISTIO_DIR}/samples/multicluster/expose-services.yaml</pre><p>Prepare the remote secrets to be used by Kiali:</p><pre>curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh<br>chmod +x kiali-prepare-remote-cluster.sh<br>./kiali-prepare-remote-cluster.sh --kiali-cluster-context ${CTX_CLUSTER1} --remote-cluster-context ${CTX_CLUSTER2}</pre><p>The previous command is using a helper script that creates a Kiali Service account, a role and a role binding in the remote cluster, and then creates a kubeconfig file to save it as a secret so Kiali can read it.</p><p>Install Kiali:</p><pre>kubectl config use-context ${CTX_CLUSTER1} <br>helm upgrade --install --namespace istio-system --set kubernetes_config.cache_enabled=false --set auth.strategy=anonymous --set deployment.logger.log_level=debug --set deployment.ingress.enabled=true --repo https://kiali.org/helm-charts kiali-server kiali-server </pre><p>Setup the network for cluster west:</p><pre>kubectl --context=${CTX_CLUSTER2} annotate namespace istio-system topology.istio.io/controlPlaneClusters=east<br>kubectl --context=${CTX_CLUSTER2} label namespace istio-system topology.istio.io/network=network2</pre><p>Install Istio service with remote profile, to set up this remote cluster to be managed by a control plane in the primary east cluster:</p><pre>DISCOVERY_ADDRESS=$(kubectl --context=east -n istio-system get svc istio-eastwestgateway -o jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;)<br><br>cat &lt;&lt;EOF &gt; cluster2.yaml<br>apiVersion: install.istio.io/v1alpha1<br>kind: IstioOperator<br>spec:<br>  profile: remote<br>  values:<br>    istiodRemote:<br>      injectionPath: /inject/cluster/west/net/network2<br>    global:<br>      remotePilotAddress: ${DISCOVERY_ADDRESS}<br>EOF<br><br>istioctl install -y --context=&quot;${CTX_CLUSTER2}&quot; -f cluster2.yaml</pre><p>Install Prometheus addon:</p><pre>kubectl --context=${CTX_CLUSTER2} apply -f ${ISTIO_DIR}/samples/addons/prometheus.yaml -n istio-system</pre><p>Generate gateway in cluster west:</p><pre>istioctl x create-remote-secret \<br>    --context=&quot;${CTX_CLUSTER2}&quot; \<br>    --name=west | \<br>    kubectl apply -f - --context=&quot;${CTX_CLUSTER1}&quot;<br><br>${ISTIO_DIR}/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster ${CTX_CLUSTER2} --network network2 | ${ISTIO_DIR}/bin/istioctl --context=${CTX_CLUSTER2} install -y -f -</pre><p>Set up Prometheus federation:</p><pre>kubectl patch svc prometheus -n istio-system --context ${CTX_CLUSTER2} -p &quot;{\&quot;spec\&quot;: {\&quot;type\&quot;: \&quot;LoadBalancer\&quot;}}&quot;<br><br>WEST_PROMETHEUS_ADDRESS=$(kubectl --context=west -n istio-system get svc prometheus -o jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;)<br><br>curl -L -o p8s.yaml https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/prometheus.yaml<br><br>sed -i &quot;s/WEST_PROMETHEUS_ADDRESS/$WEST_PROMETHEUS_ADDRESS/g&quot; p8s.yaml<br><br>kubectl --context=&quot;${CTX_CLUSTER1}&quot; apply -f p8s.yaml -n istio-system</pre><p>There will be a separate article to show how to setup unified Traces.</p><p>Install error rates application demo distributed between the two clusters:</p><pre>curl -L -o install-error-rates-demo.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/install-error-rates-demo.sh<br>chmod +x install-error-rates-demo.sh<br>./install-error-rates-demo.sh -c kubectl -dd true</pre><p>Get IP for Kiali access:</p><pre>minikube ip -p ${CTX_CLUSTER1}</pre><h3>Kiali multi-cluster demo</h3><p>At this point, we should be able to access the Kiali instance and see both clusters in the Mesh tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RC5YA-MBtj4py-bpdZYNeA.png" /></figure><p>We should be able to see graphs from the beta namespace in the remote west cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hunpUoHPRnIKvLZSv2olXw.png" /></figure><p>A video with a demo for an overview of Kiali Multi-cluster features is available:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbDrv57-EZ2g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbDrv57-EZ2g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbDrv57-EZ2g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/13e2fb767207f9723cb687740e9798d6/href">https://medium.com/media/13e2fb767207f9723cb687740e9798d6/href</a></iframe><p>This is just the first step for Kiali multi-cluster, to help going forward it’s important to get your community user feedback! We’d love to hear from you in comments, in Slack or in Github. Stay tuned for more blog posts, demos and tutorials about Kiali Multi-cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2f64a9fba2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/setup-a-multi-cluster-environment-with-istio-and-kiali-experimental-support-c2f64a9fba2">Setup a multi-cluster environment with Istio and Kiali (Experimental support)</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kiali support for Istio Ambient Mesh (Alpha)]]></title>
            <link>https://medium.com/kialiproject/kiali-support-for-istio-ambient-mesh-alpha-7f91ea8c6f54?source=rss-3b4d52a18d7a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f91ea8c6f54</guid>
            <category><![CDATA[istio-ambient]]></category>
            <category><![CDATA[kiali]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <dc:creator><![CDATA[Jcordoba]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 19:05:28 GMT</pubDate>
            <atom:updated>2023-04-26T19:05:28.017Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://istio.io/latest/blog/2022/get-started-ambient/">Istio Ambient Mesh</a> will be part of the Istio 1.18 release with <a href="https://preliminary.istio.io/latest/docs/releases/feature-stages/#feature-phase-definitions">Alpha status</a>. The Kiali add-on version will be v1.67, with initial support for Istio Ambient, to help identify the Ambient components of the mesh.</p><p>When Istio is installed with Ambient profile, Kiali will show a badge indicator in the control plane card:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOnCohH_uqkIBO4E4F-jNw.png" /><figcaption>Kiali showing in the Istio Control Plane an Istio Ambient badge</figcaption></figure><p>Whenever an application is added to the Ambient Mesh — labeling the namespace with the Ambient labels — an Ambient badge indicator will appear in the data plane card for the namespace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*lxbbjPxw6hFa080LCIcamw.png" /><figcaption>Data plane card for a namespace added into Ambient</figcaption></figure><p>That will automatically secure the communications using mTLS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*FYEcB_Z6vz6nsMRnjXizBw.png" /><figcaption>Kiali graph will show the secure communications with a locker</figcaption></figure><p>When the traffic of a workload is redirected through a <a href="https://istio.io/latest/blog/2023/rust-based-ztunnel/">ztunnel</a>, the layer 4 component of Ambient that provides mTLS, L4 authorization policies and telemetry, Kiali will show a ztunnel badge indicator in the workload details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*dVTXR8n1ESQIbZPVVlgzKA.png" /><figcaption>ztunnel badge indicator in the workload details view</figcaption></figure><p>Kiali also identifies when a <a href="https://istio.io/latest/blog/2023/waypoint-proxy-made-simple/">Waypoint proxy</a>, an Ambient component that operates in the L7 application layer, is deployed. Waypoint proxies are shown as part of the Workloads and Services lists with a “<em>W”</em> badge indicator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*anrWlC7Wi7bFc4GxiGJk8w.png" /><figcaption>Waypoint proxies workloads are identified in Kiali as part of the Ambient infrastructure</figcaption></figure><p>Kiali detects when a Waypoint proxy is applied to a whole namespace or to a specific service account, to check if a workload is managed by a Waypoint proxy, so it can show the Waypoint badge indicator in the workload details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*J_j4Owph0G4-v8tfaeVbOg.png" /><figcaption>Waypoint badge indicator in the workload details view</figcaption></figure><p>What if an application doesn’t have a sidecar? The “<em>Missing sidecar</em>” message will be replaced with an “<em>Out of mesh</em>” message if there is no sidecar and the application is not part of the Ambient Mesh.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RFpU48H0NMBkL1O00xSD4w.png" /><figcaption>Workloads are not part of the Mesh</figcaption></figure><blockquote>The Kiali behavior will be the same as before when the Istio profile is not Ambient.</blockquote><p>This is just a first approach to show Istio Ambient components in the Mesh. Kiali will continue to evolve, to support Istio Ambient Mesh changes and updates.</p><p>Please drop any feedback in the <a href="https://app.slack.com/client/T382U9E4U/CJVRXB09F">#kiali channel</a> on Istio slack!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f91ea8c6f54" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kialiproject/kiali-support-for-istio-ambient-mesh-alpha-7f91ea8c6f54">Kiali support for Istio Ambient Mesh (Alpha)</a> was originally published in <a href="https://medium.com/kialiproject">Kiali</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>