<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joe Sepich on Medium]]></title>
        <description><![CDATA[Stories by Joe Sepich on Medium]]></description>
        <link>https://medium.com/@bevvvvv19?source=rss-3e95251f90a6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*b1evV8SEHrFyzDi8</url>
            <title>Stories by Joe Sepich on Medium</title>
            <link>https://medium.com/@bevvvvv19?source=rss-3e95251f90a6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:21:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bevvvvv19/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Configuring Alerts and Rules in Grafana Mimir]]></title>
            <link>https://learning.sepich.dev/configuring-alerts-and-rules-in-grafana-mimir-cd9565dae397?source=rss-3e95251f90a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd9565dae397</guid>
            <category><![CDATA[mimirtool]]></category>
            <category><![CDATA[recording-rules]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[grafana-mimir]]></category>
            <dc:creator><![CDATA[Joe Sepich]]></dc:creator>
            <pubDate>Wed, 20 Sep 2023 17:20:09 GMT</pubDate>
            <atom:updated>2023-09-20T17:23:32.195Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Midjourney generation of what Mimir’s Grafana GUI looks like." src="https://cdn-images-1.medium.com/max/1024/1*gdGK_3sVDRAXOkQUVy43wg.png" /></figure><h3>Configuring Alerts and Rules in Grafana Mmimir</h3><p><a href="https://grafana.com/oss/mimir/?src=ggl-s&amp;mdm=cpc&amp;camp=nb-mimir-pm&amp;cnt=135432162936&amp;trm=grafana%20mimir&amp;device=c&amp;gclid=CjwKCAjwsKqoBhBPEiwALrrqiKCiuuPiKMg5F3hrPeDB0Avp5yDDUBqjN58P0m3G0IKvzjG9bF1bEhoCkuEQAvD_BwE">Grafana Mimir</a> is “an open source, horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.” The core functionality here is the ability to store monitoring metrics and query them to extract value. A primary reason to have this ability is to configure alerts. Alerts are queries that run at set intervals and fire a signal to an on-call agent when the query meets specific criteria. This guide lists the commands to roll out alerts and recording rules in a Grafan Mimir cluster.</p><h4>Prerequisites</h4><p>This guide will assume you have five things:</p><ol><li>A Grafana Mimir cluster is up and running</li><li>mimirtool installed</li><li>An alertmanager configuration</li><li>A collection of alert rules</li><li>(Optional) A collection of recording rules</li></ol><p>If you still need to get Mimir running, you can visit a previous article covering how to <a href="https://learning.sepich.dev/deploying-grafana-mimir-on-gcp-1d79e002a56">set up Grafana Mimir within GCP</a>.</p><p><a href="https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/">mimirtool</a> is a CLI binary that wraps API calls to Grafana Mimir. You can find installation instructions on Mimir’s <a href="https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/#installation">official documentation</a>.</p><p>You can also visit the official documentation for Prometheus and Mimir to find examples of what <a href="https://prometheus.io/docs/alerting/latest/configuration/">an alertmanager configuration</a> looks like. It also includes examples of yamls to declare <a href="https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/alerts.yaml">alerts</a> and <a href="https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/rules.yaml">recording rules</a>.</p><h3>Recording Rules Setup</h3><p><a href="https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-mimir-loki-managed-recording-rule/">Recording rules are metrics that are pre-computed queries</a> of a collection of other metrics. Namespaces logically separate rules, but the same ruler application evaluates them within the Mimir microservices stack. The frequency of that evaluation is part of the recording rule declaration.</p><p>Using the mimirtool CLI, you can execute the following write actions:</p><ul><li><strong>load </strong>— Initial upload of a set of recording rules. Will not execute if it requires overwriting existing configuration(s)</li><li><strong>diff </strong>— Similar to a terraform plan, this will show the difference between the production system and the proposed ruleset.</li><li><strong>sync </strong>— This write operation will ensure the production system looks exactly like the provided ruleset. Syncing includes <strong>deleting </strong>and updating existing rules.</li></ul><pre>mimirtool rules load path/to/rules.yaml<br>mimirtool rules diff path/to/rules.yaml<br>mimirtool rules sync path/to/rules.yaml</pre><p>You can also read what is on the cluster using both <strong>list</strong> and <strong>get</strong>.</p><pre>mimirtool rules list<br>mimirtool rules get namespace rule_group</pre><p>Finally, you can remove existing rules using the delete command.</p><pre>mimirtool rules delete path/to/rules.yaml</pre><p>If you need assistance getting your rules yaml setup, there is a brief example below from the <a href="https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/rules.yaml">Mimir-provided setup</a>. (rules.yaml)</p><pre>---<br>namespace: internal<br>groups:<br>    - name: backend_throughput<br>      interval: 5m<br>      rules: ...<br>    ....<br>---<br>namespace: mimir<br>groups:<br>- name: mimir_api_1<br>  rules:<br>  - expr: histogram_quantile(0.99, sum(rate(cortex_request_duration_seconds_bucket[1m]))<br>      by (le, cluster, job))<br>...</pre><h3>Alert Setup</h3><p>Alerts include a query and expression describing the condition to fire an alert. When an alert fires, it will contact an on-call agent based on an alertmanager configuration. Make sure you have the alertmanager enabled within the Mimir microservices stack since it is not a required component.</p><p>The <a href="https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/#tenant-configurations">mimirtool uploads and removes alerts</a> from the cluster. The supported commands are:</p><ul><li><strong>verify </strong>— This will verify that the configuration of the alertmanager and alerts provided is valid.</li><li><strong>load </strong>— Same as rules load. Upload the configuration to the cluster.</li><li><strong>delete </strong>— Deletes the cluster’s current configuration.</li></ul><pre>mimirtool alertmanager verify path/to/alertmanager-config.yaml path/to/alerts.yaml<br>mimirtool alertmanager load path/to/alertmanager-config.yaml path/to/alerts.yaml<br>mimirtool alertmanager delete</pre><p>If you need assistance getting your alertmanager configuration and alert yaml setup, there are abbreviated examples below from the <a href="https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/alerts.yaml">Mimir-provided setup</a>. (alerts.yaml)</p><pre>---<br>groups:<br>- name: mimir_alerts<br>  rules:<br>  - alert: MimirIngesterUnhealthy<br>    annotations:<br>      message: Mimir cluster {{ $labels.cluster }}/{{ $labels.namespace }} has {{<br>        printf &quot;%f&quot; $value }} unhealthy ingester(s).<br>      runbook_url: https://grafana.com/docs/mimir/latest/operators-guide/mimir-runbooks/#mimiringesterunhealthy<br>    expr: |<br>      min by (cluster, namespace) (cortex_ring_members{state=&quot;Unhealthy&quot;, name=&quot;ingester&quot;}) &gt; 0<br>    for: 15m<br>    labels:<br>      severity: critical<br>  - alert: MimirRequestErrors<br>    annotations:<br>      message: |<br>        The route {{ $labels.route }} in {{ $labels.cluster }}/{{ $labels.namespace }} is experiencing {{ printf &quot;%.2f&quot; $value }}% errors.<br>      runbook_url: https://grafana.com/docs/mimir/latest/operators-guide/mimir-runbooks/#mimirrequesterrors<br>    expr: |<br>      100 * sum by (cluster, namespace, job, route) (rate(cortex_request_duration_seconds_count{status_code=~&quot;5..&quot;,route!~&quot;ready&quot;}[1m]))<br>        /<br>      sum by (cluster, namespace, job, route) (rate(cortex_request_duration_seconds_count{route!~&quot;ready&quot;}[1m]))<br>        &gt; 1<br>...</pre><p>This alertmanager configuration will send critical alerts to better uptime and all alerts to a Slack channel called “alerts.” (alertmanager-config.yaml)</p><pre>global:<br>  # The slack_api_url should be replaced with your actual Slack webhook URL.<br>  slack_api_url: &quot;&lt;slack_channel&gt;&quot;<br><br>route:<br>  # Send all alerts to the &#39;slack-notifications&#39; receiver defined below.<br>  group_by: [&#39;alertname&#39;]<br><br>  # default<br>  receiver: &#39;slack-notifications&#39;<br>  routes:<br>    - receiver: &#39;better-uptime&#39;<br>      match:<br>        severity: &#39;critical&#39;<br><br>receivers:<br>  - name: &#39;slack-notifications&#39;<br>    slack_configs:<br>      - channel: &#39;#alerts&#39;<br>        send_resolved: true # Set to true if you want resolved alerts to be sent as well.<br>  - name: &#39;better-uptime&#39;<br>    webhook_configs:<br>      - url: &#39;&lt;better uptime webhook URL&gt;&#39;<br>        send_resolved: true</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd9565dae397" width="1" height="1" alt=""><hr><p><a href="https://learning.sepich.dev/configuring-alerts-and-rules-in-grafana-mimir-cd9565dae397">Configuring Alerts and Rules in Grafana Mimir</a> was originally published in <a href="https://learning.sepich.dev/">Continuously Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Grafana Mimir on GCP]]></title>
            <link>https://learning.sepich.dev/deploying-grafana-mimir-on-gcp-1d79e002a56?source=rss-3e95251f90a6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d79e002a56</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[grafana-mimir]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Joe Sepich]]></dc:creator>
            <pubDate>Mon, 15 May 2023 19:13:30 GMT</pubDate>
            <atom:updated>2023-05-15T19:13:30.317Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is Grafana Mimir?</h3><p>Grafana Labs states, “<a href="https://github.com/grafana/mimir">Grafana Mimir</a> is an open source, horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.” Released in <a href="https://grafana.com/blog/2022/03/30/qa-with-our-ceo-about-grafana-mimir/">late March of 2022</a>, Grafana Labs created Mimir as an internal project to replace Grafana’s usage of Cortex as long-term Prometheus storage.</p><h3>Grafana and Prometheus Monitoring</h3><p>Grafana and Prometheus are the industry standard open-source tools used in monitoring distributed systems by operations teams. Grafana visualizes time series in user-created dashboards and alerts when observed series meet configured criteria.</p><p>Prometheus is a pull-based monitoring tool that scrapes endpoints of the systems that it is monitoring to aggregate the metrics data into a central store. Prometheus utilizes PromQL as a query language to explore the time series data.</p><h3>Long Term Storage</h3><p>From an operational perspective, Prometheus works excellently in the short term (15-day default retention). Data retention exists purely on disk in a structure that cannot scale. A remote read/write API has been incorporated into the software, enabling offloading of long-term storage to third-party systems. The leading third-party tools used for this purpose are Thanos, Cortex, VictoriaMetrics, and InfluxDB. Grafana Labs is now transitioning its cortex system to Mimir, built upon the same software library as Cortex and Thanos.</p><h3>Deploying Mimir</h3><p>In this guide, we will deploy Mimir onto infrastructure hosted within the Google Cloud Platform. The deployment of Prometheus and Grafana should already be complete before proceeding. We will use the helm chart distributed by Grafana to run Mimir in microservices mode. There are three dependencies that the chart requires to run:</p><ul><li>A GCS bucket to store objects.</li><li>IAM access to that bucket.</li><li>A GKE cluster to run on.</li></ul><h3>GCP Setup</h3><h4>GCS Bucket</h4><p>Mimir leverages block storage by storing blocks of time series data within object storage. Create two buckets, one for the ruler and one for the time series database. According to the <a href="https://cloud.google.com/storage/docs/creating-buckets#storage-create-bucket-cli">GCP documentation</a>, you can run the following:</p><pre>gsutil mb gs://mimir-tsdb<br>gsutil mb gs://mimir-ruler</pre><p>Note that bucket names must be globally unique, so you must choose your own.</p><p><strong>Service Account (Only autopilot)</strong></p><p>The resource running Mimir will need access to the storage bucket created in the previous step. If you run GKE in autopilot mode, you must create a separate service account for authentication. Follow the instructions on <a href="https://cloud.google.com/iam/docs/creating-managing-service-accounts#iam-service-accounts-create-console">GCP documentation</a> and make sure the account has at least read/write access to your bucket.</p><p>The same permissions will need configuration in GKE standard, but this can be done by configuring API access on the node level, eliminating the need to create a service account.</p><h4>GKE Cluster</h4><p>Create a cluster using <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster">GCP documentation</a>. Note that the default configuration from Grafana for Mimir takes roughly 20 vCPU and 100 GB of memory, so provision the node pool accordingly if you are using a standard cluster. If you allow GCP to scale enough nodes automatically, you only need to worry about the ratio of vCPU to memory on each node.</p><p>For kubectl and helm commands, you can authenticate through the gcloud CLI to grab the configuration. Run the following:</p><pre>gcloud container clusters get-credentials &lt;CLUSTER-NAME&gt;</pre><p>You may need to supply optional arguments for the region or zone. See <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl">GCP documentation</a>.</p><h3>Helm Chart</h3><p>Find the mimir-distributed helm chart from Grafana on <a href="https://github.com/grafana/helm-charts/tree/main/charts/mimir-distributed">GitHub</a>. This article builds off the sample values file “small.yaml”.</p><h4>Configuration</h4><p>Small.yaml contains values for a cluster to support around 1 million active series. An active series corresponds to each set of labels describing a tracked metric. The yaml values here are the configuration for the cluster provisioning, not the app configuration. Once the cluster is running, you must tailor these values to your specific monitoring feed. Do note that the primary resources used for pure ingestion are the distributor and ingester. Do not underprovision the distributor, as it will have memory spikes and is vulnerable to Out Of Memory (OOM) errors when underprovisioned.</p><p><strong>Mimir App Configuration</strong></p><p>Configuration for the mimir application is done within a giant yaml string within the values yaml. You can find all configuration parameters in the <a href="https://grafana.com/docs/mimir/latest/operators-guide/configuring/reference-configuration-parameters/">official documentation</a>. This is where the configuration will point at your GCS bucket.</p><p>Place blocks defining GCS storage configuration in the <em>ruler_storage, alertmanager_storage, </em>and <em>store_gateway</em>. Each section requires the backend of GCS specified and a GCS config block that points at the bucket by name. If you are using GKE autopilot, you can provide the service account credentials by pasting the full JSON text. The field does NOT take a path to a JSON file. On standard GKE, configure the node pool with the desired API access instead of using a service account. The example is below:</p><pre>ruler_storage:<br>  backend: gcs<br>    gcs:<br>      bucket_name: mimir-ruler<br>      service_acccount: |<br>      {<br>        &quot;Type&quot;: &quot;service_account&quot;,<br>        …<br>      }</pre><h3>Deployment</h3><p>To deploy the configuration, you must add the helm repository to pull from, and then you can install it.</p><pre># Add the repository<br>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br>helm install &lt;mimir cluster name&gt; grafana/mimir-distributed</pre><p>Note that the setup does not expose the service externally by default. You can add a load balancer resource to expose the configured NGINX pod by deploying the following configuration in a file named mimir-nginx-lb.yaml.</p><pre>kubectl apply -f mimir-nginx-lb.yaml</pre><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  annotations:<br>    cloud.google.com/load-balancer-type: &quot;Internal&quot;<br>  name: mimir-nginx-internal-lb<br>spec:<br>  selector:<br>    app.kubernetes.io/component: nginx<br>    app.kubernetes.io/instance: mimir<br>    app.kubernetes.io/name: mimir<br>  type: LoadBalancer<br>  ports:<br>    - port: 80<br>      targetPort: 8080</pre><h3>Updates</h3><p>You can update your deployment at any time by updating your small.yaml (or whatever you named it) and then running:</p><pre>helm upgrade &lt;mimir cluster name&gt; grafana/mimir-distributed</pre><h3>Migration</h3><p>There are two main components to migration to Mimir from Prometheus:</p><ol><li>Migration of existing time series data</li><li>Plugging in Mimir between Prometheus and grafana</li></ol><h3>Historical Data Migration</h3><p>You can follow the <a href="https://grafana.com/docs/mimir/latest/migration-guide/migrating-from-thanos-or-prometheus/">documentation</a> to migrate “historical” time series data from Prometheus to Mimir.</p><p>It consists of copying data from the local Prometheus storage to the object bucket and then updating the meta.json files to the Mimir specification.</p><p>Text Directions:</p><ol><li>Get directories to copy</li><li>Copy found directories to GCS</li><li>Download metaconvert util</li><li>Add execution permissions to the util binary</li><li>Convert utils</li></ol><p>Commands:</p><ol><li>find /var/lib/prometheus/metrics2 -name chunks -exec dirname {} \;</li><li>gsutil -m cp -r &lt;DIRECTORY&gt; gs://mimir-tsdb/anonymous</li><li>curl -LO <a href="https://github.com/grafana/mimir/releases/latest/download/metaconvert-linux-amd64">https://github.com/grafana/mimir/releases/latest/download/metaconvert-linux-amd64</a></li><li>chmod +x metaconvert-linux-amd64</li><li>./metaconvert -backend=gcs -gcs.bucket-name=mimir-tsdb -tenant=anonymous</li></ol><h3>Linking Systems</h3><h3>Grafana Data Source</h3><p>From Grafana’s point of view, Mimir is a Prometheus data source. Follow the same Prometheus data source configuration and point it at Mimir. The default NGINX from the mimir-distributed helm chart has the query-frontend at http://&lt;nginx-ip&gt;:&lt;nginx-port&gt;/prometheus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*R8kAQTWV_UImBJOHj6wCLg.png" /></figure><p>If you deployed the suggested load balancer, find the IP by running kubectl get services and viewing the External IP with the configured port 80.</p><h3>Prometheus Remote Writer</h3><p>Prometheus uses its remote read/write API to push data to Mimir. Point the remote write configuration at the distributor, which is also on the default NGINX load balancer at http://&lt;nginx-ip&gt;:&lt;nginx-port&gt;/api/v1/push.</p><pre>remote_write:<br>  - url: http://&lt;nginx-ip&gt;:&lt;nginx-port&gt;/api/v1/push</pre><p>If you deployed the suggested load balancer, find the IP by running kubectl get services and viewing the External IP with the configured port 80.</p><h3>Parameter Tuning</h3><p>Optimal parameters will vary depending on the metrics and machines you are monitoring. Find the configuration parameters for mimir in this <a href="https://grafana.com/docs/mimir/latest/operators-guide/configuring/reference-configuration-parameters/">documentation</a>. The following two sections indicate parameters that break down some default limits.</p><p>The Prometheus remote writer can also be tuned. Prometheus documentation has an excellent <a href="https://prometheus.io/docs/practices/remote_write/">article on tuning it</a> and explaining how it works.</p><h3>Ingestion Limits</h3><p><a href="https://grafana.com/docs/mimir/latest/operators-guide/configuring/reference-configuration-parameters/#distributor">Distributor Instance Limits</a>: Set max_inflight_push_requests to 0</p><p><a href="https://grafana.com/docs/mimir/latest/operators-guide/configuring/reference-configuration-parameters/#limits">Limits</a>:</p><ul><li>ingestion_rate -&gt; per-tennant samples per second limit</li><li>max_global_series_per_user -&gt; 0</li><li>max_global_series_per_metric -&gt; 0</li><li>max_label_value_length may need to be increased depending on your labels</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d79e002a56" width="1" height="1" alt=""><hr><p><a href="https://learning.sepich.dev/deploying-grafana-mimir-on-gcp-1d79e002a56">Deploying Grafana Mimir on GCP</a> was originally published in <a href="https://learning.sepich.dev/">Continuously Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>