<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roman Khavronenko on Medium]]></title>
        <description><![CDATA[Stories by Roman Khavronenko on Medium]]></description>
        <link>https://medium.com/@romanhavronenko?source=rss-8c1c0cd81cd7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4IQVryntYdnr0LuuRInSRw.jpeg</url>
            <title>Stories by Roman Khavronenko on Medium</title>
            <link>https://medium.com/@romanhavronenko?source=rss-8c1c0cd81cd7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:31:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@romanhavronenko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[VictoriaMetrics: PromQL compliance]]></title>
            <link>https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e?source=rss-8c1c0cd81cd7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4318203f51e</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[victoriametrics]]></category>
            <dc:creator><![CDATA[Roman Khavronenko]]></dc:creator>
            <pubDate>Fri, 08 Oct 2021 20:13:54 GMT</pubDate>
            <atom:updated>2021-10-11T07:34:08.503Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://docs.victoriametrics.com/MetricsQL.html">MetricsQL</a> is a query language inspired by <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/">PromQL</a>. It is used as a primary query language in <a href="https://github.com/victoriaMetrics/victoriaMetrics">VictoriaMetrics</a>, time series database and monitoring solution. MetricsQL claims to be backward-compatible with PromQL, so Grafana dashboards backed by a Prometheus datasource should work the same after switching from Prometheus to VictoriaMetrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2vutYeSHF6DMxFS8UhszWQ.jpeg" /><figcaption><a href="https://www.flickr.com/photos/volvob12b/9794565953/in/photolist-fVvHdM-LZLEQJ-MVAkgk-22fLrVd-24Sjewq-ompP93-8rKJrQ-aF6a7x-22mB3L3-FaXZB7-dRopkc-MMig4k-RhKFby-22Ged9W-TxhdNj-uavR53-VjW7as-UGCPT1-njQYpC-22FcaBn-26ZXNJj-27GnRbS-24qF4mj-JL8Mge-7CZX2z-VBAc3Q-gWD3oo-Dcscwi-25hUjPT-FSad5Z-HWKMaK-H9NKMz-LxTwtS-ro4VNR-YaAVWD-2aCyQPN-Th364L-T9RoQE-tJtdxz-28USEVA-XLfi4b-Df5aPa-GLQab7-qMNDr-83VWYA-T4Zvyv-G1CiVc-EHaVFn-HNfYX9-id55vR">Photo</a> by <a href="https://www.flickr.com/people/volvob12b/">Bernard Spragg</a></figcaption></figure><p>However, VictoriaMetrics is not 100% compatible with PromQL and never will be. Please read on and we will discuss why that is.</p><p>For a long time, there was no way to measure compatibility with PromQL. There was not even a fully defined <a href="https://promlabs.com/blog/2020/08/06/comparing-promql-correctness-across-vendors#what-is-correct-in-the-absence-of-a-specification">PromQL specification</a>. But, some time ago, the <a href="https://prometheus.io/blog/2021/05/03/introducing-prometheus-conformance-program/">Prometheus Conformance Program</a> was announced with the aim to certify software with a mark of compatibility with Prometheus — “Upon reaching 100%, the mark will be granted&quot;. The open-source tool, <a href="https://github.com/prometheus/compliance">prometheus/compliance</a> was created to check for compatibility.</p><p>Compatibility is measured in quite a simple way— the tool requires a configuration file with a <a href="https://github.com/prometheus/compliance/blob/6d63e44ca06d317c879b7406ec24b01a82213aa0/promql/promql-compliance-tester.yml#L107">list of PromQL queries to run</a>, a Prometheus server to use as a reference and any other software meant to be tested. The tool sends PromQL queries to both Prometheus and the tested software, and if their responses don&#39;t match — it marks the query as having failed.</p><h4>Compliance testing</h4><p>We ran compatibility testing between Prometheus <a href="https://github.com/prometheus/prometheus/releases/tag/v2.30.0">v2.30.0</a> and VictoriaMetrics <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.67.0">v1.67.0</a> and got the following result:</p><pre>====================================================================<br>General query tweaks:<br>*  VictoriaMetrics aligns incoming query timestamps to a multiple of the query resolution step.<br>====================================================================<br>Total: 385 / 529 (72.78%) passed, 0 unsupported</pre><p>According to the test, VictoriaMetrics failed 149 tests and was compatible with Prometheus by 72.59% of the time. Let’s take a closer look at the queries that failed.</p><h4>Keeping metric name</h4><p>According to PromQL, functions that transform a metric&#39;s data should <a href="https://github.com/prometheus/prometheus/issues/380">drop the metric name from the result</a>, since the meaning of the initial metric has changed. However, this approach has some drawbacks. For example, the max_over_time function calculates the max value of the series without changing its physical meaning. Therefore, MetricsQL <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674">keeps the metric name for such functions</a>. It also enables queries over multiple metric names: max_over_time({__name__=~&quot;process_(resident|virtual)_memory_bytes&quot;}[1h]). While in PromQL such query fails with vector cannot contain metrics with the same labelset error.</p><p>Hence, test suit functions like *_over_time, ceil , floor , round , clamp_* , holt_winters , predict_linear in VictoriaMetrics do intentionally contain the metric name in the results:</p><pre>QUERY: avg_over_time(demo_memory_usage_bytes[1s])<br>-      Metric: s`{instance=&quot;demo.promlabs.com:10002&quot;, job=&quot;demo&quot;, type=&quot;buffers&quot;}`,<br>+     Metric: s`demo_memory_usage_bytes{instance=&quot;demo.promlabs.com:10002&quot;, job=&quot;demo&quot;, type=&quot;buffers&quot;}`,</pre><p>There were <strong>92 (~17% of 529 tests total)</strong> such queries in the test suite which failed because the metric name is present in the response from VictoriaMetrics, while the values in the response are identical. VictoriaMetrics isn&#39;t going to change this behavior as their users find this is more logical and <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1384">rely on it</a>.</p><h4>Better rate()</h4><p>rate and increase functions are some of the most frequently used functions in PromQL. While the logic behind these two is relatively simple and clear, the devil is in the <a href="https://github.com/prometheus/prometheus/issues/3746">details</a>.</p><p>MetricsQL intentionally has a slightly different implementation of rate and increase . It takes into account the last sample on the previous interval which allows capturing all the information from the time series when calculating rate or increase:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CB40caoHVgzC0j7zjY7peQ.png" /><figcaption>VictoriaMetrics captures the last data point from the previous interval when calculating increase()</figcaption></figure><p>Prometheus, in this case, loses the metric increase from the last sample in the previous interval and the first sample of the current interval:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZQjxfIIArhxzU8AkGYFDQ.png" /><figcaption>Prometheus accounts only for points captured by the interval when calculating increase(), losing the data that was before it.</figcaption></figure><p>Additionally, MetricsQL automatically increases the interval in square brackets (aka lookbehind window) if there aren&#39;t enough samples in the interval for calculating rate and increase. This solves the issue of <a href="https://www.percona.com/blog/2020/02/28/better-prometheus-rate-function-with-victoriametrics/">unexpected &quot;No Data&quot; errors when zooming in</a>.</p><p>MetricsQL doesn&#39;t apply extrapolation when calculating rate and increase. This solves the issue of fractional increase() results over integer counters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eHZaZLOoKFsv8Hmwg_PHsA.png" /><figcaption>increase() query over time series generated by integer counter results in decimal values for Prometheus due to extrapolation.</figcaption></figure><p>It is <a href="https://www.robustperception.io/what-range-should-i-use-with-rate">quite important</a> to choose the correct lookbehind window for rate and increase in Prometheus. Otherwise, incorrect or no data may be returned. <a href="https://grafana.com/">Grafana</a> even introduced a special variable <a href="https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/">$__rate_interval</a> to address this issue, but it may cause more problems than it solves:</p><ul><li>Users need to configure the scrape interval value in datasource settings to get it to work;</li><li>Users still need to add $__rate_interval manually to every query that uses rate;</li><li>It won&#39;t work if the datasource stores metrics with different scrape intervals (e.g. global view across multiple datasources);</li><li>It only works in Grafana.</li></ul><p>In MetricsQL, a lookbehind window in square brackets may be omitted. VictoriaMetrics automatically selects the lookbehind window depending on the current step, so rate(node_network_receive_bytes_total) works just as rate(node_network_receive_bytes_total[$__interval]). And even if the interval is too small to capture enough data points, MetricsQL will automatically expand it. That&#39;s why queries like deriv(demo_disk_usage_bytes[1s]) return no data for Prometheus and VictoriaMetrics expands the lookbehind window prior to making calculations.</p><p>There are <strong>39 (~7% of 529 tests total)</strong> queries (rate, increase, deriv, changes, irate, idelta, resets, etc.) exercising this logic which cause the difference in results between VictoriaMetrics and Prometheus:</p><pre>QUERY: rate(demo_cpu_usage_seconds_total[5m])<br>-           Value:     Inverse(TranslateFloat64, float64(1.9953032056421414)),<br>+           Value:     Inverse(TranslateFloat64, float64(1.993400981075324)),</pre><p>For more details about how rate/increase works in MetricsQL please check <a href="https://docs.victoriametrics.com/MetricsQL.html#rate">docs</a> and <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1215#issuecomment-850305711">example on github</a>.</p><h4>NaNs</h4><p>NaNs are unexpectedly complicated. Let&#39;s begin with the fact that in <a href="https://www.robustperception.io/get-thee-to-a-nannary">Prometheus there are two types of NaNs</a>: <a href="https://github.com/prometheus/prometheus/blob/19152a45d8a8f841206d321f79a60ab6d365a98f/pkg/value/value.go#L22">normal NaN</a> and <a href="https://github.com/prometheus/prometheus/blob/19152a45d8a8f841206d321f79a60ab6d365a98f/pkg/value/value.go#L28">stale NaN</a>. Stale NaNs are used as &quot;staleness makers&quot; — special values used to identify a time series that had become stale. VictoriaMetrics didn&#39;t initially support this because VictoriaMetrics needed to integrate with many systems beyond just Prometheus and had to have a way to detect staleness uniformly for series ingested via Graphite, Influx, OpenTSDB and other supported data ingestion protocols. Support of Prometheus staleness markers was <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526">recently added</a>.</p><p>Normal NaNs are results of mathematical operations, e.g. 0/0=NaN. However, in OpenMetrics there is <a href="https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#nan">no special meaning or use case for NaNs</a>.</p><p>While NaNs are expected when evaluating mathematical expressions, it is not clear how useful they are for users, or if there are any benefits to return NaNs in the result. It looks like the opposite is true because users are <a href="https://stackoverflow.com/questions/53430836/prometheus-sum-one-nan-value-result-into-nan-how-to-avoid-it">often</a> <a href="https://github.com/prometheus/prometheus/issues/7637">confused</a> <a href="https://github.com/prometheus/prometheus/issues/6780">with</a> the <a href="https://github.com/prometheus/prometheus/issues/6645">received</a> <a href="https://stackoverflow.com/questions/47056557/how-to-gracefully-avoid-divide-by-zero-in-prometheus">results</a>.</p><p>MetricsQL consistently deletes NaN from query responses. This behavior is intentional because there is no meaningful way to use such results. That&#39;s why testing queries such as demo_num_cpus * NaN or sqrt(-demo_num_cpus) return an empty response in MetricsQL, and returns NaNs in PromQL.</p><p>There were <strong>6 (~1% of 529 tests total)</strong> queries in thetest suite expecting NaNs in responses: sqrt(-metric) , ln(-metric) , log2(-metric) , log10(-metric) and metric * NaN .</p><h4>Negative offsets</h4><p>VictoriaMetrics supports negative offsets and Prometheus also does as well starting with version <a href="https://github.com/prometheus/prometheus/releases/tag/v2.26.0">2.26</a> if a specific feature flag is enabled. However, query results are different even with the enabled feature flag due to the fact that Prometheus continues the last value of the metric during the additional 5min:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSM3WanK--IZ3MPe7DuEgQ.png" /><figcaption>VictoriaMetrics vs Prometheus negative offset query. VictoriaMetrics response value is shifted by 1e7 to show the difference between the lines visually. Without this shift, they are identical except the last 5min.</figcaption></figure><p>Such behavior was unexpected to us. To get more details about it please check the following discussion:</p><p><a href="https://github.com/prometheus/prometheus/discussions/9428">Series with negative offset are continued with the last value up to 5min · Discussion #9428 · prometheus/prometheus</a></p><p>VictoriaMetrics isn&#39;t going to change the logic of negative offsets because this feature was released <a href="https://github.com/prometheus/prometheus/issues/6282#issuecomment-564301756">2 years before</a> Prometheus did it and users rely on that.</p><p>There were <strong>3 (~0.5% of 529 tests total)</strong> queries for -1m, -5m, -10m offsets in the test suite:</p><pre>QUERY: demo_memory_usage_bytes offset -1m<br>RESULT: FAILED: Query succeeded, but should have failed.</pre><h4>Precision loss</h4><p>VictoriaMetrics fails the following test case:</p><pre>QUERY: demo_memory_usage_bytes % 1.2345<br>  Timestamp: s&quot;1633073960&quot;,<br>- Value: Inverse(TranslateFloat64, float64(0.038788650870683394)),<br>+ Value: Inverse(TranslateFloat64, float64(0.038790081382158004)),</pre><p>The result is indeed different. It is off on the 5th digit after the decimal point and the reason for this is not in MetricsQL but in VictoriaMetrics itself. The query result isn&#39;t correct because the raw data point value for this specific metric doesn&#39;t match between Prometheus and VictoriaMetrics:</p><pre>curl <a href="http://localhost:9090/api/v1/query">http://localhost:9090/api/v1/query</a> --data-urlencode &#39;query=demo_memory_usage_bytes{instance=&quot;demo.promlabs.com:10000&quot;, type=&quot;buffers&quot;}&#39; --data-urlencode &#39;time=1633504838&#39; <br>...&quot;value&quot;:[1633504838,&quot;148164507.40843752&quot;]}]}}%                                                                                  </pre><pre>curl <a href="http://localhost:8428/api/v1/query">http://localhost:8428/api/v1/query</a> --data-urlencode &#39;query=demo_memory_usage_bytes{instance=&quot;demo.promlabs.com:10000&quot;, type=&quot;buffers&quot;}&#39; --data-urlencode &#39;time=1633504838&#39;<br>...&quot;value&quot;:[1633504838,&quot;148164507.4084375&quot;]}]}}%</pre><p>VictoriaMetrics may reduce the precision of values with more than 15 decimal digits due to the <a href="https://faun.pub/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932">used compression algorithm</a>. If you want to get more details about how and why this happens, please read the &quot;Precision loss&quot; section in <a href="https://medium.com/@valyala/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87">Evaluating Performance and Correctness</a>. In fact, any solution that works with floating point values has precision loss issues because of the nature of <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic">floating-point arithmetic</a>.</p><p>While such precision loss may be important in rare cases, it doesn&#39;t matter in most practical cases because the <a href="https://en.wikipedia.org/wiki/Observational_error">measurement error</a> is usually much larger than the precision loss.</p><p>While VictoriaMetrics does have higher precision loss than Prometheus, we believe it is completely justified by the <a href="https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f">compression gains</a> our solution generates. Moreover, only <strong>3 (~0.5% of 529 tests total)</strong> queries from the test suite fail due to precision loss.</p><h4>Query succeeded, but should have failed</h4><p>The following query fails for PromQL but works in MetricsQL:</p><pre>QUERY: {__name__=~&quot;.*&quot;}<br>RESULT: FAILED: Query succeeded, but should have failed.</pre><p>PromQL rejects such a query to prevent database overload because query <a href="https://github.com/prometheus/prometheus/issues/2162">selects all the metrics</a> from it. At the same time, PromQL does not prevent a user from running an almost identical query{__name__=~&quot;.+&quot;} , which serves the same purpose.</p><p>The other example of a failing query is the following:</p><pre>QUERY: label_replace(demo_num_cpus, &quot;~invalid&quot;, &quot;&quot;, &quot;src&quot;, &quot;(.*)&quot;)<br>RESULT: FAILED: Query succeeded, but should have failed.</pre><p>The query fails for PromQL because it doesn&#39;t allow using ~ char in label names. VictoriaMetrics accepts data ingestion from various protocols and systems where such char is allowed, so it <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/672#issuecomment-670189308">has to support</a> a wider list of allowed chars.</p><p>There were<strong> 2 (~0.3% of 529 tests total)</strong> queries that failed because of incompatibility but we can’t imagine a situation where it would harm a user’s experience.</p><h4>Summary</h4><p>There are differences between MetricsQL and PromQL. MetricsQL was created long after the PromQL with the goal of improving the user experience and making the language easier to use and understand.</p><p>How compatibility is measured in the <a href="https://prometheus.io/blog/2021/05/03/introducing-prometheus-conformance-program/">Prometheus Conformance Program</a> isn&#39;t ideal because it really only shows if the tested software uses Prometheus PromQL library under the hood or not. This is particularly complicated for solutions written in programming languages other than Go.</p><p>By the way, the percentage of failing tests is easy to increase or decrease by changing the number of range intervals (e.g. 1m, 5m etc.) in tests. In the case of VictoriaMetrics, about 90 tests have failed not because of wrong calculations, but because of the metric name present in the response. Of course, there is no ideal way to be fair to everyone. That&#39;s why this post exists to explain the differences.</p><p>We also want to say a big thank you to <a href="https://github.com/juliusv">Julius Volz</a>, the author of these <a href="https://promlabs.com/promql-compliance-tests/">compliance tests</a>. Thanks to his work and patience we were able to fix most of the real incompatibility issues in MetricsQL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4318203f51e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VictoriaMetrics: how to migrate data from Prometheus. Filtering and modifying time series.]]></title>
            <link>https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-filtering-and-modifying-time-series-6d40cea4bf21?source=rss-8c1c0cd81cd7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d40cea4bf21</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[victoriametrics]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Roman Khavronenko]]></dc:creator>
            <pubDate>Mon, 24 Aug 2020 21:24:17 GMT</pubDate>
            <atom:updated>2020-08-24T21:24:17.583Z</atom:updated>
            <content:encoded><![CDATA[<p>Previous story <a href="https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043">VictoriaMetrics: how to migrate data from Prometheus</a> explains how to use <a href="https://github.com/VictoriaMetrics/vmctl">vmctl</a> tool for migrating Prometheus snapshots into VM. But some details like time series filtering and other specifics remain uncovered. This story will give more details about relabeling, compression, filtering time series by time and labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yc3JBeuit9HtI9bew9x7Ng.jpeg" /><figcaption><a href="https://flic.kr/p/666muB">Photo</a> by <a href="https://www.flickr.com/photos/eflon/">eflon</a> / <a href="https://creativecommons.org/licenses/by/2.0/">Attribution 2.0 Generic (CC BY 2.0)</a></figcaption></figure><h3>Relabeling</h3><p>Vmctl itself does not provide labels management functionality because it is already supported by VM itself. User can define <a href="https://github.com/VictoriaMetrics/VictoriaMetrics#relabeling">relabeling rules</a> for all ingested metrics by specifying --relabelConfig flag. For example, to change metric up{instance=&quot;localhost:9090&quot;} to up{instance=&quot;localhost&quot;, port=&quot;9090&quot;} create relabel.yml file with following configuration:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6ce12465c3366af9cadf4d577008eac/href">https://medium.com/media/f6ce12465c3366af9cadf4d577008eac/href</a></iframe><p>And point VM to read the configuration file:</p><pre>./bin/victoria-metrics --relabelConfig relabel.yml</pre><p>According to the configuration, every ingested time series which satisfies the specified regular expression will be automatically updated with new label port. See it yourself by manually sending new time series via <a href="https://github.com/VictoriaMetrics/VictoriaMetrics#how-to-import-data-in-prometheus-exposition-format">Prometheus import format</a>:</p><pre>curl -d &#39;requests_total{instance=&quot;localhost:9090&quot;} 123&#39; -X POST &#39;<a href="http://localhost:8428/api/v1/import/prometheus&#39;">http://localhost:8428/api/v1/import/prometheus&#39;</a>                                        </pre><pre>curl -G &#39;<a href="http://localhost:8428/api/v1/export&#39;">http://localhost:8428/api/v1/export&#39;</a> -d &#39;match=requests_total&#39;</pre><pre>{&quot;metric&quot;:{&quot;__name__&quot;:&quot;requests_total&quot;,&quot;instance&quot;:&quot;localhost&quot;,&quot;port&quot;:&quot;9090&quot;},&quot;values&quot;:[123],&quot;timestamps&quot;:[1598089314604]}</pre><p>So if you need to relabel time series before migration or backfilling — just configure VM with according relabeling configuration.</p><h3>Filtering by time</h3><p>Vmctl uses <a href="http://github.com/prometheus/prometheus/tsdb">Prometheus TSDB package</a> for reading Prometheus snapshots. The package is well documented and simple to use. Besides reading time series from the snapshot it also allows to filter them by time or label values.</p><p>Filtering time series by time in vmctl is controlled by following two flags:</p><pre>--prom-filter-time-start - The time filter to select timeseries with timestamp equal or higher than provided value. E.g. &#39;2020-01-01T20:07:00Z&#39;<br>--prom-filter-time-end - The time filter to select timeseries with timestamp equal or lower than provided value. E.g. &#39;2020-01-01T20:07:00Z&#39;</pre><p>Let&#39;s try to filter time series from the snapshot in previous <a href="https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043">story</a> by time:</p><pre>./bin/vmctl prometheus --prom-snapshot=snapshots/20200806T072444Z-365a858149c6e2d1 --prom-filter-time-start=2020-08-02T00:00:00Z --prom-filter-time-end=2020-08-03T00:00:00Z<br>Prometheus import mode<br>Prometheus snapshot stats:<br>  blocks found: 7;<br>  blocks skipped by time filter: 5;<br>  min time: 1596349620000 (2020-08-02T07:27:00+01:00);<br>  max time: 1596607200000 (2020-08-05T07:00:00+01:00);<br>  samples: 168012658;<br>  series: 6943.<br>* Stats numbers are based on blocks meta info and don&#39;t account for applied filters.</pre><p>Note that the total amount of blocks in the provided snapshot is 7 but 5 of them are marked as skipped because of specified time filter. Every block contains a meta information with max and min timestamps in it, so vmctl can filter blocks that are out of a specified time range in advance without actually reading the data. Nevertheless, time filter will be applied again during the import time to filter out samples inside the blocks as well. Because of that, stats info from the message above doesn&#39;t show the exact number of samples and series to import.</p><h3>Filtering by labels</h3><p>To filter time series in the snapshot by labels vmctl provides two flags:</p><pre>--prom-filter-label - Prometheus label name to filter timeseries by. E.g. &#39;__name__&#39; will filter timeseries by name.<br>--prom-filter-label-value - Prometheus regular expression to filter label from &quot;prom-filter-label&quot; flag. (default: &quot;.*&quot;)</pre><p>The snapshot we use contains a set of time series generated by recording rules. The <a href="https://prometheus.io/docs/practices/rules/#naming-and-aggregation">convention</a> for rules naming implies separating aggregation level, metric and operation type via colon mark — level:metric:operation. Let&#39;s take advantage of that and migrate recording rules only:</p><pre>./bin/vmctl prometheus --prom-snapshot=snapshots/20200806T072444Z-365a858149c6e2d1 --prom-filter-label=&#39;__name__&#39; --prom-filter-label-value=&#39;.*:.*&#39;<br>Prometheus import mode<br>Prometheus snapshot stats:<br>  blocks found: 7;<br>  blocks skipped by time filter: 0;<br>  min time: 1596349620000 (2020-08-02T07:27:00+01:00);<br>  max time: 1596698684531 (2020-08-06T08:24:44+01:00);<br>  samples: 228164332;<br>  series: 24248.<br>* Stats numbers are based on blocks meta info and don&#39;t account for applied filters.<br>Found 7 blocks to import. Continue? [Y/n] y<br>7 / 7 [------------------------------------------] 100.00% 2 p/s<br>2020/08/23 11:08:25 Import finished!<br>2020/08/23 11:08:25 VictoriaMetrics importer stats:<br>  idle duration: 291.517073ms;<br>  time spent while importing: 3.105398727s;<br>  total samples: 6639476;<br>  samples/s: 2138043.00;<br>  total bytes: 172.6 MB;<br>  bytes/s: 55.6 MB;<br>  import requests: 32;<br>  import requests retries: 0;<br>2020/08/23 11:08:25 Total time: 4.403631893s</pre><p>The import process was quite fast and this is expected — only 6639476 samples out of 228164332 were imported and the rest was filtered out by specified label filter. So let&#39;s query imported series in Grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8n6ZoKGiGQS9_6ZY4TWUA.png" /></figure><p>Seems like everything is correct and only recording rules were imported into VM. The same approach may be used to import a subset of time series for a specific job or specific prefixes in the name.</p><h3><strong>Significant figures</strong></h3><p>In the previous example we imported only recording rules from the snapshot and number of imported samples was 6639476 — which is (6639476*100) / 228164332 = 2.9% of all samples in the snapshot. Let&#39;s check occupied disk space by VM for storing time series for recording rules only:</p><pre>du -d2 -h victoria-metrics-data<br>  0B victoria-metrics-data/snapshots<br>  0B victoria-metrics-data/indexdb/snapshots<br>4.0K victoria-metrics-data/indexdb/162DDEC63073CF09<br> 32K victoria-metrics-data/indexdb/162DDEC63073CF0A<br> 36K victoria-metrics-data/indexdb<br>  0B victoria-metrics-data/data/big<br>7.8M victoria-metrics-data/data/small<br>7.8M victoria-metrics-data/data<br>7.8M victoria-metrics-data</pre><p>The compression ratio for imported samples is 7.8MB / 6639476 = 1.2B per sample and it is not as good results as in the previous <a href="https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043">story</a> where we had 0.32B per sample. From those results we know, that the whole snapshot after being imported into VM occupies ~74MB of disk space. And now we see that 2.9% of samples occupy 7.8MB which is 10.5% of the snapshot. But why recording rules samples are so expensive? Let&#39;s check what is stored in one of the recording rules which we imported:</p><pre>curl -G &#39;<a href="http://localhost:8428/api/v1/export?start=1596458059&amp;end=1596458059&#39;">http://localhost:8428/api/v1/export?start=1596458059&amp;end=1596458059&#39;</a> -d &#39;match=instance:cpu_utilization:ratio_avg{job=&quot;node_exporter_1&quot;}&#39;<br>{&quot;metric&quot;:{&quot;__name__&quot;:&quot;instance:cpu_utilization:ratio_avg&quot;,&quot;job&quot;:&quot;node_exporter_1&quot;,&quot;instance&quot;:&quot;localhost:9100&quot;},&quot;values&quot;:[0.05055757575781,0.05058181818236,0.05067878787913,0.05083030303065,0.05109090909107,0.05139393939439,0.05166060606074,0.05195151515192,0.05221212121262,0.05255151515177,0.05289090909092,0.05315151515134,0.05050303030279,0.04776363636338,0.044975757576],&quot;timestamps&quot;:[1596458071478,1596458091478]}</pre><p>Most compression algorithms detect patterns and sequences in data in order to encode it using fewer bits than original representation. For example, cumulative counters could be compressed via <a href="https://en.wikipedia.org/wiki/Delta_encoding">delta encoding</a> because every next counter value is larger than previous. In our case, values like 0.05055757575781 are the result of <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic">floating point arithmetic</a> and <a href="https://en.wikipedia.org/wiki/Observational_error">measurement errors</a> like in the <a href="http://demo.robustperception.io:9090/graph?g0.range_input=1h&amp;g0.expr=(0.1%20%2B%200.1)%20%3E%20bool%20(0.3%20-%200.1)&amp;g0.tab=1">following example</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*LfVjeQeC8cG0mvYfP_IkEw.png" /></figure><p>Moreover, such values likely showing <a href="https://en.wikipedia.org/wiki/False_precision">false precision</a>, result into <a href="https://github.com/prometheus/prometheus/issues/4580">unexpected issues</a> and overconfidence in the <a href="http://demo.robustperception.io:9090/graph?g0.range_input=1h&amp;g0.expr=0.3%20-%200.1&amp;g0.tab=1">accuracy</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BEMYCN4h_yPNzOq6SKfVtg.png" /></figure><p>Such values heavily impact the compression ratio according to <a href="https://en.wikipedia.org/wiki/Information_theory">information theory</a> because of the need to store meaningless figures. And even more important, such values are hard for a human to interpret or even tell immediately which value 0.05055757575781 or 0.05058181818236 is bigger or smaller without spending seconds to compare digits one by one. If you ask your colleague or friend to pronounce one of the numbers above then with high probability value will be rounded to first 3 or 5 figures because the rest isn&#39;t really that important to mention.</p><p>In order to improve compression, vmctl allows to limit the number of <a href="https://en.wikipedia.org/wiki/Significant_figures">significant figures</a> before importing. For example, number of significant figures can be limited to 8 by specifying flag --vm-significant-figures=8. Then, value 0.05055757575781 will be rounded to 0.050557576 and when import process is finished — the same export query will return the following results:</p><pre>curl -G &#39;<a href="http://localhost:8428/api/v1/export?start=1596458059&amp;end=1596458059&#39;">http://localhost:8428/api/v1/export?start=1596458059&amp;end=1596458059&#39;</a> -d &#39;match=instance:cpu_utilization:ratio_avg{job=&quot;node_exporter_1&quot;}&#39;<br>{&quot;metric&quot;:{&quot;__name__&quot;:&quot;instance:cpu_utilization:ratio_avg&quot;,&quot;job&quot;:&quot;node_exporter_1&quot;,&quot;instance&quot;:&quot;localhost:9100&quot;},&quot;values&quot;:[0.050557576,0.050581818,0.050678788,0.050830303,0.051090909,0.051393939,0.051660606,0.051951515,0.052212121,0.052551515,0.052890909,0.053151515,0.05050303,0.047763636,0.044975758],&quot;timestamps&quot;:[1596458071478,1596458091478,1596458111478,1596458131478,1596458151478,1596458171478,1596458191478,1596458211478,1596458231478,1596458251478,1596458271478,1596458291478,1596458311478,1596458331478,1596458351478]}</pre><p>Limiting number of significant figures for recording rules samples improved compression ratio to 5.6MB / 6639476 = 0.84B per sample:</p><pre>du -d2 -h victoria-metrics-data                                                                      <br>  0B victoria-metrics-data/snapshots<br> 52K victoria-metrics-data/indexdb/162DF6472D579ACA<br>4.0K victoria-metrics-data/indexdb/162DF6472D579AC9<br>  0B victoria-metrics-data/indexdb/snapshots<br> 56K victoria-metrics-data/indexdb<br>  0B victoria-metrics-data/data/big<br>5.6M victoria-metrics-data/data/small<br>5.6M victoria-metrics-data/data<br>5.6M victoria-metrics-data</pre><p>Worth to mention, that there are real-world scenarios where high precision is very important, measured and calculated carefully to eliminate possible errors. From my personal experience, such cases are rare and often require some specific software to keep accuracy and precision on high level. But when you need to store just ordinary time series for CPU and memory usage — applying --vm-significant-figures is safe enough and will result into much higher compression ratio, will save some extra disk space after the migration and eliminate false precision. The most common case for using this flag is to reduce number of significant figures for time series created by aggregations such as average, rate, etc. Please note, that <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent">vmagent</a> also can be configured with this flag for all ingested or scraped time series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d40cea4bf21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VictoriaMetrics: how to migrate data from Prometheus]]></title>
            <link>https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-d44a6728f043?source=rss-8c1c0cd81cd7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d44a6728f043</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[victoriametrics]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Roman Khavronenko]]></dc:creator>
            <pubDate>Wed, 12 Aug 2020 17:16:21 GMT</pubDate>
            <atom:updated>2020-09-21T19:32:27.711Z</atom:updated>
            <content:encoded><![CDATA[<p>The most important thing in evaluating any new technology is the ability to compare to something you already know. That&#39;s why evaluating <a href="https://victoriametrics.com/">VictoriaMetrics</a> (hereinafter VM) as time series long-term storage is so easy for <a href="https://prometheus.io/">Prometheus</a> users. It supports the same scraping protocol, the same configuration format, similar query language <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/MetricsQL">MetricsQL</a>, and even more features to become a drop-in replacement for Prometheus. And historical data migration is an inevitable part of this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*szNXHtmoTs1Oinkk.jpg" /><figcaption><a href="https://www.flickr.com/photos/craigcloutier/3307011628/sizes/l/">Photo</a> by <a href="https://www.flickr.com/photos/craigcloutier/">Craig Cloutier</a> / <a href="https://creativecommons.org/licenses/by-sa/2.0/">CC BY-SA 2.0</a></figcaption></figure><p>Even though VM and Prometheus have a lot of common in terms of protocols and formats, the implementation is completely different. VM is a highly optimized <a href="https://en.wikipedia.org/wiki/Time_series_database">TSDB</a> written from scratch specifically to address performance issues while processing high volumes of time series data. It supports both &quot;pull&quot; and &quot;push&quot; protocols for data ingestion. So migrating data from Prometheus to VM is a process of converting and ingesting data via one of the supported protocols. For the data migrating process I&#39;ll use a simple tool called <a href="https://github.com/VictoriaMetrics/vmctl">vmctl</a> which does exactly this: reads Prometheus snapshot and ingests data into VM.</p><p><strong>Prometheus snapshot</strong></p><p>Unfortunately, I didn’t find any public Prometheus data that can be used as a reference for the migration process. So I had to create my own with the following config:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7eb8fc68fee549d09cd74284217da00e/href">https://medium.com/media/7eb8fc68fee549d09cd74284217da00e/href</a></iframe><p>The <a href="https://gist.github.com/hagen1778/20ba5af021db40569000608992fecb7d">rules.yml</a> file consists of <a href="https://github.com/prometheus/node_exporter">node exporter</a> specific recording rules borrowed from <a href="https://gitlab.com/gitlab-com/runbooks/-/blob/master/rules/node.yml">GitLab</a> and <a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/rules/rules.libsonnet">node exporter examples</a> to generate more time series.</p><p>Before starting migration we need to take a snapshot of Prometheus data. The concept and importance of making snapshots before doing any manipulations with data is well described <a href="https://www.robustperception.io/taking-snapshots-of-prometheus-data">here</a>. Prometheus was scraping data for few days and collected about 320MB of data:</p><pre>du -d1 -h snapshots<br>320M snapshots/20200806T072444Z-365a858149c6e2d1</pre><p>So let’s try to migrate data from the snapshot to VM via vmctl tool!</p><p><strong>Vmctl</strong></p><p><a href="https://github.com/VictoriaMetrics/vmctl">Vmctl</a> supports data migration from <a href="https://github.com/VictoriaMetrics/vmctl#migrating-data-from-influxdb">InfluxDB</a>, <a href="https://github.com/VictoriaMetrics/vmctl#migrating-data-from-prometheus">Prometheus</a> and <a href="https://github.com/VictoriaMetrics/vmctl#migrating-data-from-thanos">Thanos</a>. Actually, Thanos and Prometheus are using the same storage engine under the hood and as result, they have the same data layout on disk. Since snapshots are just hard links to the data folder — there is no difference for vmctl which snapshot it reads: Prometheus or Thanos snapshot.</p><p>To start the migration process we need to specify the following flags:</p><ul><li><em>— prom-snapshot</em> — file path to Prometheus snapshot;</li><li><em>— vm-addr</em> — address of running VictoriaMetrics service (localhost:8428 by default).</li></ul><p>I’ll be running migration on my 13&quot; mbp-pro where I have a single-node version of VM (v1.39.4), vmctl binary (v0.1.1) and copied Prometheus snapshot. VM is started with only one extra flag:</p><pre>./bin/victoria-metrics --selfScrapeInterval=1s</pre><p>Flag ` — selfScrapeInterval` makes VM to scrape its own metrics so we can analyze them later. Let’s run the vmctl:</p><pre>./bin/vmctl prometheus --prom-snapshot=snapshots/20200806T072444Z-365a858149c6e2d1<br>Prometheus import mode<br>Prometheus snapshot stats:<br>blocks found: 7;<br>blocks skipped: 0;<br>min time: 1596349620000 (2020-08-02T07:27:00+01:00);<br>max time: 1596698684531 (2020-08-06T08:24:44+01:00);<br>samples: 228164332;<br>series: 24248.<br>Filter is not taken into account for series and samples numbers.<br>Found 7 blocks to import. Continue? [Y/n] y</pre><p>As the first step, vmctl provides the snapshot stats and waits for user permission to proceed. We see that “few days” of Prometheus scraping is actually a 228164332 samples for 24248 series. The number of series was almost constant all the time since there was no churn in time series. Let’s proceed with answering Y:</p><pre>7 / 7 [--------------------------------------------] 100.00% 0 p/s<br>2020/08/11 20:06:29 Import finished!<br>2020/08/11 20:06:29 VictoriaMetrics importer stats:<br>  idle duration: 3.328381194s;<br>  time spent while importing: 1m26.482662623s;<br>  total samples: 228164332;<br>  samples/s: 2638266.74;<br>  total bytes: 4.5 GB;<br>  bytes/s: 52.3 MB;<br>  import requests: 1045;<br>  import requests retries: 0;<br>2020/08/11 20:06:29 Total time: 1m29.745175897s<br>./bin/vmctl prometheus   139.39s user 50.16s system 211% cpu 1:29.77 total</pre><p>As unit in progress bar, vmctl uses Prometheus <a href="https://prometheus.io/docs/prometheus/latest/storage/#on-disk-layout">data blocks</a>, so it may not be that responsive while migrating huge ones. It is so because time series filtering is done on-flight while reading data from the snapshot. And it is hard to say in advance how much data is left to import.</p><p>During migration vmctl was ingesting on average 2.6kk samples/s and the whole process took 1m29s to migrate 320MB snapshot. More details about printed stats and performance tuning may be found <a href="https://github.com/VictoriaMetrics/vmctl#importer-stats">here</a>. For now, when both vmctl and VM service are on the same machine and no network involved —I&#39;ll try to tweak it a bit by increasing importer concurrency:</p><pre>./bin/vmctl prometheus --prom-snapshot=snapshots/20200806T072444Z-365a858149c6e2d1 --vm-concurrency=8<br>...<br>2020/08/11 20:11:40 Import finished!<br>2020/08/11 20:11:40 VictoriaMetrics importer stats:<br>  idle duration: 7.245308365s;<br>  time spent while importing: 53.683616464s;<br>  total samples: 228164332;<br>  samples/s: 4250166.94;<br>  total bytes: 4.5 GB;<br>  bytes/s: 84.2 MB;<br>  import requests: 1046;<br>  import requests retries: 0;<br>2020/08/11 20:11:40 Total time: 55.485859442s<br>./bin/vmctl prometheus  --vm-concurrency=8  191.30s user 22.18s system 384% cpu 55.525 total</pre><p>Now it is 4.2kk samples per second with importer concurrency set to 8. To scale it further I&#39;d recommend to increase concurrency to number of available CPU cores for VM service. If with this setting VM does not utilize all available CPU cores during the import — try to increase --prom-concurrency or even spawn multiple vmctl processes on separate instances to avoid resource contention.</p><p>Let’s check the VM metrics during the import. I’ll use <a href="https://grafana.com/">Grafana</a> configured with <a href="https://prometheus.io/docs/visualization/grafana/">Prometheus datasource</a> pointed to <a href="https://github.com/VictoriaMetrics/VictoriaMetrics#grafana-setup">VM address</a> and the official Grafana <a href="https://grafana.com/grafana/dashboards/10229">dashboard for VM</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5as3PesYsSWzGO0H7bbuDQ.png" /><figcaption>VictoriaMetrics stats during import. Ingestion speed.</figcaption></figure><p>We see that all import requests were served via `/api/v1/import` endpoint. See more about ways how to import data into VM <a href="https://github.com/VictoriaMetrics/VictoriaMetrics#how-to-import-time-series-data">here</a>. Let’s check the disk usage now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uj55QeEaijO52yKrVt-zLA.png" /><figcaption>VictoriaMetrics stats during import. Disk usage.</figcaption></figure><p>So in total we see that the imported snapshot occupies about 73MB of disk space with avg 0.346 bytes per sample. Let’s verify this by checking VM data folder (set by flag &quot; — storageDataPath&quot;):</p><pre>du -d2 -h victoria-metrics-data<br>424K victoria-metrics-data/cache/rollupResult<br>1.9M victoria-metrics-data/cache/metricName_tsid<br>1.8M victoria-metrics-data/cache/metricID_metricName<br>524K victoria-metrics-data/cache/metricID_tsid<br>4.6M victoria-metrics-data/cache<br>  0B victoria-metrics-data/snapshots<br>  0B victoria-metrics-data/indexdb/snapshots<br>504K victoria-metrics-data/indexdb/162A4C7CA766B94A<br>4.0K victoria-metrics-data/indexdb/162A4C7CA766B949<br>508K victoria-metrics-data/indexdb<br>  0B victoria-metrics-data/data/big<br> 74M victoria-metrics-data/data/small<br> 74M victoria-metrics-data/data<br> 79M victoria-metrics-data</pre><p>The &quot;/cache&quot; folder contains various temporary caches for optimizing query speed which could be cleaned up at any time. It is not empty because some queries were already served for Grafana to plot the dashboard.</p><p>The &quot;/indexdb&quot; folder contains inverted index for all stored time series. The index is used for speeding up lookups while searching or ingesting data. The size of &quot;/indexdb&quot; depends on the number of unique time series stored in VM.</p><p>Folder &quot;/data&quot; contains the actual time series values. It is split in two subfolders: “/data/small” and “/data/big”. These are parts of VM internal structure <a href="https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282">MergeSet</a> and define the merging stage. While accepting and processing new data, VM merges previously saved data parts into bigger ones. The merging process significantly improves compression, reading speed and reduces the number of files on disk.</p><p>As we see, current compression ratio is quite good comparing to Prometheus:</p><ul><li>(73MB + 424KB) / 228164332 = 0.32B per sample in VM</li><li>320MB / 228164332 = 1.4B per sample in Prometheus</li></ul><p>Now let’s verify that imported data is actually accessible in VM. Since most of our scrape targets were node exporter jobs - let&#39;s install the node exporter <a href="https://grafana.com/grafana/dashboards/1860">dashboard</a> in Grafana:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UCXYb2Gp1Olw-3x1cE4Fnw.png" /><figcaption>Node exporter dashboard</figcaption></figure><p>We see that metrics from the snapshot are queryable and time range is exactly what we’ve seen in snapshot stats. Try it yourself and use vmctl to migrate historical data from Prometheus to VM. See also <a href="https://victoriametrics.github.io/#backfilling">backfilling</a> docs for more details and common gotchas.</p><p>See more about compression improvements and time series filtering via vmctl in the <a href="https://medium.com/@romanhavronenko/victoriametrics-how-to-migrate-data-from-prometheus-filtering-and-modifying-time-series-6d40cea4bf21">Filtering and modifying time series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d44a6728f043" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>