<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Charles-Antoine Mathieu on Medium]]></title>
        <description><![CDATA[Stories by Charles-Antoine Mathieu on Medium]]></description>
        <link>https://medium.com/@ca.mathieu?source=rss-32a576c324cf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KUeHECLq57VdwTfseHvFkg.png</url>
            <title>Stories by Charles-Antoine Mathieu on Medium</title>
            <link>https://medium.com/@ca.mathieu?source=rss-32a576c324cf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:46:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ca.mathieu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[VictoriaMetrics, a Prometheus Remote Storage solution]]></title>
            <link>https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61?source=rss-32a576c324cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/57081a3d8e61</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Charles-Antoine Mathieu]]></dc:creator>
            <pubDate>Tue, 19 Sep 2023 06:47:28 GMT</pubDate>
            <atom:updated>2023-09-19T19:57:04.835Z</atom:updated>
            <content:encoded><![CDATA[<h3>VictoriaMetrics, a stress-free Prometheus Remote Storage for 1 Billion metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MspZt0keQoXury0Z" /><figcaption>Photo by <a href="https://unsplash.com/@mcverry?utm_source=medium&amp;utm_medium=referral">Mikail McVerry</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Starting point</h3><p>Back in 2020, our homegrown centralized timeseries storage solution, <strong>BigGraphite</strong>, started to show the usual signs of aging. Could a more modern alternative be able to centralize our metrics for a fraction of the infrastructure cost and operational burden while being more reliable, performant and feature full ?</p><p>At that time, our scale involved 1500+ <strong>Prometheus </strong>instances across eight datacenters worldwide, for about about 1 billion active timeseries, with a resolution of one datapoint per minute with the goal to be able to ingest at least twice as much.</p><pre>Series : sum(max_over_time(prometheus_tsdb_head_series[30d])) / 1e6 =&gt; 1000<br>Churn : avg_over_time(sum(rate(prometheus_tsdb_head_series_created_total[5m]))[30d:]) =&gt; 3000/s</pre><h3>Contenders</h3><p>In the ever-evolving landscape of Observability, finding the perfect timeseries solution can be a daunting task. At that time, we surveyed the available open-source and those looked like the most viable options:</p><ol><li><a href="https://cortexmetrics.io/"><strong>Cortex</strong></a>: Horizontally scalable, highly available, multi-tenant, long term storage for Prometheus. (v1.4)</li><li><a href="https://victoriametrics.com/"><strong>VictoriaMetrics</strong></a>: Simple &amp; Reliable Monitoring That Scales (v1.44)</li><li><a href="https://thanos.io/"><strong>Thanos</strong></a>: Open source, highly available Prometheus setup with long term storage capabilities. (v0.16)</li><li><a href="https://www.timescale.com/"><strong>TimescaleDB</strong></a>: PostgreSQL++ for time series and events</li><li><a href="https://m3db.io/"><strong>M3DB</strong></a>: Open Source Metrics Engine (v0.15)</li><li><a href="https://www.warp10.io/"><strong>Warp10</strong></a>: The most advanced/flexible/efficient/awesome/scalable<br>Time Series platform</li><li><a href="https://www.influxdata.com/influxdb/"><strong>InfluxDB</strong></a>: It’s About Time</li><li><a href="http://opentsdb.net/"><strong>OpenTSDB</strong></a>: The Scalable Time Series Database</li></ol><p>We first considered their suitability for our technical ecosystem and requirements. After careful evaluation, we eliminated <strong>InfluxDB</strong>(OpenSource) and <strong>OpenTSDB </strong>for their ability to scale. <strong>Thanos </strong>was ruled out due to concerns about read latency, which we will discuss in more detail later in this post. <strong>Warp10 </strong>and <strong>TimescaleDB </strong>didn’t align so well with our existing tech stack. We also experimented with <strong>M3DB</strong>, but it was before 1.0 so it was found relatively immature at that time.</p><p>Consequently, we decided to focus our benchmarking efforts on <strong>Cortex</strong>, which emerged as the top contender, and <strong>VictoriaMetrics</strong>, which was “the dark horse” in our selection process.</p><h3>Performance Benchmark</h3><p>The performance of timeseries databases can vary significantly depending on factors such as dataset characteristics (cardinality, number of labels, etc.). To evaluate the candidate solutions, we conducted extensive performance benchmarking using a substantial amount of production metrics. To ensure accuracy and consistency, we used <strong>Thanos</strong> as a reference, to systematically cross-compare the results from the tested solutions with the original data in Prometheus.</p><p>As <strong>Thanos</strong> retrieves and aggregates data from Prometheus instances across the globe, the intercontinental round-trip time (RTT) imposed a lower limit of approximately 300ms on its response time. While this latency fell short of our desired performance for user-facing queries, we were pleased to find that the solution performed admirably overall.</p><p>To conduct our benchmarking, we developed a specialized tool and executed a diverse range of query patterns. These included instant queries versus range queries, various time ranges (e.g., last 5 minutes, last 1 hour, last 6 hours, last 1 day, etc.), random offsets in the past, different levels of metric cardinality, aggregated metrics (such as sum or top-k), and different levels of parallelism. These tests ran daily for several weeks, generating comprehensive data for analysis.</p><p>The benchmark was conducted around fall 2020 and the results consistently demonstrated a clear pattern: <strong>VictoriaMetrics </strong>exhibited faster response times and maintained a shorter and more stable tail latency. Executing the same query twice also always yielded similar completion times with <strong>VictoriaMetrics </strong>where <strong>Cortex </strong>response times displayed more entropy. Moreover, we achieved these favorable outcomes without needing to fine-tune any configuration parameters for <strong>VictoriaMetrics</strong>, whereas <strong>Cortex </strong>demanded significant efforts to optimize performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*9y3qYd35RDSwpyEQeUUziA.png" /></figure><figure><img alt="Impact of cardinality over response time" src="https://cdn-images-1.medium.com/max/728/1*vbS-gguIjgVWGxOp_QzI3A.png" /><figcaption>Impact of series cardinality on response time</figcaption></figure><p>It is important to note that <strong>the benchmarks were conducted quite some time ago</strong>, and the Observability landscape has undergone rapid evolution since then. Both the <strong>Cortex/Mimir</strong> and <strong>VictoriaMetrics </strong>teams have released numerous exciting improvements to their projects. <strong>Cortex </strong>transitioned from a Cassandra backend to S3, the <strong>Cortex </strong>and <strong>Thanos </strong>codebases were partially merged, and GrafanaLabs recently introduced <strong>Mimir about </strong>which we heard several success stories since then. Therefore, if you are seeking a timeseries database solution, we highly recommend performing your own due diligence and benchmarking the available options using a representative dataset.</p><h3>Other considerations</h3><p><strong>VictoriaMetrics</strong> architecture is simpler compared to <strong>Cortex/Mimir</strong>. As we were already fatigued by the complexity of <strong>BigGraphite</strong>, its streamlined design was a significant advantage for us. Its simplicity translated to easy setup, maintenance, and troubleshooting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*ZOIXoNdSSirbg8xE6DU1HA.png" /><figcaption>VictoriaMetrics architecture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X0_U5ionuYx4Zvrvq1Hl2Q.png" /><figcaption>Cortex architecture</figcaption></figure><h3>Our deployment</h3><p>When building a scalable timeseries solution, one crucial consideration is the cost involved. <strong>VictoriaMetrics</strong>’ impressive compression performance enables us to pack a significant number of datapoints in minimal disk space, averaging around <strong>1 byte per datapoint</strong>.</p><p>A common approach to reducing the cost of timeseries storage is downsampling, which involves aggregating larger intervals of data (such as an hour or a day) into a single datapoint, typically computed as an average. While downsampling is a highly effective technique utilized by <strong>Graphite</strong>, it does result in smoothed-out graphs due to the averaging process and graphs can look different depending on the time window you are looking at. Maintaining a distribution for each interval would mitigate these issues, but it adds complexity and significantly increases the number of timeseries to index so we decided not to go in that direction.</p><p>To be able to display graphs without having to query tens or hundreds of thousands of timeseries each time our <strong>Prometheus </strong>instances already perform aggregations per <em>datacenter, application and pool</em> (prod, canary, dev,…). That has the benefit of reducing cardinality by the number of instances. Instance metrics are really useful only as long as the instance lives and a bit after for troubleshooting, that’s why we decided to keep those metrics only for 7 days. Aggregated metrics on the other hand are relevant at the service level and keeping them for 90 days was deemed enough for almost all use cases of live monitoring. Lastly, for some use cases like SLA/SLO/SLI and long term trends it could however be useful to have a longer retention so we setup an extra cluster with a retention of 1+ year. For all other use cases, especially Business Intelligence, Criteo possesses a best-in-class data analytics infrastructure that will be happy to ingest and process any volume of data with no cardinality limits.</p><p>By keeping low value, high cardinality and high churn metrics only for a shorter period we were able to cut the costs of the infrastructure by more than an order of magnitude.</p><p>Regarding high availability (HA), our <strong>VictoriaMetrics </strong>cluster is resilient to a single node failure due to its replication factor of 2. This means that the ingested data is always written to two <em>vmstorage </em>nodes, in the event of a node failure, the data is automatically rerouted to another node, ensuring data integrity and continuity. To safeguard against a datacenter failure or isolation and to be able to drain a cluster for maintenance purposes we have simply implemented two identical infrastructures in two different regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I7L3OxE5Sj7ltGxz.png" /><figcaption>Our multi retention cluster architecture</figcaption></figure><h3>Our current scale</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lrecflAK1edvSCoyvoLdgw.png" /></figure><h3>Write path</h3><p>We deployed a few <em>vmagent</em> nodes per datacenter, those are responsible for collecting metrics from that datacenter and forwarding them to the <strong>VictoriaMetrics </strong>clusters abroad. This approach offers several advantages:</p><ul><li>Having only one remote write per <strong>Prometheus </strong>instance simplifies the remote write queue tuning process and enhances resilience, as any failures related to the remote write storage have a limited impact on Prometheus.</li><li>On-disk spooling. In scenarios where the connection to the remote datacenter becomes temporarily unavailable, the <em>vmagent </em>nodes are able to buffer data on disk until connectivity is restored, thus preventing data loss.</li><li><a href="https://victoriametrics.com/blog/victoriametrics-remote-write/">Save network costs with VictoriaMetrics remote write protocol</a></li></ul><p>The good thing about having one cluster per retention window is that it is easy to scale and tune each according to their ingestion rate. It is for example possible to rely on different node flavors to optimize costs/performance ratios like in a <a href="https://medium.com/bestcloudforme/data-aging-with-elasticsearch-hot-warm-cold-infrastructure-6e5eef4c9e65"><strong>hot-warm-cold Elasticsearch architecture</strong></a>.</p><p>Then it’s just one pair of <em>vminsert</em> per cluster with the default configuration. If a <em>vmstorage </em>is down the <em>vminsert</em> simply redirect the write to another node</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*lhsb3MvsmZYUZruS.gif" /></figure><h3>Read path</h3><p>As much as we like some features of MetricsQL (like the better handling of rate extrapolation, or <a href="https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/expand-with-exprs?_gl=1*1ime9rb*_ga*MTI5Mzg5NjIxMS4xNjc4NzAyNzg5*_ga_N9SVT8S3HK*MTY4NDY5MTQ4OS44LjAuMTY4NDY5MTQ4OS4wLjAuMA.."><strong>WITH templates</strong></a>) we wanted to avoid vendor locking. Our user base is wide, basically everybody in the Criteo R&amp;D has to deal with metrics at some point, they already have to know <strong>Graphite </strong>and PromQL so we wanted to avoid them having to worry about subtilities like when you can or cannot use MetricsQL. To do so we forked vmauth and added a PromQL check to ensure only PromQL queries are being evaluated. Additionally, we implemented a replay mechanism that duplicates queries to our backup cluster, allowing us to perform proper load testing after maintenance operations, such as version upgrades.</p><p><em>vmselect </em>nodes<em> </em>are able to target <em>vmstorage </em>nodes from all clusters at once and that’s basically all we needed to unify the three layers of retention.</p><p>We kept all configuration parameters to their default values for <em>vmselect </em>nodes, they are able to run queries over a maximum of 300k timeseries whereas BigGraphite has a limit at 7k which is a pretty big quality of life improvement for troubleshooting big applications.</p><h3>Are we satisfied?</h3><p>Absolutely! In terms of stability and reliability, SLAs speak for themselves, we proudly display &gt;0.998% availability over a year so we do sleep really well</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*d4YhtTl_LsgKu8dkyaVkiA.png" /></figure><p>In terms of performance, VictoriaMetrics has exceeded our expectations. It effortlessly handles the ingestion of 2 million datapoints per second per host without flinching. Even when fetching 300,000 timeseries for troubleshooting queries, the response remains impressively fast. The Grafana Metrics browser is highly responsive, enabling smooth exploration of available metrics. Furthermore, the VictoriaMetrics team has always been highly responsive and frequently releases updates, continuously pushing the boundaries of what the system can achieve.</p><p>Did we suffer from the lesser <a href="https://promlabs.com/blog/2021/10/14/promql-vendor-compatibility-round-three/"><strong>compatibility with PromQL</strong></a> ? No, to be honest, if you only run PromQL on it it makes little to no difference and when it did we found that it was for the better most often than not.</p><p>Last but not least, in terms of costs we are able to store about 15 times more datapoints for about four times less infra costs with better SLAs than with <strong>BigGraphite</strong>.</p><p><strong>VictoriaMetrics </strong>: 15 computes + 46 storage for 20T datapoints<br><strong>BigGraphite </strong>: 226 Computes + 156 storage for 1.3T datapoints</p><h3>What could be improved?</h3><p>No solution is perfect so here what’s we think is perfectible:</p><ol><li>Currently, replacing a <em>vmstorage</em> node is a manual process that involves data copying as VictoriaMetrics does not support cluster rebalancing. However, our storage servers have a very low failure rate so for now we had to perform this action only once.</li><li>VictoriaMetrics is able to ingest data in the past but it <a href="https://docs.victoriametrics.com/#backfilling">requires to clear the <em>vmselect </em>cache to be sure that the newly ingested data is available</a></li><li>Our development teams, responsible for libraries, SDKs, and frameworks, would greatly appreciate the ability to query data from all tenants without having to federate and duplicate metrics in their respective tenants. This functionality could potentially be achieved through the use of tools like <a href="https://github.com/jacksontj/promxy">GitHub — jacksontj/promxy: An aggregating proxy to enable HA prometheus</a>, which has recently shown signs of renewed activity.</li></ol><p><a href="https://bit.ly/eng-careers-medium">Engineering</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57081a3d8e61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/criteo-engineering/victoriametrics-a-prometheus-remote-storage-solution-57081a3d8e61">VictoriaMetrics, a Prometheus Remote Storage solution</a> was originally published in <a href="https://medium.com/criteo-engineering">Criteo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>