<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manikbindlish on Medium]]></title>
        <description><![CDATA[Stories by Manikbindlish on Medium]]></description>
        <link>https://medium.com/@manikbindlish19?source=rss-d24ea2843a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Azc92uqXvZD-y1xR</url>
            <title>Stories by Manikbindlish on Medium</title>
            <link>https://medium.com/@manikbindlish19?source=rss-d24ea2843a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:40:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manikbindlish19/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Influxdb as HA solution on freeware version:]]></title>
            <link>https://medium.com/@manikbindlish19/influxdb-as-ha-solution-on-freeware-version-597520d4291b?source=rss-d24ea2843a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/597520d4291b</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[time-series-database]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[influxdb]]></category>
            <dc:creator><![CDATA[Manikbindlish]]></dc:creator>
            <pubDate>Thu, 28 Apr 2022 15:28:17 GMT</pubDate>
            <atom:updated>2022-04-28T15:28:17.793Z</atom:updated>
            <content:encoded><![CDATA[<p>There are various database tools are available in market. Some of them are opensource and some of them are licenseware. Ofcourse, licenseware tools have special features on top of freeware.<br>As per requirement of special features, these can be used by projects to opt for freeware and licenseware. We will discuss here for influxdb.</p><p>InfluxDB is an open-source time series database (TSDB) developed by the company InfluxData. It is written in the Go programming language for storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics</p><p>Influxdb is faster than other comparative tools.</p><p><strong>InfluxDB outperformed Cassandra in all three tests:</strong><br>• 4.5x greater write throughput<br>• 2.1x less disk space<br>• delivering up to 45x faster response times for tested queries.</p><p><strong>Some general points about Influxdb:</strong><br>• Free of external dependencies, yet open and flexible enough for complex deployments. Get real value faster.<br>• Performance:can write ~1.4 million values/s<br>• Flexible queries<br>• Handling large data volumes<br>• Closed-source clustering</p><p><strong>Problem Case:</strong><br>Influxdb is much faster and good to use but problem arise when we have to use its freeware version in production. There is no High Availability supported in this. So either we have to move for licenseware or have to compromise with HA. I am sure, for production, HA cannot be compromised</p><p><strong>Solution:</strong><br>For above problem, there is a solution which we can do with below architecture(in diagram):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/0*SlpMqiyiyfF1iwPc.png" /><figcaption>Influx as HA solution</figcaption></figure><p>Architecture for Influx as HA</p><p>We can achieve Infuxdb as HA solution on freeware as well by enhancing it’s usage with other services.</p><p><strong>Environment used:</strong><br>We have deployed influxdb and other services in kubernetes cluster.<br>So we have created it’s pods(under statefulset) and created headless service for influxdb database.</p><p><strong>Services Used:</strong><br>We have used below components, to use with influxdb. You can use as per your requirement.</p><p>1.) API: This is producer of data, which will write data or messages in kafka’s topic<br>2.) Kafka(Messaging queue): Here it will keep data in topics(under partitions)<br>3.) Influxdb pods: Influxdb will get data from kafka and other services can query on influxdb, to retrieve data.</p><p><strong>How to achieve:</strong><br>1.) API is just writing it’s data in kafka topics.<br>2.) In kafka we can create multiple consumer groups. In above diagram we have created 3 consumer groups: CG1, CG2 and CG3<br>3.) Each consumer group will get the data from kafka topic(topic name should be same for each consumer group). This data will get duplicate in each consumer group. So CG1, CG2 and CG3, now have duplicated data.<br>4.) Each consumer group is connected with individual Influx pods. Like (CG1 -&gt; Influx1, CG2 -&gt; Influx2 , CG3 -&gt; Influx3 ).<br>5.) Each influx will get duplicate data from consumer groups(influx1 from CG1, influx2 from CG2, influx3 from CG3)<br>6.) A headless service is created for each influxdb pods.<br>7.) Once u user from CLI, API, Grafana, GUI, etc will query to influxdb. In that case, data can be fetch from either of influxdb pods.</p><p>There are some scenarios for failure of resources:<br>Above architecture is designed in such a way, if any of resource is down. There should be no impact on service continuity which is main objective of HA.</p><p><strong>Design:</strong><br>Considering 3 worker nodes in k8s cluster. It can be change as per requirement.<br>1.) There is a topic in kafka.<br>2.) We have created 3 consumer groups(in kafka), to align with 3 different influxdb pods(each on individual worker node)<br>3.) For each consumer group, we have created 3 consumers. So there will be total 9 consumers(three in one consumer group. E.g: CG1(C1+C2+C3), CG2(C1+C2+C3), CG3(C1+C2+C3))<br>4.) Each consumer in a consumer group, divide the data from partitions of a topic. So C1, C2 and C3 will divide the data in CG1 and same process for CG2, CG3.</p><p><strong>Scenario1: Influxdb pod is down</strong><br>1.) Three Influxdb pods are created on three different worker nodes of k8s cluster.<br>2.) All three pods are having duplicate data.<br>3.) If any 2 of influxdb pods get down/fail, still a user can query to retrieve data from influxdb via it’s service<br>4.) Writing on this influxdb pod will also continue to work.</p><p><strong>Scenario2: Worker Node is down</strong><br>1.) Three Influxdb pods are created on three different worker nodes of k8s cluster.<br>2.) All three pods are having duplicate data.<br>3.) If any 2 of worker nodes get down/fail, still a user can query to retrieve data from influxdb via it’s service. Third node is available with influxdb pod.<br>4.) Writing on this influxdb pod will also continue to work.</p><p><strong>Scenario3: Kafka consumer is down</strong><br>If a consumer from any consumer group is down.<br>e.g. Consumer C1 is down under Consumer Group CG1<br>1.) Consumers of each Consumer group are created on each worker node. For our case. We have C1(from CG1, CG2 and CG3) on node1, C2(from CG1, CG2 and CG3) on node2 and C3(from CG1, CG2 and CG3) on node3<br>2.) Consumers in each consumer group fetch the data from topic’s partition and divide among themselves.<br>3.) Suppose there are 12 partitions. Each consumer will get 4 partitions under one consumer group.<br>4.) If any consumer gets down. Then other will take care of that. It means C2 and C3, deployed on other nodes will get 6 partitions each. So no impact on data.<br>5.) Once C1, will be restored, it will start consuming the pending data.</p><p><strong>Scenario4: Kafka consumer group is down</strong><br>e.g. Consumer CG1 is down.<br>1.) Consumers of each Consumer group are created on each worker node. For our case. We have C1(from CG1, CG2 and CG3) on node1, C2(from CG1, CG2 and CG3) on node2 and C3(from CG1, CG2 and CG3) on node3<br>2.) CG1 is aligned with influx1.<br>3.) Consumers of CG1 are deployed on each worker node. CG1 will be down, only if all worker nodes are down or all consumers are down.<br>4.) CG1 or any consumer group will work fine, if any of consumer is alive under that group.</p><p><strong>Conclusion:</strong><br>With above approach, it is easy to use influxdb as HA solution.<br>It can be use for production.<br>There is multi-layer of High availability on influx and Kafka as sell. So it provides complete service continuity.</p><p><strong>Abbreviation:</strong><br>CG: Consumer Group<br>C: Consumer</p><p>originally published at <a href="https://plazza.orange.com/people/manik.bindlish@orange.com/blog/2022/04/22/influxdb-as-ha-solution-on-freeware-version">https://plazza.orange.com/people/manik.bindlish@orange.com/blog/2022/04/22/influxdb-as-ha-solution-on-freeware-version</a>on 22 April 2022</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=597520d4291b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>