<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Utku on Medium]]></title>
        <description><![CDATA[Stories by Utku on Medium]]></description>
        <link>https://medium.com/@utkumetinn?source=rss-99ec837f52e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wZYHawmhEPk65LQ8khZ7og.jpeg</url>
            <title>Stories by Utku on Medium</title>
            <link>https://medium.com/@utkumetinn?source=rss-99ec837f52e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:36:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@utkumetinn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Resilient Redis Clients with Lettuce]]></title>
            <link>https://engineering.getmidas.com/building-resilient-redis-clients-with-lettuce-c81f2e799dfa?source=rss-99ec837f52e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c81f2e799dfa</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Utku]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 09:16:22 GMT</pubDate>
            <atom:updated>2026-04-01T18:35:42.987Z</atom:updated>
            <content:encoded><![CDATA[<h4>Practical Configuration Patterns for Cluster Failover</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mw7SQuyV7oV0tJY5mVgX0w.png" /></figure><p>Redis is often treated as a fast, reliable building block in modern systems, but the client configuration around it matters just as much as the server side. In clustered and dynamic environments, especially in cloud-managed setups, failures are rarely clean. Nodes can disappear, DNS records can change, topology information can become stale, and reconnect storms can amplify an already bad situation. A Redis client that works perfectly in happy-path scenarios may become a source of thread exhaustion, cascading timeouts, and noisy operational failures during failover.</p><p>This is where resilient client configuration patterns become important. Lettuce already provides the primitives needed to handle these situations, but the defaults are usually too generic for production systems that care about availability and fast recovery. The goal is not to eliminate failure, but to reduce the blast radius when it happens: refresh topology quickly, avoid hanging on dead nodes, keep timeouts bounded, and reconnect in a controlled way.</p><p>This post walks through a practical Redis resilience setup using Lettuce. The focus is on a few key areas: cluster topology refresh, socket-level protections, command timeout strategies, node filtering, DNS resolution, and reconnect backoff. Together, these patterns help applications behave more predictably during cluster failover and infrastructure churn.</p><h3>Why Redis client resilience matters</h3><p>In many services, Redis calls sit directly on the request path. That means a single stuck call does not only affect cache latency; it can also block application threads, increase queueing, and degrade unrelated requests. During a failover, this becomes even more visible. The client may still believe an old primary is healthy, DNS may still point to an outdated address, or reconnect loops may hammer an already unstable cluster.</p><p>A resilient Redis client should therefore aim to do four things well:</p><ul><li>It should discover topology changes quickly</li><li>Stop talking to unhealthy nodes as soon as possible</li><li>Fail fast instead of blocking threads for long periods</li><li>Reconnect gradually rather than all at once</li></ul><p>The following sections show how to achieve that with Lettuce.</p><h3>1. Refresh cluster topology aggressively and safely</h3><p>The first layer of resilience is making sure the client knows what the cluster actually looks like.</p><pre>ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()<br>    .enableAllAdaptiveRefreshTriggers()<br>    .enablePeriodicRefresh(Duration.ofSeconds(30))<br>    .dynamicRefreshSources(true)<br>    .build();</pre><h4>Periodic refresh</h4><p>Periodic refresh allows Lettuce to regularly update its view of the cluster, including which nodes are available, which hash slots they own, and how requests should be routed. It serves as a baseline safeguard against stale topology information.</p><p>This mechanism helps the client recover from outdated cluster state even when no explicit failure is detected during normal command execution. Instead of relying only on errors or redirects, Lettuce can refresh its topology on a fixed schedule and keep its routing information reasonably up to date.</p><p>That said, topology refresh can become expensive in large clusters because it may require querying multiple nodes to build an updated cluster view. Using an overly short refresh interval can introduce extra overhead without providing meaningful benefits. In this case, a lower interval does not always mean a better configuration.</p><h4>Adaptive refresh triggers</h4><p>Periodic refresh alone may not be enough during an active failover or slot movement. If the cluster topology changes unexpectedly, waiting for the next scheduled refresh can still lead to a burst of routing errors in the meantime. Adaptive refresh triggers address this by forcing an immediate topology refresh when Lettuce observes signals such as MOVED redirects or repeated reconnect failures.</p><p>This gives the client a reactive recovery path during incidents, rather than relying only on scheduled background refreshes. As a result, the client can respond faster to topology changes and reduce the duration of stale routing behavior.</p><p>However, enabling adaptive refresh can also increase the number of topology refresh operations during unstable periods. Since these events may occur frequently and across many connections, Lettuce applies rate limiting through a timeout mechanism. This prevents refresh storms while still allowing the client to react quickly when the cluster state changes.</p><h4>Dynamic refresh sources</h4><p>This setting is particularly important in real-world cluster environments. When enabled, Lettuce uses all discovered nodes as possible sources for topology refreshes instead of relying only on the nodes defined in the initial cluster configuration.</p><p>This makes refresh behavior more resilient. If the original seed node becomes unavailable or unreachable, the client can still refresh its topology from other known nodes. Without this, a failed or unreachable seed node may cause refresh attempts to fail completely, leaving the client with stale topology information and increasing the likelihood of routing errors or exceptions.</p><p>However, this resilience comes with a cost. When dynamic refresh sources is enabled, Lettuce queries multiple nodes for their view of the cluster and compares the results to build an updated topology. In clusters with many nodes, this can become expensive and add unnecessary overhead. For that reason, disabling this feature is often the better choice in very large clusters.</p><p>In practice, dynamic refresh sources are most effective when combined with node filtering, which we will cover later.</p><h3>2. Use socket options to detect bad connections early</h3><p>Topology awareness is only one side of the problem. The other side is how the client behaves at the network layer when nodes become slow, unreachable, or half-dead.</p><pre>SocketOptions socketOptions = SocketOptions.builder()<br>    .connectTimeout(Duration.ofMillis(500))<br>    .keepAlive(SocketOptions.KeepAliveOptions.builder()<br>            .enable()<br>            .idle(Duration.ofSeconds(15))<br>            .interval(Duration.ofSeconds(5))<br>            .count(3)<br>            .build())<br>    .tcpUserTimeout(SocketOptions.TcpUserTimeoutOptions.builder()<br>            .enable()<br>            .tcpUserTimeout(Duration.ofSeconds(30))<br>            .build())<br>    .build();</pre><h4>Connect timeout</h4><p>Connect timeout defines how long the client will wait while establishing a TCP connection to a Redis node. In most cases, this value should be lower than the Redis command timeout.</p><p>This matters because after a topology refresh, the client may try to connect to a node that is unreachable or unhealthy. If the connect timeout is too high, the connection attempt itself can consume a large portion of the total request time budget before the command even has a chance to run.</p><p>Using a shorter connect timeout allows the client to fail fast and move on more quickly when a node cannot accept connections. This helps reduce recovery time and limits the impact of unhealthy nodes on overall request latency.</p><h4>Keep-alive</h4><p>TCP keep-alive helps detect dead idle connections. When a connection remains idle for a certain period, the operating system sends periodic probes to verify that the peer is still reachable. If the peer does not respond, the socket is eventually closed.</p><p>This is useful for cleaning up broken idle connections, but it is important to understand its limits. Keep-alive only applies when the connection is idle. If the application is actively sending commands, the connection is no longer considered idle, so keep-alive probes are not sent.</p><p>Because of that, keep-alive should not be treated as a complete solution for stuck or unhealthy active traffic. It helps with idle connection detection, but it does not protect against every kind of connection issue.</p><h4>TCP user timeout</h4><p>This is where TCP user timeout becomes especially valuable. It defines how long transmitted data is allowed to remain unacknowledged before the operating system forcefully closes the connection.</p><p>Unlike keep-alive, this option applies to active data transmission. If the client sends packets but acknowledgements never arrive because of network issues, hardware problems, or an unresponsive peer, the connection does not remain open indefinitely. Once the configured timeout is reached, the OS closes the socket.</p><p>This makes TCP user timeout one of the most effective protections against failures where the connection appears to be open, but no forward progress is actually happening.</p><p>It is also important to keep TCP user timeout aligned with keep-alive settings. If the keep-alive threshold is set higher, TCP user timeout may expire first and close the connection before keep-alive has a chance to detect the issue. In practice, that means TCP user timeout can effectively take over failure detection for active traffic. This is not necessarily wrong, but it should be configured intentionally so connection behavior remains predictable.</p><p>To enable these TCP-level options on Linux, include Netty’s native epoll transport:</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.netty&lt;/groupId&gt;<br>    &lt;artifactId&gt;netty-transport-native-epoll&lt;/artifactId&gt;<br>    &lt;version&gt;${netty.version}&lt;/version&gt;<br>    &lt;classifier&gt;${classifier}&lt;/classifier&gt;<br>&lt;/dependency&gt;</pre><h3>3. Keep Redis command timeouts intentionally small</h3><p>A resilient network setup still is not enough if individual Redis operations are allowed to hang for too long.</p><pre>TimeoutOptions timeoutOptions = TimeoutOptions.builder()<br>    .timeoutSource(new LettuceTimeoutOptions())<br>    .build();</pre><p>Lettuce allows you to define a custom timeout source for Redis commands. This is useful because not every command should be handled with the same timeout policy.</p><p>In many web applications, leaving Lettuce’s default command timeout of 60 seconds unchanged can be risky. A single stuck Redis call may block a Tomcat request thread for up to a full minute. During a failover or network issue, multiple blocked requests can accumulate quickly, exhaust the thread pool, and turn a Redis problem into a broader application outage.</p><p>For that reason, command timeouts should be configured explicitly based on how Redis is used in the system. In most cases, the implementation does not need to be complex. A simple and intentional timeout strategy is usually enough to reduce the blast radius of Redis-related incidents.</p><pre>public class LettuceTimeoutOptions extends TimeoutOptions.TimeoutSource {<br>    private static final Duration DEFAULT_COMMAND_TIMEOUT = Duration.ofMillis(1000);<br>    private static final Duration DEFAULT_META_COMMAND_TIMEOUT = Duration.ofMillis(3000);<br>    private static final Set&lt;ProtocolKeyword&gt; META_COMMAND_TYPES = Set.of(<br>            CommandType.CLUSTER,<br>            CommandType.KEYS,<br>            CommandType.INFO,<br>            CommandType.CLIENT,<br>            CommandType.CONFIG<br>    );<br><br>    private static final Map&lt;ProtocolKeyword, Duration&gt; COMMAND_TIMEOUTS = Map.of(<br>            CommandType.GET, DEFAULT_COMMAND_TIMEOUT,<br>            CommandType.SET, DEFAULT_COMMAND_TIMEOUT<br>    );<br><br>    @Override<br>    public long getTimeout(final RedisCommand&lt;?, ?, ?&gt; command) {<br>        ProtocolKeyword commandType = command.getType();<br>        return META_COMMAND_TYPES.contains(commandType)<br>                ? DEFAULT_META_COMMAND_TIMEOUT.toMillis()<br>                : COMMAND_TIMEOUTS.getOrDefault(commandType, DEFAULT_COMMAND_TIMEOUT).toMillis();<br>    }<br>}</pre><p>Here, regular commands such as GET and SET are capped at 1 second, while cluster or metadata-related commands are given a slightly larger budget.</p><p>The exact timeout values will differ from one system to another. A latency-sensitive API may prefer even lower limits, while background or batch workloads may be able to tolerate longer ones. What matters most is setting these boundaries deliberately, rather than relying on a default timeout that is too high for the request path.</p><h3>4. Filter unhealthy nodes during failover</h3><p>Cluster failover is not always instantaneous or clean from the client’s perspective. A failed node may still appear in the topology for a short time while the cluster converges.</p><pre>ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()<br>    .topologyRefreshOptions(topologyRefreshOptions)<br>    .socketOptions(socketOptions)<br>    .timeoutOptions(timeoutOptions)<br>    .nodeFilter(this::isHealthyNode)<br>    .validateClusterNodeMembership(false)<br>    .build();</pre><p>The key setting here is the node filter:</p><pre>private boolean isHealthyNode(RedisClusterNode node) {<br>    return !(node.is(RedisClusterNode.NodeFlag.FAIL)<br>            || node.is(RedisClusterNode.NodeFlag.EVENTUAL_FAIL)<br>            || node.is(RedisClusterNode.NodeFlag.HANDSHAKE)<br>            || node.is(RedisClusterNode.NodeFlag.NOADDR));<br>}</pre><p>This prevents Lettuce from trying to use nodes that are explicitly flagged as failed, in handshake state, or missing an address. Without this, the client may continue attempting connections to nodes that are technically still visible in topology output but are not suitable for traffic.</p><p>Node filtering becomes even more effective when paired with dynamic refresh sources. One helps the client discover more refresh paths, while the other reduces the chance of selecting obviously unhealthy nodes among those paths.</p><p>The validate cluster node membership option is also pragmatic in dynamic environments. Strict membership validation can become a source of friction during transitional failover states, especially in managed Redis environments.</p><h3>5. Fix DNS behavior for dynamic infrastructure</h3><p>In cloud setups, topology correctness is not only about Redis nodes and slots. DNS resolution also matters.</p><pre>ClientResources clientResources = DefaultClientResources.builder()<br>    .addressResolverGroup(getDnsAddressResolverGroup())<br>    .reconnectDelay(Delay.fullJitter(<br>            Duration.ofMillis(100),<br>            Duration.ofSeconds(10),<br>            100, TimeUnit.MILLISECONDS<br>    ))<br>    .build();</pre><h4>Address resolver group</h4><p>By default, DNS resolution behavior may not be ideal for dynamic infrastructures. Some resolvers cache results longer than you want, which becomes problematic when DNS records are updated during failover. The client can continue connecting to stale IP addresses even though the Redis endpoint name now points somewhere else.</p><p>Using Netty’s DNS resolver with disabled caches avoids that problem and also keeps DNS resolution non-blocking:</p><pre>private final EventLoopGroup eventLoopGroup = new NioEventLoopGroup();<br><br>private DnsAddressResolverGroup getDnsAddressResolverGroup() {<br>    return new DnsAddressResolverGroup(<br>            new DnsNameResolverBuilder(eventLoopGroup.next())<br>                    .datagramChannelType(NioDatagramChannel.class)<br>                    .resolveCache(NoopDnsCache.INSTANCE)<br>                    .cnameCache(NoopDnsCnameCache.INSTANCE)<br>                    .authoritativeDnsServerCache(NoopAuthoritativeDnsServerCache.INSTANCE)<br>                    .consolidateCacheSize(0)<br>    );<br>}</pre><p>These settings disable Netty’s own DNS caches, so the client is less likely to reuse stale addresses after a failover and more likely to pick up DNS changes quickly.</p><p>They also move DNS resolution into Netty’s asynchronous IO model. Instead of blocking an application thread while waiting for a lookup to finish, resolution is handled asynchronously by event loop threads.</p><p>This does not make the entire system cache-free. DNS behavior can still be influenced by other layers such as the OS, Kubernetes, or upstream resolvers. Still, disabling the client’s internal caches removes one common source of stale address reuse and makes failover behavior more predictable.</p><p>This approach does come with some overhead. Without caching, the client may perform more DNS lookups. In very stable environments that overhead may not be necessary.</p><p>One operational detail is worth calling out: the custom event loop group creates non-daemon threads, so it should be shut down explicitly when the application stops. You also need to include Netty’s DNS resolver dependency:</p><pre>&lt;dependencies&gt;<br>    &lt;dependency&gt;<br>        &lt;groupId&gt;io.netty&lt;/groupId&gt;<br>        &lt;artifactId&gt;netty-resolver-dns&lt;/artifactId&gt;<br>        &lt;version&gt;${netty.version}&lt;/version&gt;<br>    &lt;/dependency&gt;<br>&lt;/dependencies&gt;</pre><h3>6. Add jittered reconnect backoff</h3><p>The last pattern is reconnecting behavior.</p><pre>.reconnectDelay(Delay.fullJitter(<br>        Duration.ofMillis(100),<br>        Duration.ofSeconds(10),<br>        100, TimeUnit.MILLISECONDS<br>))</pre><p>When a Redis node or cluster becomes unavailable, many application instances may lose their connections at the same time. If every client immediately reconnects in lockstep, the system can create a thundering herd against recovering nodes or control-plane endpoints.</p><p>Using full jitter spreads reconnect attempts across time. Instead of synchronized retries, clients back off with randomness inside a bounded range. This improves cluster recovery and reduces self-inflicted load during incidents.</p><p>The exact bounds depend on your environment, but the general rule is consistent: avoid deterministic reconnect storms.</p><h3>Things to Consider</h3><p>These settings should not be evaluated in isolation, because they affect each other and the overall system load.</p><p>Periodic refresh and adaptive refresh help keep the cluster topology up to date, but they also increase refresh traffic. Dynamic refresh sources make topology discovery more resilient, but they can be costly in large clusters because more nodes are queried. Socket and timeout settings help the client fail faster, but they should still be set according to how long the application can reasonably wait. DNS resolver changes help the client reach the correct nodes after topology changes, and reconnect jitter helps avoid too many clients reconnecting at the same time.</p><p>The key point is to apply these settings with their trade-offs in mind. A more aggressive setup can work well for small or medium-sized clusters. In larger clusters, the same configuration may create unnecessary overhead. The final configuration should match the size of the cluster and the behavior of the system using it.</p><h3>Conclusion</h3><p>Redis failover resilience is not only a server-side concern. The client also needs to be configured to detect topology changes quickly, avoid stale or unhealthy paths, and fail fast enough to protect the rest of the application. Lettuce provides the necessary building blocks, but achieving production-ready behavior requires deliberate decisions around topology refresh, socket options, timeout strategies, DNS resolution, node filtering, and reconnect backoff.</p><p>A well-designed Redis resilience configuration does not try to eliminate failure entirely. Instead, its purpose is to make failures shorter, more contained, and easier to recover from. The real objective is not perfect availability, but controlled degradation and fast convergence back to a healthy state.</p><h3>References</h3><ul><li>TCP Keepalive is a lie — <a href="https://codearcana.com/posts/2015/08/28/tcp-keepalive-is-a-lie.html">https://codearcana.com/posts/2015/08/28/tcp-keepalive-is-a-lie.html</a></li><li>When TCP sockets refuse to die — <a href="https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/">https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/</a></li><li>AWS Lettuce client configuration — <a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/BestPractices.Clients-lettuce.html">https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/BestPractices.Clients-lettuce.html</a></li><li>Lettuce documentation — <a href="https://redis.github.io/lettuce/">https://redis.github.io/lettuce/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c81f2e799dfa" width="1" height="1" alt=""><hr><p><a href="https://engineering.getmidas.com/building-resilient-redis-clients-with-lettuce-c81f2e799dfa">Building Resilient Redis Clients with Lettuce</a> was originally published in <a href="https://engineering.getmidas.com/">Midas Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>