Client hangs on socket error during connection initialization #3104
Description
Description
During the establishment of a connection to Redis (initial client.connect() or any reconnection later on), there is a time window when receiving a socket error hangs the client indefinitely in READY state. Under the right conditions, this is 100% reproducible.
Background
Initially I bumped into this behavior by stopping Redis container in Colima on MacOS while the connection is active. It drops the connection -> client tries to reconnect -> successfully opens a socket -> immediately closes with Socket closed unexpectedly, triggering the described behavior.
This behavior of the container runtime could be platform specific, so I made a reproducible example with a TCP proxy that adds an artificial delay to make it easy to disconnect at the right moment and trigger the condition.
Client code behavior
From socket.ts:
this.#socket = await this.#createSocket();
this.emit('connect');
try {
// While this is in-flight, receiving socket error causes the hang.
// #onSocketError() will be called to emit an error
// and do nothing because `this.#isReady === false`.
await this.#initiator();
// this.#initiator() successfully resolves
} catch (err) {
// This doesn't happen, no error in this.#initiator()
...
}
// Proceed to set `this.#isReady = true`,
// effectively stopping the loop in `#connect`
// and leaving the client in READY state without a chance to reconnect.
this.#isReady = true;
this.#socketEpoch++;
this.emit('ready');Reproducible example
The code creates a TCP proxy with an artificial delay. Then when the client tries to connect, it abruptly closes the connection causing Socket closed unexpectedly error. With the delay in place, it's easy to get an error during this.#initiator() call, and reproduce the hang.
import net from "node:net";
import { GenericContainer } from "testcontainers";
import { createClient } from "redis";
async function main() {
const redisPort = 6379;
const proxyPort = 6382;
const container = await new GenericContainer("redis")
.withExposedPorts({ container: 6379, host: redisPort })
.start();
const proxy = await startProxy({
redisPort,
proxyPort,
socketDelayMs: 2000,
});
const client = createClient({ socket: { port: proxyPort } });
client.on("error", (err) => console.log(`Client error: ${err}`));
console.log("Call client.connect()");
client.connect().finally(() => console.log("CONNECT finished"));
// Sleep enough to let the client establish the connection, but before handshake commands finish.
await timeout(500);
console.log("Close the connection");
proxy.forceClose();
// The client hangs indefinitely there and doesn't perform any commands or reconnections.
// The client will report READY state.
// This will not finish.
console.log("Send a command");
let commandFinished = false;
client.set("key", "value").finally(() => {
console.log("SET finished");
commandFinished = true;
});
// Give client a chance to finalize the command.
await timeout(5000);
console.log(`Closing after 5s, commandFinished = ${commandFinished}`);
await container.stop();
}
function startProxy({ redisPort, proxyPort, socketDelayMs }) {
return new Promise((res) => {
const clientSockets = [];
const proxy = net.createServer((clientSocket) => {
console.log("Client connected to proxy");
clientSockets.push(clientSocket);
// Connect to actual Redis server
const redisSocket = net.createConnection(redisPort, "localhost");
// Proxy data from client to Redis with delay
clientSocket.on("data", (data) => {
setTimeout(() => {
redisSocket.write(data);
}, socketDelayMs);
});
// Proxy data from Redis to client with delay
redisSocket.on("data", (data) => {
setTimeout(() => {
clientSocket.write(data);
}, socketDelayMs);
});
});
// Add a method to forcefully close all connections to cause "Socket closed unexpectedly" error.
proxy.forceClose = () => {
clientSockets.forEach((socket) => socket.destroy());
proxy.close();
};
proxy.listen(proxyPort, "0.0.0.0", () => res(proxy));
});
}
async function timeout(ms) {
return new Promise((res) => setTimeout(res, ms));
}
main();Node.js Version
v22.14.0
Redis Server Version
7.4.1
Node Redis Version
5.8.3
Platform
MacOS
Logs
// Logs from the script above
❯ node test-containers.mjs
Call client.connect()
Client connected to proxy
Close the connection
Forcefully closing 1 client socket(s)
Send a command
Client error: Error: Socket closed unexpectedly
CONNECT finished
Closing after 5s, commandFinished = false