Netty ChannelManager.tryToOfferChannelToPool discard chunks after offering to pool

[maxpower-artvandelay]

Hello, the Netty ChannelManager(AHC 1.9.11) seems to have an issue on tryToOfferChannelToPool.

public final void tryToOfferChannelToPool(Channel channel, boolean keepAlive, String partition) {
    if (channel.isConnected() && keepAlive && channel.isReadable()) {
        LOGGER.debug("Adding key: {} for channel {}", partition, channel);
        channelPool.offer(channel, partition);
        if (maxConnectionsPerHostEnabled)
            channelId2KeyPool.putIfAbsent(channel.getId(), partition);
        Channels.setDiscard(channel);
    } else {
        // not offered
        closeChannel(channel);
    }
}

Looks like channelPool.offer should happen after Channels.setDiscard, otherwise the channel becomes available for selection while the previous request cleanup is still happening. This will cause a "java.util.concurrent.TimeoutException: Request timed out" under certain loads.

This is the snippet I used to reproduce the issue:

final AsyncCompletionHandler<Response> handler = new AsyncCompletionHandler<Response>() {
    @Override
    public void onThrowable(Throwable t) {
        System.out.println(t.toString());
    }

    @Override
    public Response onCompleted(Response response) throws Exception {
        return response;
    }
};

Request request = httpClient.prepareGet("http://.../").build();
for (int i = 0; i < 500; ++i) {
    httpClient.executeRequest(request, handler);
}

I think the fix should be something similar to:

Channels.setDiscard(channel);
channelPool.offer(channel, partition);

instead of :

channelPool.offer(channel, partition);
Channels.setDiscard(channel);

[slandelle]

More exactly: tryToOfferChannelToPool sets channel flag to discard after offering, but channel could possibly have been selected in the meanwhile.

[slandelle]

@maxpower-artvandelay Nice catch, thanks for fixing!