Netty handshake timeout for large request body when used with proxy server

[nitsh]

I am calling an email marketing service with a huge body(html). The size of the body = 117184.

I am using AsyncHttpClient as follows.

When I call the request directly, it is successful but it fails with the following exception when I try to use an internal https proxy server. But a small body with proxy server works fine. Following is the scala code :

val client = asyncHttpClient(config().setProxyServer(proxyServer("XX-XYXY-100X.XXXX.local", 8080)))
val whenResponse = client.preparePost("https://api.sendgrid.com/v3/mail/send")
  .addHeader("Authorization", "Bearer XXXXXXXXX")
  .addHeader("Content-Type", "application/json")
  .setBody(randomBody)
  .execute
  .get(1000, TimeUnit.SECONDS)

Basically :

large body and no proxy server : works fine.
small body and with and without proxy server : works fine.
large body with proxy server : DOESN'T WORK.
I see the following exception for requests with large body using proxy server.

Exception in thread "main" java.util.concurrent.ExecutionException: javax.net.ssl.SSLException: handshake timed out
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
    at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:207)
    at send.RequestSender$.send1(RequestSender.scala:27)
    at send.RequestSender$.delayedEndpoint$send$RequestSender$1(RequestSender.scala:47)
    at send.RequestSender$delayedInit$body.apply(RequestSender.scala:14)
    at scala.Function0.apply$mcV$sp(Function0.scala:34)
    at scala.Function0.apply$mcV$sp$(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App.$anonfun$main$1$adapted(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:389)
    at scala.App.main(App.scala:76)
    at scala.App.main$(App.scala:74)
    at send.RequestSender$.main(RequestSender.scala:14)
    at send.RequestSender.main(RequestSender.scala)
Caused by: javax.net.ssl.SSLException: handshake timed out
    at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source)

PS: stackoverflow

[slandelle]

There's absolutely no reason for TLS handshake to be related to upload size, as the former happens BEFORE (just after opening a new socket) the latter.

Does this happen with one single request, or does it happen under load? Are you sure you're not simply saturating your bandwidth and/or your proxy with your large uploads, so that TLS handshake TCP segments can't get through?

PS: I no longer answer on stackoverflow

[nitsh]

@slandelle Just one request irrespective of the load. I've tried randomly generating the request body of the same size and it doesn't work for that either. The same request with large body works fine without the proxy server FYI.
It has always failed consistently for the request, also in different machines.

[slandelle]

Could you please provide a way to reproduce?

[nitsh]

@slandelle Yes, I was just able to figured out how to reproduce :
Step 1 : Start squid local proxy server. You can also start on docker. Image : https://hub.docker.com/r/datadog/squid/ .
Step 2 : Run the following code.

import org.asynchttpclient.Dsl._
import java.util.concurrent.TimeUnit
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import scala.util.Random

    val randomLength = 117184
    def randomBody = Random.nextString(randomLength)
    val client = asyncHttpClient(config().setProxyServer(proxyServer("localhost", 32768)))
    val whenResponse = client.preparePost("https://api.sendgrid.com/v3/mail/send")
      .addHeader("Content-Type", "application/json")
      .setBody(randomBody)
      .execute
      .get(1000, TimeUnit.SECONDS)
    println(whenResponse)

You should ideally see an error message from sendgrid saying Permission denied, wrong credentials or a RemotelyClosedException

[nitsh]

Wanted to add one more point : The same request works fine when using a different http client, scalaj-http which is a wrapper of java.net.HttpURLConnection

[slandelle]

Thanks for reporting.

[nitsh]

@slandelle Thanks for the quick fix! When can I expect it to be published?

[slandelle]

As soon as uploading on Sonatype stops failing...

[slandelle]

I managed to release. Just need to wait for sync with maven central.