#1379 introduced ConnectionSemaphore to deal with connection limits.
When no such limit is set, which is the default, we could bypass all this code.