Is the continuous batching function enabled by default in vllm?

[SeibertronSS]

Is the continuous batching function enabled by default in vllm? Can this feature be turned on or off selectively?

[zhuohan123]

Yes, this is enabled by default and cannot be turned off. Turning off continuous batching requires a rewrite of our system architecture, which also brings no benefit in performance. Therefore, we did not implement this.

[Kelsey2018]

So, what the continuous batching specifically is in VLLM?

[zhaozhixin]

Does continues batching has a way to be adjusted? I found that the latency increased significantly when using api_server than offline inference.

[aniketmaurya]

How do we control the batch size and timeout for continous batching though?