OutOfMemoryError in springboot stomp/websocket server with embedded tomcat when many clients connect

[tsiyona]

• I have a springboot application that acts as a websocket server using stomp. The embedded server is tomcat. I am using spring boot version 3.2.9 and springframework version 6.1.14.

• I also have a springboot client application that starts 5000 threads. Each thread opens a websocket connection with the server. The client is also using stomp.

• My server machine has 62G ram. As websocket connections are being opened, the memory consumption in the server rises. At 5000 connections, I reach 20G memory usage (I configured my server app to run with -Xmx30g heap size to prevent server failure).
  When I allow less memory, the server throws OutOfMemoryError.

• I saved a heap dump when reaching OOM error. Opened it with Eclipse MAT and saw that the dominant object is org.apache.tomcat.websocket.server.WsFrameServer.

• So tomcat as a suspect, I changed my server application to use jetty instead of tomcat.
  Results: The same server application with jetty instead of tomcat uses a total of 1g ram for 5000 connected websocket clients, plus I made each client send a message to the server every 5 sec. The problem does not exist with jetty.

So is tomcat really a problem using stomp/ws, or is it the combination of my springboot version plus stomp?

[bclozel]

It's hard to tell given the elements you have provided. Can you share more information on what's using memory? Maybe breaking down in more details how the memory is being used under that particular object.

It's not clear whether we're leaking memory somewhere, if we should improve our Tomcat support or if the problem is with Tomcat itself.

Thanks!

[tsiyona]

When I analyze my heap dump in Eclipse Memory Analyzer I see the below info showing the dominator_tree. The dominator class is WsFrameServer. Please note the total Retained Heap value consumed by this class instances.
This heap dump is from my websocket server running on tomcat. This heap dump profile was generated as a result of OOM Error.

• Does tomcat hold one instance of WsFrameServer per websocket connection made by the client?
• What tomcat configuration might I be missing that causes this class instances to inflate?
• Isn't 3M per instance too large? 5000 connections x 3M = 15.7GB which explains my OutOfMemoryError

Note that my websocket clients merely open a connection. They do not even explicitly send any messages to the server yet.

Class Name                                                                                  | Shallow Heap |  Retained Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------
org.apache.tomcat.util.threads.TaskThread @ 0x7fc20c7b8  http-nio-0.0.0.0-9084-exec-1 Thread|          120 |      3,184,464 |      0.02%
org.apache.tomcat.util.threads.TaskThread @ 0x7fc200810  http-nio-0.0.0.0-9084-exec-6 Thread|          120 |      3,184,304 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41484ae80                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x414f7cd88                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41556a250                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415577230                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x4155787e0                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415a106c8                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415d3c4f8                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x415e8f758                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416d08270                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416d14a70                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416f25978                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x416f326c0                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41720a300                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x4172179d8                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x417220690                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x41722ce90                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x4186047d0                              |          120 |      3,154,904 |      0.02%
org.apache.tomcat.websocket.server.WsFrameServer @ 0x418610ff0                              |          120 |      3,154,904 |      0.02%
Total: 20 of 232,891 entries; 232,871 more                                                  |    9,241,248 | 14,741,608,128 |    100.00%
-----------------------------------------------------------------------------------------------------------------------------------------

[tsiyona]

One more input: I removed stomp from the code. Made the client and server use raw websockets instead. After 5000 client websocket established connections to the server, the server heap usage grew by ~300-400 MB only. So maybe here hides the culprit.
I don't know if this can narrow down the suspects. It seems that the combination of springboot+tomcat+stomp/ws doesn't function well.
Maybe some tomcat configuration is required?

[tsiyona]

One more update: I limit the message size now. It was 1M and I changed it to the default 8K and it affects the heap usage dramatically. From 16G to slightly less than 1G:

    @Bean
    @ConditionalOnWebApplication
    public ServletServerContainerFactoryBean createWebSocketContainer() {
        final ServletServerContainerFactoryBean container = new ServletServerContainerFactoryBean();
        container.setMaxBinaryMessageBufferSize(1024 * 8);   // default is 8K
        container.setMaxTextMessageBufferSize(1024 * 8);      // default is 8K
        return container;
    }

[tsiyona]

Apparently I resolved the issue. Thanks for listening :)

[bclozel]

I guess Tomcat allocates memory that scales with the message buffer size. Thanks for letting us know.