P2P message rate limit

[fyyhtx]

Background

Java-tron currently does not limit the rate of processing P2P messages, but the processing capacity of network nodes is limited by physical resources such as bandwidth, CPU, and memory. Processing a large number of P2P messages will cause excessive consumption of resources. Competition for resources among different applications may cause the following problems:

• Risk of resource overload: Burst traffic or high concurrent requests may cause the node processing capacity to be saturated, affecting normal service response.
• Degradation of service quality (QoS): Critical traffic (such as transaction/block broadcast) may be delayed or lost because non-critical traffic occupies resources.

For example, SyncBlockChainMessage is sent to request synchronization of block lists, and FetchInvDataMessage is sent to request blocks or transactions. Java-tron currently does not limit the frequency of processing these request messages. If a large number of these request messages are received in a short period of time, when processing these messages, since the size of the reply message is much larger than the size of the request message, it is very likely that the node's bandwidth, CPU, memory and other system resources will be over-consumed, thereby affecting the stability of the node service.

By comparing with the code implementation of Ethereum, it is found that Ethereum has set corresponding qps limits for related messages. Before processing a message, it will first check whether the rate exceeds the limit. If it exceeds the limit, the message will no longer be processed and the corresponding error code will be responded.

• Status request frequency cannot exceed 5 qps
• Goodbye message frequency cannot exceed 1 qps
• Ping message frequency cannot exceed 5 qps
• GetMetaData request frequency cannot exceed 5 qps
• Block synchronization request no more than 64 blocks within 30s

Rationale

You can use the frequency of normal message sending as a benchmark to set appropriate frequency limits for different messages without affecting normal functions. The current P2P messages of java-tron are sorted out as follows:

• SYNC_BLOCK_CHAIN: When remainNum = 0 (only a few blocks to be synchronized, less than 4000), it has been controlled by code logic and no rate limit is required. When remainNum > 0, the SYNC_BLOCK_CHAIN ​​message will be sent twice in succession under normal logic. At least 2000 blocks need to be processed before the next SYNC_BLOCK_CHAIN ​​message can be sent. Each block takes at least 2ms to process, and it takes at least 4s to process 2000 blocks. The rate of SYNC_BLOCK_CHAIN ​​messages can be limited, and the specific threshold will be determined after verification.
• FETCH_INV_DATA (synchronization phase): The block acquisition task is scheduled once per second, so under normal logic, it will only be acquired once within 1 second. The rate of FETCH_INV_DATA messages can be limited, and the specific threshold will be determined after verification.
• Under normal logic, the number of blocks at a time will not exceed 100 blocks. The number of blocks at a time can be limited to no more than 100 blocks.
• P2P_HELLO: Handshake message, which is sent only once in the entire life cycle of a peer, has been controlled by code logic.
• P2P_DISCONNECT: Disconnect message. After receiving the message, the connection will be disconnected and no response will be given. You don't need to consider the frequency limit for now.
• BLOCK_CHAIN_INVENTORY: SYNC_BLOCK_CHAIN ​​response message. If no SYNC_BLOCK_CHAIN ​​message is sent, an error will be reported and the connection will be disconnected when BLOCK_CHAIN_INVENTORY is received. This has been controlled by code logic.
• INVENTORY: In theory, if the other node sends an invalid list (such as a list of historical transaction blocks), the bandwidth cost is higher and it is difficult to limit the frequency. It can be controlled by limiting the transaction tps speed (but the current threshold is relatively large). Under normal circumstances, the peer will send a block-type inventory message to the local node once every 3 seconds.
• FETCH_INV_DATA(Broadcast phase): You can only request lists that have been sent to the other party, and you cannot request the same list repeatedly. This has been controlled by code logic.
• BLOCK: The response message of the FETCH_INV_DATA message has been controlled by code logic.
• TRXS: The response message of the FETCH_INV_DATA message has been controlled by code logic.
• PBFT_COMMIT_MSG: PBFT messages. The PBFT function will not be enabled for the time being. After receiving the message, it will be discarded directly without any restrictions for the time being.

When the limit is exceeded, the message is discarded directly and the number of times the peer exceeds the limit is recorded for use by the peer scoring logic.

Implementation

1. Rate Limiting Logic

• When remainNum > 0, the SyncBlockChainMessage frequency cannot exceed 2 qps
• The FetchInvDataMessage request block frequency during the synchronization phase cannot exceed 1 qps
• The number of blocks in a single transaction cannot exceed 100 blocks
• The P2P_DISCONNECT message frequency cannot exceed 1 qps
• The frequency of INVENTORY messages of type block cannot exceed 1 qps

2. Add the definition class P2pRateLimiter, which provides the following methods:

• register: register the rate limit for messages of a specified type
• acquire: record data related to the message rate, and block the processing thread when the rate limit is exceeded
• tryAcquire: record data related to the message rate, and return false when the rate limit is exceeded

3. Define the variable p2pRateLimiter in the PeerConnection class.
4. Register the limit rates of SyncBlockChainMessage, FetchInvDataMessage and P2P_DISCONNECT messages to 2, 1 and 1 respectively.
5. When processing the corresponding message, check whether the corresponding frequency exceeds the preset limit. If so, discard it; otherwise, continue to process the message.
6. When processing the FetchInvDataMessage message, verify that the number of block hashes carried cannot exceed 100. If it exceeds 100, the peer connection will be disconnected.

Do you have ideas regarding the implementation of this feature?
yes
Are you willing to implement this feature?
yes

[jakamobiii]

It seems necessary to limit the speed of P2P messages, and Ethereum has done the same thing. I would also like to ask, has the Tron network encountered similar problems so far?

[fyyhtx]

@jakamobiii As far as I know, there has been no such problem so far. Or it may have happened, but it did not cause obvious impact and therefore went undetected. But in any case, it is better to prevent it before it happens.

[jwrct]

@fyyhtx Will Ethereum's Goodbye message also disconnect? If so, referring to Ethereum's logic, can Tron's P2P_DISCONNECT message also be limited to 1 qps?

[fyyhtx]

@jwrct Yes, the Ethereum code also disconnects the peer after sending the Goodbye message. Your suggestion is very good, we can consider limiting the frequency of P2P_DISCONNECT messages to 1 qps.

[Sunny6889]

This is an excellent proposal. It would be beneficial to make it configurable, as some FullNodes may only wish to synchronize with the Mainnet or consider other proposals.

[fyyhtx]

@Sunny6889 This is a great suggestion, I think I will consider your suggestion carefully in the future implementation process.

[jwrct]

@fyyhtx I noticed that the latest java-tron code still has the processing logic for P2P_PING and P2P_PONG. Although the sending logic has been deleted, do you think it is necessary to add rate limit processing for them as well?

[fyyhtx]

@jwrct The removal of P2P_PING and P2P_PONG processing logic is indeed missed. It would be better to remove them in the version that releases P2P message rate limit or earlier, so that there is no need to consider rate limiting them.

[fyyhtx]

The preliminary implementation plan is as follows:

1. Rate Limiting Logic

• When remainNum > 0, the SyncBlockChainMessage frequency cannot exceed 2 qps
• The FetchInvDataMessage request block frequency during the synchronization phase cannot exceed 1 qps
• The number of blocks in a single transaction cannot exceed 100 blocks
• The P2P_DISCONNECT message frequency cannot exceed 1 qps
• The frequency of INVENTORY messages of type block cannot exceed 1 qps

2. Add the definition class P2pRateLimiter, which provides the following methods:

• register: register the rate limit for messages of a specified type
• acquire: record data related to the message rate, and block the processing thread when the rate limit is exceeded
• tryAcquire: record data related to the message rate, and return false when the rate limit is exceeded

3. Define the variable p2pRateLimiter in the PeerConnection class.
4. Register the limit rates of SyncBlockChainMessage, FetchInvDataMessage and P2P_DISCONNECT messages to 2, 1 and 1 respectively.
5. When processing the corresponding message, check whether the corresponding frequency exceeds the preset limit. If so, discard it; otherwise, continue to process the message.
6. When processing the FetchInvDataMessage message, verify that the number of block hashes carried cannot exceed 100. If it exceeds 100, the peer connection will be disconnected.

[fyyhtx]

The first version of the implementation code has been preliminarily completed. Those who are interested can help review it. For specific codes, please refer to: fyyhtx@aa682c9 and fyyhtx@e50e11f