Details
-
Task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0
Description
A incremental step to implement RocksDB state store checkpoint format V2.
Once conf STATE_STORE_CHECKPOINT_FORMAT_VERSION is set to be higher than version 2, the executor returns checkpointID to the driver (only done for RocksDB). The driver stores is locally. For the next batch, the checkpointID is sent to the executor to be used to load the state store. If the local version of the executor doesn't match the uniqueID, it will reload from the checkpoint.
Attachments
Issue Links
- links to