Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-49411

Communicate RocksDB State Store CheckpointID Between Driver and Executor

    XMLWordPrintableJSON

Details

    Description

      A incremental step to implement RocksDB state store checkpoint format V2.

      Once conf STATE_STORE_CHECKPOINT_FORMAT_VERSION is set to be higher than version 2, the executor returns checkpointID to the driver (only done for RocksDB). The driver stores is locally. For the next batch, the checkpointID is sent to the executor to be used to load the state store. If the local version of the executor doesn't match the uniqueID, it will reload from the checkpoint.

      Attachments

        Issue Links

          Activity

            People

              siying Siying Dong
              siying Siying Dong
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: