Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-53898

MapOutputTrackerMaster.shufflestatuses is mistakenly cleaned by Shuffle cleanup

    XMLWordPrintableJSON

Details

    Description

      MapOutputTrackerMaster.shufflestatuses can be mistakenly cleaned by Shuffle Cleanup feature, leading to SparkException (crashing the SparkContext) by the subsequent access to that already removed shuffle metadata. A real case (limited to local cluster currently) is the ongoing subquery could access the shuffle metadata which has been already cleanedup after the main query completes. See the detailed discussion at: https://github.com/apache/spark/pull/52213#discussion_r2415632474.

      Attachments

        Issue Links

          Activity

            People

              Ngone51 wuyi
              Ngone51 wuyi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: