Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0, 4.0.1
Description
MapOutputTrackerMaster.shufflestatuses can be mistakenly cleaned by Shuffle Cleanup feature, leading to SparkException (crashing the SparkContext) by the subsequent access to that already removed shuffle metadata. A real case (limited to local cluster currently) is the ongoing subquery could access the shuffle metadata which has been already cleanedup after the main query completes. See the detailed discussion at: https://github.com/apache/spark/pull/52213#discussion_r2415632474.
Attachments
Issue Links
- is caused by
-
SPARK-47764 Cleanup shuffle dependencies for Spark Connect SQL executions
-
- Resolved
-
- links to