Spark Query Execution

Spark Query Execution can be split into 3 parts.

• Logical Plan
• Physical Plan
• Executed Plan

Logical Plan

• Logical planning is a phase in which Spark will create a logical plan and this plan will be analyzed and optimized.
• Tree representation of the query.
• It is an abstraction that carries information about what is supposed to happen.
• It does not contain precise information on how it happens.
• It consists of Relational Operators and Expressions.

Physical Plan

Get Tharun Kumar Sekar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in
• Execution layer does not understand Dataframe / Logical Plan.
• Logical plan has to be converted to a Physical plan.
• Physical plan acts as a bridge between Logical and Execution layer.
• Similar to Logical plan it is a tree.
• Contains more specific information of how things are going to happen (choices of algorithms).

Executed Plan

• Final version of the query plan.
• This will be executed against the data.

Ways to look at the generated plans in SCALA:

• df.queryExecution.sparkPlan
• df.queryExecution.executedPlan

Common Issue CheckList

• Enough partitions for concurrency.
  If you have 20 cores, make sure you have at least 20 partitions or more than that.
• Minimize memory consumption by filtering the data you need.
• Minimize the amount of data shuffled. Shuffle is expensive.
• Know the standard library and use the right functions in the right place.