Spark Developer Tips

Spark Coding Tips

1. When working on the maps in Scala and spark : When we are applying any map transformation on the dataset , we cannot refer any other dataset inside the map , as it may be empty and cause null pointer exception. Because the map runs on executors and the dataset results are present in the driver node for other datasets.

“org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations”

2. When working on one small dataset and one large dataset, we can use the broadcast option to send the small dataset to the executors to reduce the latency and improve the performance.

3. When we want to do an empty check in the dataset and perform some operation based on the result we can convert to RDD and apply isEmpty. Ex: ds.rdd.isEmpty

Get Bhaskar Berry’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

4. In distributed system we generally serialize and deserialize the classes in Scala and spark to reduce the transfer time and improve the performance , but in general when we serialize the class it is a costly operation and we do get serialization issues, so to avoid the serialization issues we should convert the code to function where ever possible and do convert to objects and apply closures or map transformation.

How to resolve it?

Hard to say what exactly caught for serialization issues because it depends on your Spark job code. But there are some tips you should try:

• Declare your variables and functions inside an Object as much as possible. An Object looks like a global variable and we can use anywhere inside your code.
• Declare your own class extends Serializable to make sure your class will be transferred properly.
• If there is a variable which can not serialize then you can use an annotation @transient like this:

@transient lazy val queue: AmazonSQS = AWSQueueUtils.getClient(config)

• If you use flatMap or mapPartitions then you should follow the below practice due to the TravelsableOnce usage

val result: Int = df.rdd.mapPartitions(p => Iterator {
  // Collect data per partition first
  val partition: Seq[Row] = p.toList
  // Any calculation on partition val
  ...
}).collect().sum