Apache Spark DataFrames From Strings — Scala API

Spark Version: 2.x

This post is part of Apache Spark DataFrames — Scala Series.

Apache Spark DataFrames From Strings — Scala API

Hello Readers, In this post, I am going to show you how to create a DataFrame from a Collection of Strings using Scala API. You can copy paste the code line by line in Jupyter Notebook with Scala-Toree Kernel or to your favorite IDE with Scala and Spark dependencies or even use Spark’s Scala shell and run these line by line.

Let us consider that I have a list or sequence of strings and the end goal is to create a dataframe from this collection of Strings.

Step 1: Get or Create Spark Session

import org.apache.spark.sql.SparkSession

val spark =
      SparkSession.builder()
        .appName("DataFrames From Strings")
        .master("local[4]")
        .getOrCreate()

local[4] means run in local mode with 4 cores.

Step 2: Define Collection Of Strings

val employees = Seq(
      "7369, SMITH, CLERK, 7902, 17-Dec-80, 800, 20, 10",
      "7499, ALLEN, SALESMAN, 7698, 20-Feb-81, 1600, 300, 30",
      "7521, WARD, SALESMAN, 7698, 22-Feb-81, 1250, 500, 30",
      "7566, JONES, MANAGER, 7839, 2-Apr-81, 2975, 0, 20",
      "7654, MARTIN, SALESMAN, 7698, 28-Sep-81, 1250, 1400, 30",
      "7698, BLAKE, MANAGER, 7839, 1-May-81, 2850, 0, 30",
      "7782, CLARK, MANAGER, 7839, 9-Jun-81, 2450, 0, 10",
      "7788, SCOTT, ANALYST, 7566, 19-Apr-87, 3000, 0, 20",
      "7839, KING, PRESIDENT, 0, 17-Nov-81, 5000, 0, 10",
      "7844, TURNER, SALESMAN, 7698, 8-Sep-81, 1500, 0, 30",
      "7876, ADAMS, CLERK, 7788, 23-May-87, 1100, 0, 20"
    )

Step 3: Import Spark Implicits

Importing Spark implicits is Scala specific. spark.implicits extends SQLImplicits.

By importing Spark Implicits, a lot of methods will be available for converting common Scala objects into DataFrames. Check this link for the scala doc on this.

import spark.implicits._

Step 4: Function to Cleanse Data

If you observe above dataset, the field delimiter is “,” but there is a space as well. So, let’s define a function that takes each line and splits it by “,” (comma) and trims each field to get rid of extra space and finally returns a Row object.

Get Mallikarjuna G’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A row object represents a record which is essential to create a dataFrame.

import org.apache.spark.sql.Row

val splitAndStrip = (line: String) => {
      val rec = line.split(",").map(x => x.trim)
      Row(rec(0).toInt, rec(1), rec(2), rec(3).toInt, rec(4), rec(5).toDouble, rec(6).toDouble, rec(7).toInt)
    }

Step 5: Create DataFrame From RDD[Row]

To create a DataFrame from an RDD of Rows, we two options:

1. to use toDF() method
2. to use createDataFrame(rowRDD: RDD[Row], schema: StructType)

In this post, we will see how to use toDF() method. But, it is important to note that, toDF method works if RDD is of type RDD[INT], RDD[Long], RDD[String] and RDD[T <: scala.Product]. [Note: "<:" means T must be sub type of scala.Product in which case, we can use either a Case class or a Tuple. This is because, Product is the super class of both Case class and Tuple.]

Now in this case, each Row has to be either converted to a custom Case class or to a Tuple (N-Tuple)

Method 1: N Tuple:

// Method 1: N Tuple
val colList = List("empno", "ename", "job", "mgr", "hiredate", "sal", "comm", "deptno")

val empDF = spark.sparkContext.parallelize(employees, 4)
  .map(splitAndStrip)
  .map{ 
        case Row(empno: Int,ename: String,job: String,mgr: Int,hiredate: String,sal: Double,comm: Double,deptno: Int) =>
      (empno, ename, job, mgr, hiredate, sal, comm, deptno)
  }.toDF(colList: _*)

Method 2: Case Class:

case class EmpFromStrings(empno: Int, ename: String, job: String, mgr: Int, hiredate: String, sal: Double, comm: Double, deptno: Int)

val employeeDF = spark.sparkContext.parallelize(employees, 4)
  .map(splitAndStrip)
  .map{
    case Row(empno: Int,ename: String, job: String, mgr: Int, hiredate: String, sal: Double, comm: Double, deptno: Int) =>
      EmpFromStrings(empno, ename, job, mgr, hiredate, sal, comm, deptno)
  }.toDF(colList: _*)

Step 6: Output

+-----+------+---------+----+---------+------+------+------+
|empno| ename|      job| mgr| hiredate|   sal|  comm|deptno|
+-----+------+---------+----+---------+------+------+------+
| 7369| SMITH|    CLERK|7902|17-Dec-80| 800.0|  20.0|    10|
| 7499| ALLEN| SALESMAN|7698|20-Feb-81|1600.0| 300.0|    30|
| 7521|  WARD| SALESMAN|7698|22-Feb-81|1250.0| 500.0|    30|
| 7566| JONES|  MANAGER|7839| 2-Apr-81|2975.0|   0.0|    20|
| 7654|MARTIN| SALESMAN|7698|28-Sep-81|1250.0|1400.0|    30|
| 7698| BLAKE|  MANAGER|7839| 1-May-81|2850.0|   0.0|    30|
| 7782| CLARK|  MANAGER|7839| 9-Jun-81|2450.0|   0.0|    10|
| 7788| SCOTT|  ANALYST|7566|19-Apr-87|3000.0|   0.0|    20|
| 7839|  KING|PRESIDENT|   0|17-Nov-81|5000.0|   0.0|    10|
| 7844|TURNER| SALESMAN|7698| 8-Sep-81|1500.0|   0.0|    30|
| 7876| ADAMS|    CLERK|7788|23-May-87|1100.0|   0.0|    20|
+-----+------+---------+----+---------+------+------+------+

Step 7: Observations

Why are the curly brackets used and not the regular parentheses in the map?

Ans: In general, there are many cases where we should prefer curly braces (e.g. multiline expressions, for comprehensions) etc. In this case it’s not just curly braces, instead of parentheses, it’s curly braces with omited parentheses. The curly braces here allows us to write pattern matching anonymous function inside map/filter, whereas parenthesis won’t.

That’s all reg creating Dataframes from strings. I hope this is useful in your learning spark journey.

If you have come this far, it means that you liked what you are reading. Why not reach little more and like and share this post. Please comment below, I would really love to hear your thoughts and opinions on my articles.