🌟Big Data? Meet Apache Spark!

Learn how to use Apache Spark with Python and PySpark code examples

Official logo of Apache Spark

Apache Spark is a powerful and very popular open-source framework for distributed computing, designed to handle large-scale data processing tasks efficiently. It supports multiple programming languages, including Python via PySpark, and provides a range of APIs for tasks like data processing, machine learning, and streaming. If you or your company are handling vast amounts of data then you are probably using this framework.

Hi, my name is CyCoderX and in this article, I will explain the basics of Apache Spark, demonstrate its functionality and include output results to illustrate the concepts.

Let’s dive in!

I write articles for everyone to enjoy, and I’d love your support by following me for more Python, SQL, Data Engineering and Data Science content.😊

Introduction

Apache Spark, created in 2009 by Matei Zaharia at UC Berkeley’s AMP Lab, emerged as a solution to the growing need for processing large amounts of data. At the time, the volume of data being collected on the internet was skyrocketing, growing from megabytes to petabytes, which made it increasingly difficult to analyze using a single machine.

But there was already a clever programming model called MapReduce was introduced. This model allowed data to be mapped into key-value pairs, shuffled and sorted into groups by key, and then reduced to compute a final result. While this approach made it possible to distribute large datasets across multiple machines, it still faced a significant bottleneck due to the reliance on disk I/O operations.

Spark addressed this issue by performing most of its processing in memory rather than reading from disk, which can be up to 100 times faster. This shift from disk to memory-based operations revolutionized big data analytics and machine learning, making Spark a game-changer in the field. It is now widely used by organizations such as Amazon to analyze e-commerce data, NASA’s Jet Propulsion Lab to analyze deep space data, and over 80% of Fortune 500 companies to process their massive datasets.

Despite its reputation for handling distributed big data processing, Apache Spark can also be run locally on a personal machine. Written in Java and running on the JVM, Spark offers APIs that can be used with various programming languages such as Python, SQL, and more.

To get started with Spark, one can easily install it and begin working with it locally. For instance, imagine you have a CSV file with four columns — City, Population, Latitude, and Longitude — and your task is to find the city with the largest population between the tropics. The first step would be to initialize a Spark session and load the data into memory, allowing Spark to perform the necessary computations efficiently.

Setting Up PySpark

Unfortunately, we in Python to use PySpark we will need to install it with pip. Do make sure though that you have Java SDK and Spark are installed on your system.

Installation

pip install pyspark

To verify installation you can run:

python -c "import pyspark; print(pyspark.__version__)"

The first step to interacting with Spark is to create a SparkSession:

from pyspark.sql import SparkSession  

spark = SparkSession.builder \
    .appName("PySpark Example") \
    .getOrCreate()

This sets up a basic Spark environment for running your tasks.

Loading and Inspecting Data

Lets load a sample dataset into a DataFrame and inspect it. For this example, I will use the array data.

Loading Data

data = [("Alice", 29), ("Bob", 35), ("Cathy", 45)]  
columns = ["Name", "Age"]  

df = spark.createDataFrame(data, columns)  
df.show()

Output

+-----+---+  
| Name|Age|  
+-----+---+  
|Alice| 29|  
|  Bob| 35|  
|Cathy| 45|  
+-----+---+

Rethinking My Python Developer Path

Now lets now assume we have the following csv file:

City,Population,Area,Population Density
New York,8419600,789.4,10661
Los Angeles,3980400,1213.9,3283
Chicago,2716000,606.1,4484
Houston,2328000,1625.2,1432
Phoenix,1690000,1340.6,1255
San Antonio,1547253,1194.3,1296
San Diego,1423851,964.5,1475
Dallas,1341000,880.1,1520
Austin,978908,437.2,2244
Miami,467963,143.1,3267
Mexico City,8918653,1485.0,6013
Lagos,9000000,1171.3,7674
Bangkok,8300000,1569.0,5297
Jakarta,10400000,662.3,15796
Manila,1780148,38.55,46190
Singapore,5612300,728.6,7700

Explanation of Columns:

• City: Name of the city
• Population: The population of the city
• Area (km²): This column contains the area of the city, measured in square kilometers (km²). It shows the size of the city in terms of its land coverage.
• Population Density (people/km²): This column shows how densely populated the city is, which is calculated by dividing the population by the area of the city. It is expressed as the number of people per square kilometer.

Task:

We want to find the city with the highest population density (people per square kilometer).

Python Spark Code:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Initialize the Spark session
spark = SparkSession.builder \
    .appName("City with Highest Population Density") \
    .getOrCreate()

# Load the CSV file into a Spark DataFrame
df = spark.read.csv("cities_population_density.csv", header=True, inferSchema=True)

# Find the city with the highest population density
city_with_highest_density = df.orderBy(col("Population Density (people/km²)").desc()).first()

# Show the result
print(f"The city with the highest population density is {city_with_highest_density['City']} with a population density of {city_with_highest_density['Population Density (people/km²)']} people/km².")

# Stop the Spark session
spark.stop()

Explanation of Code:

Initialize Spark Session:

• Creates a Spark session to interact with Spark, acting as the entry point for all operations.

Load Data:

• Reads the CSV file (cities_population_density.csv) into a Spark DataFrame.
• Automatically infers data types and treats the first row as column headers.

Sort Data:

• Sorts the DataFrame by the Population Density column in descending order using orderBy() and desc().

Find the Highest:

• Retrieves the city with the highest population density from the sorted DataFrame using .first().

Display the Result:

• Prints the name of the city with the highest population density and its value.

Stop Spark Session:

• Stops the Spark session to free up resources once the operation is complete.

Expected Output:

Given the data above, the city with the highest population density is Manila, with a population density of 46,190 people/km². The result would look like this:

The city with the highest population density is Manila with a population density of 46190 people/km².

Data Transformations

Tha above code was just an example script with PySpark’s cababilities. There are many transformations like filtering, adding columns and grouping that we can perform on the data and they are very similar to the Pandaslibrary. Sparks provides nice and easy functions

Filtering Data

filtered_df = df.filter(df.Age > 30)  
filtered_df.show()

Adding a Column

df_with_years = df.withColumn("YearsUntilRetirement", 65 - df.Age)  
df_with_years.show()

FAANG vs. Startup Which to Choose?

Join my LinkedIn Group to connect with like-minded professionals and stay ahead of industry trends!

5 Linux Command Tricks That Will Change Your Life as a Programmer

Joins and Aggregations

PySpark is useful for distributed data processing and shares similarities with Python libraries like Pandas, making it accessible for Python users. The syntax of PySpark functions often resembles that of Pandas, which helps streamline learning for those already familiar with Python data manipulation.

Joining DataFrames

PySpark allows you to join two DataFrames easily. Here’s an example:

data2 = [("Alice", "HR"), ("Bob", "Engineering"), ("Cathy", "Finance")]  
columns2 = ["Name", "Department"]  

df2 = spark.createDataFrame(data2, columns2)  

joined_df = df.join(df2, on="Name", how="inner")  
joined_df.show()

Output

+-----+---+------------+  
| Name|Age| Department|  
+-----+---+------------+  
|Alice| 29|          HR|  
|  Bob| 35|Engineering|  
|Cathy| 45|     Finance|  
+-----+---+------------+

Aggregating Data

Aggregation functions in PySpark are similar to Pandas operations:

from pyspark.sql.functions import avg  

avg_age = df.groupBy().agg(avg("Age")).collect()[0][0]  
print(f"Average Age: {avg_age}")

Output

Average Age: 36.333333333333336

Machine Learning with PySpark

PySpark’s MLlib module provides scalable machine learning algorithms. Here’s an example using linear regression:

Preparing Data

from pyspark.ml.feature import VectorAssembler  
from pyspark.ml.regression import LinearRegression  

data = [(1.0, 2.0), (2.0, 4.0), (3.0, 6.0), (4.0, 8.0)]  
columns = ["Feature", "Label"]  

df = spark.createDataFrame(data, columns)  

vector_assembler = VectorAssembler(inputCols=["Feature"], outputCol="Features")  
transformed_data = vector_assembler.transform(df)

Training the Model

lr = LinearRegression(featuresCol="Features", labelCol="Label")  
model = lr.fit(transformed_data)  

print(f"Intercept: {model.intercept}")  
print(f"Coefficients: {model.coefficients}")

Output

Intercept: 0.0  
Coefficients: [2.0]

This example demonstrates how PySpark can handle machine learning tasks at scale, similar to Python-based frameworks like Scikit-learn.

Explore Secret HTTP Requests in Python

Conclusion

Apache Spark, via the PySpark wrapper, offers a powerful yet familiar way to process large-scale data. With its syntax close to Python and functionalities that resemble Pandas, it bridges the gap between local data manipulation and distributed computing.

Get CyCoderX’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

From simple transformations to advanced machine learning, PySpark allows you to scale your operations seamlessly while using Python’s intuitive syntax. Whether you’re managing data pipelines, conducting analyses, or deploying machine learning models, PySpark is a valuable tool for data processing at scale.
Good Luck!

The Python Operator You Didn’t Know You Needed!

Note that I am not sponsored or affiliated with any services or products mentioned; my advice comes from my own experiences. Always conduct your own research and verify online information.

Final Words:

Thank you for taking the time to read my article. Article first published on Medium by CyCoderX.

Hi, I’m CyCoderX! An engineer passionate about sharing knowledge, I write articles about Python, SQL, Data Science, Data Engineering and more!

Please consider supporting me by:

1. Clapping 50 times for this story
2. Leaving a comment telling me your thoughts
3. Highlighting your favorite part of the story
4. Connecting with me on LinkedIn and joining my LinkedIn Group

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Discord | Newsletter | Podcast
• Create a free AI-powered blog on Differ.
• More content at PlainEnglish.io