Apache Spark Fundamentals

This article contains notes on Apache SPARK, based on Manish Kumar’s YouTube tutorial, along with insights gathered from various other notes and resources. The following topics have been covered:

1. What is Apark? Why it is needed?
2. Hadoop vs Spark.

What is Apache Spark?

Apache Spark is a unified computing engine and a set of libraries for parallel data processing on a computer cluster.

Let’s break down this definition:

1. Unified

Unified means that data analysts, data scientists, and data engineers can all use the same platform for various tasks like:

• Data analysis
• Data transformation
• Modeling.

This allows for seamless collaboration and a consistent tool for different roles.

2. Computing Engine

Spark does not store data by itself. Instead, it performs computations (like addition, multiplication, etc.) on data.

To store data, Spark provides connectors to various data sources, such as:

• HDFS
• S3
• Databases

Compute refers to processing tasks — not storing data, just doing operations on it.

3. Parallel Data Processing

• Parallel processing means dividing a large task into smaller sub-tasks and executing them simultaneously on multiple processors.
• Example: Instead of using a single processor to search 1 million records, Spark divides the task:
• One processor searches the first 500,000 records.
• Another processor searches the next 500,000 records.
• This significantly speeds up the process.

4. Computer Cluster

• A cluster is a group of interconnected computers, where:
• One computer acts as the master (coordinator).
• Other computers act as slaves (workers).
• Spark distributes the computation tasks across all the nodes in the cluster, allowing it to scale horizontally and handle large datasets efficiently.

In Summary:

Apache Spark is a powerful framework for performing parallel data processing across multiple machines (a cluster) without handling storage itself. Its unified nature allows users across roles to perform various tasks seamlessly, making it a go-to solution for big data processing.

WHY IT IS NEEDED?

Earlier, we mainly handled data in tabular formats using traditional databases like MySQL or Oracle. But over time, data started coming in various formats such as:

• Structured data — tables in databases
• Semi-structured data — JSON, XML, YAML
• Unstructured data — text files, images, videos, logs, etc.

These traditional systems could only handle structured data well and struggled with others.

At the same time, Big Data emerged. This is data that is too large and complex for traditional tools to process efficiently. Big Data is defined by the three Vs:

• Volume: amount of data (5GB, 5TB )
• Velocity: speed of data (1sec, 1 hour)
• Variety: forms of data (structured, unstructured, semi-structured_

Data is not classified as big data just because it is 5GB or 5TB. The velocity and variety also matters. So if data is generated at 5GB/sec and data includes both structured and unstructured, then we can call it big data.

Get Riyanka Pal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To address this growing complexity, the traditional ETL (Extract, Transform, Load) approach evolved into ELT (Extract, Load, Transform), especially with the advent of Data Lakes. The ELT approach allows raw data to be stored first and transformed later, making the system more flexible for handling large volumes of unstructured and varied data.

As the volume and variety of data grew, we faced two major challenges:

· Storage — Storing such large and varied datasets.

· Processing — The need for significant RAM and CPU resources to handle data at scale.

To address these, we had two primary approaches:

Monolithic vs Distributed: Monolithic libraries are designed to run on a single machine whereas distributed libraries are designed to run on multiple machines. Hence monolithic libraries rely on vertical scaling (increasing the capacity of a single machine by adding more CPU, RAM, or storage to the existing machine), whereas distributed libraries rely on horizontal scaling (adding more machines to a system and distributing the load across multiple nodes).

We choose Distributed approach, Due to the nature of Big Data, where data is vast and growing rapidly, a distributed approach was necessary. This approach allows us to spread the data across multiple machines, thus providing:

• Horizontal scalability (adding more machines instead of just upgrading a single machine)
• Efficient data processing without overwhelming a single system’s resources (CPU, RAM)

Previously, we worked with Hadoop, but in the next section, we’ll explore the differences between Hadoop and Spark to understand why Spark became the framework of choice for many.

This is where Apache Spark comes in. Spark is a distributed processing framework that efficiently handles Big Data across a cluster of machines.

Hadoop vs Spark: Understanding the Differences:

Big data technologies have evolved significantly, and two of the most popular frameworks in this space are Hadoop and Spark. While both are used for processing large datasets, they differ in architecture, speed, and usability. Here’s a quick comparison to help you understand the key differences.

Common Misconceptions:

1.Hadoop is a database:
❌ Incorrect: Hadoop is a framework, not a database. It provides distributed storage (HDFS) and processing (MapReduce) but doesn’t support traditional DB operations without extra tools like Hive.

2.Spark is 100x faster than Hadoop:
Spark can be up to 100 times faster than Hadoop MapReduce in memory, but this depends on the use case and workload. Spark achieves this speed through:

• In-memory computation
• DAG (Directed Acyclic Graph) execution engine

However, when Spark has to read/write from disk, the speed advantage drops significantly. So, this “100x” is not a universal truth — it’s more of a best-case scenario.

3. Spark processes data in RAM, Hadoop doesn’t:
Spark is designed to maximize in-memory processing, which is why it’s generally faster.
Hadoop (specifically MapReduce), on the other hand, writes intermediate results to disk between each map and reduce step, which slows things down.

That said:

• Spark can fall back to disk if memory is insufficient.
• Hadoop can use memory too, but not as efficiently or extensively as Spark.

Hadoop vs Spark: Key Differences

🚀 Performance

• Hadoop: Writes intermediate data to disk after every map or reduce task, making it slower.
• Spark: Performs in-memory computation, significantly reducing disk I/O and increasing speed.

📊 Batch vs Streaming

• Hadoop: Primarily built for batch processing.
• Spark: Supports both batch and real-time streaming data processing through Spark Streaming.

💻 Ease of Use

• Hadoop: Requires complex coding, usually in Java. Tools like Hive simplify querying with SQL-like syntax.
• Spark: Developer-friendly with support for interactive shells and APIs in Python, Scala, Java, and R.

🔐 Security

• Hadoop: Uses Kerberos authentication and ACL-based authorization for security.
• Spark: Lacks strong built-in security features, often relying on external tools or integration with secure Hadoop ecosystems.

💥 Fault Tolerance

• Hadoop: Achieves fault tolerance through data replication across nodes in HDFS.
• Spark: Uses Directed Acyclic Graphs (DAGs) to recalculate lost data and ensure resilient execution.

✅ Final Thoughts

Both Hadoop and Spark are powerful, but they serve different purposes.

• If you’re dealing with massive batch jobs and need strong security and mature tooling, Hadoop might be the right fit.
• If speed, flexibility, and real-time processing are critical for your workloads, Spark is likely the better choice.