Map Reduce Understanding

MapReduce execution process

Overall flow:

INPUT DATA
↓
MAP
↓
COMBINER
↓
SHUFFLE
↓
SORT
↓
REDUCER
↓
FINAL OUTPUT

1. MAP — Gives Parallelism

The Map phase processes different pieces/partitions of data in parallel.

INPUT: 
Partition 1 → "apple apple banana"
Partition 2 → "banana apple"

Mapper produces key-value pairs:

Partition 1:
(apple, 1)
(apple, 1)
(banana, 1)

Partition 2:
(banana, 1)
(apple, 1)

Note: The key is the word and the value is 1

Gives parallelism means different partitions can be processed simultaneously:

              INPUT
          /            \
         ↓              ↓
     Mapper 1        Mapper 2
         ↓              ↓
      (apple,1)      (banana,1)
      (apple,1)      (apple,1)
      (banana,1)

2. COMBINER — Local Aggregation

The Combiner performs a local aggregation at the mapper end before data is sent across the network.

For Partition 1:
(apple,1)
(apple,1)
(banana,1)

becomes:
(apple,2)
(banana,1)

Partition 2:
(banana,1)
(apple,1)

becomes:
(banana,1)
(apple,1)

so instead of sending:
apple,1
apple,1
banana,1
banana,1
apple,1

3. SHUFFLE — Sending Mapper Output to Reducers

Now Spark needs to bring all values belonging to the same key together.

We have:
Mapper 1 → (apple,2), (banana,1)
Mapper 2 → (banana,1), (apple,1)

The shuffle redistributes the data:

Apple → (2,1) → Reducer
Banana → (1,1) → Reducer

In case of multiple reducers: Same keys must go to the same reducer.

apple → Reducer 1
apple → Reducer 1

banana → Reducer 2
banana → Reducer 2

4. SORT — Organizing Data on the Reducer Side

After shuffle, the reducer receives data organized by key.

Reducer:

apple  → [2, 1]
banana → [1, 1]

The sorting/grouping ensures that values belonging to the same key are together so that the reducer can process them.

Get Saurabh Soni’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Shuffle → Bring same keys together
Sort → Organize/group them by key

5. REDUCER — Final Aggregation

Now the reducer applies the aggregation function.

apple  → 2 + 1 = 3
banana → 1 + 1 = 2

FINAL OUTPUT:
(apple, 3)
(banana, 2)

With multiple reducers:

             SHUFFLE
                ↓
       ┌────────┴────────┐
       ↓                 ↓
  Reducer 1          Reducer 2
   apple              banana
     ↓                   ↓
  (apple,3)          (banana,2)