-Som Shekhar Sharma

-Navneet Sharma
Mr. Som Shekhar Sharma

Total 5+ years of IT experience

3 years on Big data technologies
Hadoop, HPCC
Expereinced in NoSQL DBs
(HBase, Cassandra, Mongo,
Couch)
Data Aggregator Engine: Apache
Flume
CEP : Esper
Worked on Analytics, retail, e-
commerce, meter data
management systems etc
Cloudera Certified Apache
Hadoop developer
Mr. Navneet Sharma

Total 9+ years of IT experience

Expert on core java and advance java
concepts
1+ year experience on Hadoop and
its ecosystem
Expert on Apache Kafka and Esper
NoSQL DBs (HBase, Cassandra)
Cloudera Certified Apache Hadoop
Developer
In this module you will learn
What is big data?
Challenges in big data
Challenges in traditional Applications
New Requirements
Introducing Hadoop
Brief History of Hadoop
Features of Hadoop
Over view of Hadoop Ecosystems
Overview of MapReduce
Big Data Concepts
Volume
No more GBs of data
TB,PB,EB,ZB

Velocity
High frequency data like in stocks

Variety
Structure and Unstructured data
Challenges In Big Data
Complex
No proper understanding of the underlying data

Storage
How to accommodate large amount of data in single
physical machine

Performance
How to process large amount of data efficiently and
effectively so as to increase the performance
Traditional Applications
D a t a Tr a n s f e r

Network
Application Server

Data Base

D a t a Tr a n s f e r
Challenges in Traditional Application- Part1
Network
We dont get dedicated network line from application to
data base
All the companys traffic has to go through same line

Data
Cannot control the production of data
Bound to increase
Format of the data changes very often
Data base size will increase
Statistics Part1
Assuming N/W bandwidth is 10MBPS

Application Size (MB) Data Size Total Round Trip Time

(sec)
10 10MB 1+1 = 2
10 100MB 10+10=20
10 1000MB = 1GB 100+100 = 200 (~3.33min)
10 1000GB=1TB 100000+100000=~55hour

Calculation is done under ideal condition

No processing time is taken into consideration
Observation
Data is moved back and forth over the low latency
network where application is running
90% of the time is consumed in data transfer

Application size is constant

Conclusion
Achieving Data Localization
Moving the application to the place where the data is
residing OR
Making data local to the application
Challenges in Traditional Application- Part2
Efficiency and performance of any application is
determined by how fast data can be read

Traditionally primary motive was to increase the

processing capacity of the machine
Faster processor
More RAM
Less data and complex computation is done on data
Statistics Part 2

How data is read ?

Line by Line reading
Depends on seek rate and disc latency

Average Data Transfer rate = 75MB/sec

Total Time to read 100GB = 22 min
Total time to read 1TB = 3 hours
How much time you take to sort 1TB of data??
Observation

Large amount of data takes lot of time to read

RAM of the machine also a bottleneck

Summary
Storage is problem
Cannot store large amount of data
Upgrading the hard disk will also not solve the problem
(Hardware limitation)
Performance degradation
Upgrading RAM will not solve the problem (Hardware
limitation)
Reading
Larger data requires larger time to read
Solution Approach
Distributed Framework
Storing the data across several machine
Performing computation parallely across several
machines
Traditional Distributed Systems

Data Transfer
Data Transfer

Bottleneck if number of
users are increased

Data Transfer
Data Transfer
New Approach - Requirements
Supporting Partial failures
Recoverability
Data Availability
Consistency
Data Reliability
Upgrading
Supporting partial failures
Should not shut down entire system if few machines
are down
Should result into graceful degradation of performance
Recoverability
If machines/components fails, task should be taken up
by other working components
Data Availability

Failing of machines/components should not result

into loss of data

System should be highly fault tolerant

Consistency
If machines/components are failing the outcome of
the job should not be affected
Data Reliability
Data integrity and correctness should be at place
Upgrading
Adding more machines should not require full restart
of the system
Should be able to add to existing system gracefully and
participate in job execution
Introducing Hadoop
Distributed framework that provides scaling in :
Storage
Performance
IO Bandwidth
Brief History of Hadoop
What makes Hadoop special?
No high end or expensive systems are required
Built on commodity hardwares
Can run on your machine !

Can run on Linux, Mac OS/X, Windows, Solaris

No discrimination as its written in java

Fault tolerant system

Execution of the job continues even of nodes are failing
It accepts failure as part of system.

28
What makes Hadoop special?
Highly reliable and efficient storage system

In built intelligence to speed up the application

Speculative execution

Fit for lot of applications:

Web log processing
Page Indexing, page ranking
Complex event processing
 Features of Hadoop
Partition, replicate and distributes the data
Data availability, consistency
Performs Computation closer to the data
Data Localization
Performs computation across several hosts
MapReduce framework
Hadoop Components
Hadoop is bundled with two independent components

HDFS (Hadoop Distributed File System)

Designed for scaling in terms of storage and IO bandwidth

MR framework (MapReduce)
Designed for scaling in terms of performance
Overview Of Hadoop Processes
Processes running on Hadoop:
NameNode

DataNode Used by HDFS

Secondary NameNode

Task Tracker
Used by MapReduce
Framework
Job Tracker
Hadoop Process contd
Two masters :
NameNode aka HDFS master
If down cannot access HDFS
Job tracker- aka MapReduce master
If down cannot run MapReduce Job, but still you can access
HDFS
Overview Of HDFS
Overview Of HDFS
NameNode is the single point of contact
Consist of the meta information of the files stored in
HDFS
If it fails, HDFS is inaccessible

DataNodes consist of the actual data

Store the data in blocks
Blocks are stored in local file system
Overview of MapReduce
Overview of MapReduce
MapReduce job consist of two tasks
Map Task
Reduce Task

Blocks of data distributed across several machines are

processed by map tasks parallely

Results are aggregated in the reducer

Works only on KEY/VALUE pair

Does Hadoop solves every one problem?
I am DB guy, I am proficient in writing SQL and trying very hard to
optimize my queries, but still not able to do so. Moreover I am not Java
geek. Will this solve my problem?
Hive

Hey, Hadoop is written in Java, and I am purely from C++ back ground,
how I can use Hadoop for my big data problems?
Hadoop Pipes

I am a statistician and I know only R, how can I write MR jobs in R?

RHIPE (R and Hadoop Integrated Environment)

Well how about Python, Scala, Ruby, etc programmers? Does Hadoop
support all these?
Hadoop Streaming

38
RDBMS and Hadoop
Hadoop is not a data base

RDBMS Should not be compared with Hadoop

RDBMS should be compared with NoSQL or Column

Oriented Data base
Downside of RDBMS
Rigid Schema
Once schema is defined it cannot be changed
Any new additions in the column requires new schema
to be created
Leads to lot of nulls being stored in the data base
Cannot add columns at run time

Row Oriented in nature

Rows and columns are tightly bound together
Firing a simple query select * from myTable where
colName=foo requires to do the full table scanning
NoSQL DataBases
Invert RDBMS upside down

Column family concept (No Rows)

One column family can consist of various columns
New columns can be added at run time
Nulls can be avoided
Schema can be changed

Meta Information helps to locate the data

No table scanning
Challenges in Hadoop-Security
Poor security mechanism

Uses whoami command

Cluster should be behind firewall

Already integrated with Kerberos but very trickier

Challenges in Hadoop- Deployment
Manual installation would be very time consuming
What if you want to run 1000+ Hadoop nodes

Can use puppet scripting

Complex

Can use Cloudera Manager

Free edition allows you to install Hadoop till 50 nodes
Challenges in Hadoop- Maintenance
How to track whether the machines are failed or not?
Need to check what is the reason of failure

Always resort to the log file if something goes wrong

Should not happen that log file size is greater than the
data
Challenges in Hadoop- Debugging
Tasks run on separate JVM on Hadoop cluster
Difficult to debug through Eclipse

Need to run the task on single JVM using local runner

Running application on single node is totally different
than running on cluster
Directory Structure of Hadoop
Hadoop Distribution- hadoop-1.0.3

$HADOOP_HOME (/usr/local/hadoop)

conf bin logs lib

mapred-site.xml start-all.sh All the log files All the 3rd partty
core-site.xml stop-all.sh for all the jar files are
hdfs-site.xml start-dfs.sh, etc corresponding present
masters process will be You will be
slaves created here requiring while
hadoop-env.sh
working with
HDFS API
conf Directory
Place for all the configuration files
All the hadoop related properties needs to go into one
of these files
mapred-site.xml
core-site.xml
hdfs-site.xml
bin directory
Place for all the executable files

You will be running following executable files very

often
start-all.sh ( For starting the Hadoop cluster)
stop-all.sh ( For stopping the Hadoop cluster)
logs Directory
Place for all the logs

Log files will be created for every process running on

Hadoop cluster
NameNode logs
DataNode logs
Secondary NameNode logs
Job Tracker logs
Task Tracker logs
Single Node (Pseudo Mode) Hadoop Set up
Installation Steps
Pre-requisites
Java Installation
Creating dedicated user for hadoop
Password-less ssh key creation
Configuring Hadoop
Starting Hadoop cluster
MapReduce and NameNode UI
Stopping Hadoop Cluster
Note
The installation steps are provided for CentOS 5.5 or
greater

Installation steps are for 32 bit OS

All the commands are marked in blue and are in italics

Assuming a user by name training is present and this

user is performing the installation steps
Note
Hadoop follows master-slave model

There can be only 1 master and several slaves

HDFS-HA more than 1 master can be present

In pseudo mode, Single Node is acting both as master and

slave

Master machine is also referred to as NameNode machine

Slave machines are also referred to as DataNode machine

Pre-requisites
Edit the file /etc/sysconfig/selinux
Change the property of SELINUX from enforcing to
disabled
You need to be root user to perform this operation

Install ssh
yum install open-sshserver open-sshclient
chkconfig sshd on
service sshd start
Installing Java
Download Sun JDK ( >=1.6 ) 32 bit for linux

Download the .tar.gz file

Follow the steps to install Java

tar -zxf jdk.x.x.x.tar.gz
Above command will create a directory from where you ran
the command
Copy the Path of directory (Full Path)
Create an environment variable JAVA_HOME in
.bashrc file ( This file is present inside your users home
directory)
Installing Java contd
Open /home/training/.bashrc file

export JAVA_HOME=PATH_TO_YOUR_JAVA_HOME

export PATH=$JAVA_HOME/bin:$PATH

Close the file and run the command source .bashrc

Verifying Java Installation
Run java -version
This command should show the Sun JDK version
Disabling IPV6
Hadoop works only on ipV4 enabled machine not on
ipV6.

Run the following command to check

cat /proc/sys/net/ipv6/conf/all/disable_ipv6
The value of 0 indicates that ipv6 is disabled

For disabling ipV6, edit the file /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
Configuring SSH (Secure Shell Host)
Nodes in the Hadoop cluster communicates using ssh.
When you do ssh ip_of_machine you need to enter
password
if you are working with 100 + nodes, you need to enter
100 times password.
One option could be creating a common password for all
the machines. (But one needs to enter it manually)

Better option would be to do communication in

password less manner
Security breach
Configuring ssh contd
Run the following command to create a password less key
ssh-keygen -t rsa -P
The above command will create two files under
/home/training/.ssh folder
id_rsa ( private key)
id_rsa.pub (public key)

Copy the contents of public key to a file authorized_keys

cat /home/training/.ssh/id_rsa.pub >> /home/training/.ssh/authorized_keys

Change the permission of the file to 755

chmod 755 /home/training/.ssh/authorized_keys
Verification of passwordless ssh
Run ssh localhost

Above command should not ask you password and you

are in localhost user instead of training user

Run exit to return to training user

Configuring Hadoop
Download Hadoop tar ball and extract it
tar -zxf hadoop-1.0.3.tar.gz
Above command will create a directory which is
Hadoops home directory
Copy the path of the directory and edit .bashrc file

export HADOOP_HOME=/home/training/hadoop-1.0.3

export PATH=$PATH:$HADOOP_HOME/bin
Edit $HADOOP_HOME/conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>

</configuration>
Edit $HADOOP_HOME/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.block.size</name>
<value>67108864</value>
</property>
</configuration>
Edit $HADOOP_HOME/conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/training/hadoop-temp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>

</configuration>
Edit $HADOOP_HOME/conf/hadoop-env.sh

export JAVA_HOME=PATH_TO_YOUR_JDK_DIRECTORY
Note
No need to change your masters and slaves file as you
are installing Hadoop in pseudo mode / single node

See the next module for installing Hadoop in multi

node
Creating HDFS ( Hadoop Distributed File System)
Run the command
hadoop namenode format

This command will create hadoop-temp directory

(check hadoop.tmp.dir property in core-site.xml)
Start Hadoops Processes
Run the command
start-all.sh

The above command will run 5 process

NameNode
DataNode
SecondaryNameNode
JobTracker
TaskTracker

Run jps to view all the Hadoops process

 Viewing NameNode UI
In the browser type localhost:50070
 Viewing MapReduce UI
In the browser type localhost:50030
Hands On
Open the terminal and start the hadoop process
start-all.sh

Run jps command to verify whether all the 5 hadoop

process are running or not

Open your browser and check the namenode and

mapreduce UI

In the NameNode UI, browse your HDFS

Multi Node Hadoop Cluster Set up
Java Installation

Major and Minor version across all the nodes/machines has to be

same
Configuring ssh
authorized_keys file has to be copied into all the
machines

Make sure you can do ssh to all the machines in a

password less manner
Configuring Masters and slaves file
1.1.1.1 (Master) 1.1.1.2 (Slave 0) 1.1.1.3 (Slave 1)

Masters file Slaves file

1.1.1.1 / Master 1.1.1.2 / Slave 0

1.1.1.3 / Slave 1
 Configuring Masters and slaves file-Master acting
as Slave
1.1.1.1 (Master and slave) 1.1.1.1 (Slave 0) 1.1.1.1 (Slave 1)

Masters file Slaves file

1.1.1.1 / Master 1.1.1.2 / Slave 0

1.1.1.3 / Slave 1
1.1.1.1 / Master
Edit $HADOOP_HOME/conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
<name>mapred.job.tracker</name>
<value>IP_OF_MASTER:54311</value>
</property>

</configuration>
Edit $HADOOP_HOME/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>

<property>
<name>dfs.block.size</name>
<value>67108864</value>
</property>
</configuration>
Edit $HADOOP_HOME/conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/training/hadoop-temp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://IP_OF_MASTER:54310</value>
</property>

</configuration>
NOTE
All the configuration files has to be the same across all
the machines

Generally you do the configuration on NameNode

Machine / Master machine and copy all the
configuration files to all the DataNodes / Slave
machines
Start the Hadoop cluster
Run the following command from the master machine
start-all.sh
Notes On Hadoop Process
In pseudo mode or single node set up, all the below 5
process runs on single machine
NameNode
DataNode
Secondary NameNode
JobTracker
Task Tracker
 Hadoop Processs contd- Master not acting as
slave
1.1.1.1 (Master) 1.1.1.1 (Slave 0) 1.1.1.1 (Slave 1)

Process running on
Master / NameNode
machine

NameNode DataNode DataNode

Job Tracker Task Task
Secondary Tracker Tracker
NameNode
 Hadoop Processs contd- Master acting as slave
1.1.1.1 (Master and 1.1.1.1 (Slave 0) 1.1.1.1 (Slave 1)
slave)

Process running on
Master / NameNode
machine
NameNode DataNode DataNode
Job Tracker Task Task
Secondary Tracker Tracker
NameNode
DataNode
TaskTracker
Hadoop Set up in Production
SecondaryNameNode
NameNode JobTracker

DataNode DataNode
TaskTracker TaskTracker
Important Configuration properties
In this module you will learn
fs.default.name
mapred.job.tracker
hadoop.tmp.dir
dfs.block.size
dfs.replication
fs.default.name
Value is hdfs://IP:PORT [hdfs://localhost:54310]

Specifies where is your name node is running

Any one from outside world trying to connect to

Hadoop cluster should know the address of the name
node
mapred.job.tracker
Value is IP:PORT [localhost:54311]

Specifies where is your job tracker is running

When external client tries to run the map reduce job

on Hadoop cluster should know the address of job
tracker
dfs.block.size
Default value is 64MB
File will be broken into 64MB chunks

One of the tuning parameters

Directly proportional to number of mapper tasks

running on Hadoop cluster
dfs.replication
Defines the number of copies to be made for each
block

Replication features achieves fault tolerant in the

Hadoop cluster
Data is not lost even if the machines are going down
OR it achieves Data Availablity
dfs.replication contd
Each replica is stored in different machines
hadoop.tmp.dir
Value of this property is a directory

This directory consist of hadoop file system

information
Consist of meta data image
Consist of blocks etc
Directory Structure of HDFS on Local file system and
Important files
hadoop-temp

dfs mapred

data name namesecondary

current current previou current

s.check
VERSION FILE VERSION point
Blocks and Meta
file
fsimage
edits
NameSpace IDs

Data Node NameSpace ID

NameSpace ID has to be
same
You will get Incompatible
NameNode NameSpace ID NameSpace ID error if
there is mismatch
DataNode will not
come up
NameSpace IDs contd
Every time NameNode is formatted, new namespace id
is allocated to each of the machines (hadoop namenode format)

DataNode namespace id has to be matched with

NameNodes namespace id.

Formatting the namenode will result into creation of

new HDFS and previous data will be lost.
SafeMode
Starting Hadoop is not a single click

When Hadoop starts up it has to do lot of activities

Restoring the previous HDFS state
Waits to get the block reports from all the Data Nodes etc

During this period Hadoop will be in safe mode

It shows only meta data to the user
It is just Read only view of HDFS
Cannot do any file operation
Cannot run MapReduce job
SafeMode contd
For doing any operation on Hadoop SafeMode should
be off

Run the following command to get the status of

safemode
hadoop dfsadmin -safemode get

For maintenance, sometimes administrator turns ON

the safe mode
hadoop dfsadmin -safemode enter
SafeMode contd
Run the following command to turn off the safemode
hadoop dfsadmin -safemode leave
Assignment
Q1. If you have a machine with a hard disk capacity of 100GB
and if you want to store 1TB of data, how many such
machines are required to build a hadoop cluster and store 1
TB of data?

Q2. If you have 10 machines each of size 1TB and you have
utilized the entire capacity of the machine for HDFS? Then
what is the maximum file size you can put on HDFS
1TB
10TB
3TB
4TB
HDFS Shell Commands
In this module you will learn
How to use HDFS Shell commands
Command to list the directories and files
Command to put files on HDFS from local file system
Command to put files from HDFS to local file system
Displaying the contents of file
What is HDFS?
Its a layered or Virtual file system on top of local file
system
Does not modify the underlying file system

Each of the slave machine access data from HDFS as if

they are accessing from their local file system

When putting data on HDFS, it is broken

(dfs.block.size) ,replicated and distributed across
several machines
HDFS contd

Hadoop Distributed File System

 HDFS contd
Behind the scenes when you
put the file
File is broken into blocks
Each block is replicated
BIG FILE
Replicas are stored on local
file system of data nodes

BIG
FILE
Accessing HDFS through command line
Remember it is not regular file system

Normal unix commands like cd,ls,mv etc will not work

For accessing HDFS through command line use

hadoop fs options
options are the various commands
Does not support all the linux commands
Change directory command (cd) is not supported
Cannot open a file in HDFS using VI editor or any other editor
for editing. That means you cannot edit the file residing on
HDFS
HDFS Home Directory
All the operation like creating files or directories are
done in the home directory
You place all your files / folders inside your home
directory of linux (/home/username/)

HDFS home directory is

fs.default.name/user/dev
hdfs://localhost:54310/user/dev/
dev is the user who has created file system
Common Operations
Creating a directory on HDFS
Putting a local file on HDFS
Listing the files and directories
Copying a file from HDFS to local file system
Viewing the contents of file
Listing the blocks comprising of file and their location
Creating directory on HDFS

hadoop fs -mkdir foo

It will create the foo directory under HDFS home

directory
Putting file on HDFS from local file system

hadoop fs -copyFromLocal /home/dev/foo.txt foo

Copies the file from local file system to HDFS home

directory by name foo

Another variation

hadoop fs -put /home/dev/foo.txt bar

Listing the files on HDFS
hadoop fs ls

File Replication File File Size Last Last Absolute name

Mode factor owner group Modified modified of the file or
date time directory

Directories are treated as meta information and is stored by NameNode, not by

DataNode. Thats why the size is also zero
Getting the file to local file system

hadoop fs -copyToLocal foo /home/dev/foo_local

Copying foo from HDFS to local file system

Another variant

hadoop fs -get foo /home/dev/foo_local_1

Viewing the contents of file on console

hadoop fs -cat foo

Getting the blocks and its locations

hadoop fsck /user/training/sample -files -blocks -locations

This will give the list of blocks which the file sample
is made of and its location

Blocks are present under dfs/data/current folder

Check hadoop.tmp.dir
Hands On
Please refer the Hands On document
HDFS Components
In this module you will learn
Various process running on Hadoop
Working of NameNode
Working of DataNode
Working of Secondary NameNode
Working of JobTracker
Working of TaskTracker
Writing a file on HDFS
Reading a file from HDFS
Process Running on Hadoop
NameNode
Secondary NameNode Runs on Master/NN
Job Tracker

DataNode
TaskTracker Runs on Slaves/DN
How HDFS is designed?
For Storing very large files - Scaling in terms of storage

Mitigating Hardware failure

Failure is common rather than exception

Designed for batch mode processing

Write once read many times

Build around commodity hardware

Compatible across several OS (linux, windows, Solaris, mac)

Highly fault tolerant system

Achieved through replica mechanism
Storing Large Files
Large files means size of file is greater than block size ( 64MB)

Hadoop is efficient for storing large files but not efficient for storing
small files
Small files means the file size is less than the block size

Its better to store a bigger files rather than smaller files

Still if you are working with smaller files, merge smaller files to make a big
file

Smaller files have direct impact on MetaData and number of task

Increases the NameNode meta data
Lot of smaller files means lot of tasks.
Scaling in terms of storage

Want to increase the storage Capacity of existing Hadoop cluster?

Scaling in terms of storage

Add one more node to the existing cluster?

Mitigating Hard Ware failure
HDFS is machine agnostic

Make sure that data is not lost if the machines are

going down
By replicating the blocks
Mitigating Hard Ware failure contd

Replica of the block is present in three machines ( Default replication

factor is 3)

Machine can fail due to numerous reasons

Faulty Machine
N/W failure, etc
Mitigating Hard Ware failure contd

Replica is still intact in some other machine

NameNode try to recover the lost blocks from the failed machine and
bring back the replication factor to normal ( More on this later)
Batch Mode Processing

BIG FILE
Client is putting file
on Hadoop cluster

BIG
FILE
Batch Mode Processing contd

BIG FILE
Client wants to
analyze this file

BIG
FILE
Batch Mode Processing contd

BIG FILE
Client wants to
analyze this file

BIG Even though the file is partitioned but client does

FILE not have the flexibility to access and analyze the
individual blocks
Client always see the big file
HDFS is virtual file system which gives the
holistic view of data
High level Overview
NameNode
Single point of contact to the outside world
Client should know where the name node is running
Specified by the property fs.default.name

Stores the meta data

List of files and directories
Blocks and their locations

For fast access the meta data is kept in RAM

Meta data is also stored persistently on local file system
/home/training/hadoop-temp/dfs/name/previous.checkpoint/fsimage
NameNode contd
Meta Data consist of mapping
Of files to the block

Data is stored with

Data nodes
NameNode contd
If NameNode is down, HDFS is inaccessible
Single point of failure

Any operation on HDFS is recorded by the NameNode

/home/training/hadoop-temp/dfs/name/previous.checkpoint/edits file

Name Node periodically receives the heart beat signal

from the data nodes
If NameNode does not receives heart beat signal, it
assumes the data node is down
NameNode asks other alive data nodes to replicate the
lost blocks
DataNode
Stores the actual data
Along with data also keeps a meta file for verifying the
integrity of the data
/home/training/hadoop-temp/dfs/data/current

Sends heart beat signal to NameNode periodically

Sends block report
Storage capacity
Number of data transfers happening
Will be considered down if not able to send the heart beat

NameNode never contacts DataNodes directly

Replies to heart beat signal
DataNode contd
Data Node receives following instructions from the
name node as part of heart beat signal
Replicate the blocks (In case of under replicated blocks)
Remove the local block replica (In case of over
replicated blocks)

NameNode makes sure that replication factor is always

kept to normal (default 3)
 Recap- How a file is stored on HDFS
Behind the scenes when you
put the file
File is broken into blocks
Each block is replicated
BIG FILE
Replicas are stored on local
file system of data nodes

BIG
FILE
Normal Replication factor

Number of replicas per bl0ck = 3

BIG
FILE
Under Replicated blocks

One Machine is down

Number of replicas of a block = 2
BIG
FILE
Under Replicated blocks contd

Ask one of the data nodes to

replicate the lost block so as to make
BIG the replication factor to normal
FILE
Over Replicated Blocks

The lost data node comes up

Total Replica of the block = 4
BIG
FILE
Over Replicated Blocks contd

Ask one of the data node to

remove local block replica
BIG
FILE
Secondary NameNode contd
Not a back up or stand by NameNode

Only purpose is to take the snapshot of NameNode

and merging the log file contents into metadata file on
local file system

Its a CPU intensive operation

In big cluster it is run on different machine
Secondary NameNode contd
Two important files (present under this directory
/home/training/hadoop-temp/dfs/name/previous.checkpoint )
Edits file
Fsimage file

When starting the Hadoop cluster (start-all.sh)

Restores the previous state of HDFS by reading fsimage
file
Then starts applying modifications to the meta data
from the edits file
Once the modification is done, it empties the edits file
This process is done only during start up
Secondary NameNode contd
Over a period of time edits file can become very big
and the next start become very longer

Secondary NameNode merges the edits file contents

periodically with fsimage file to keep the edits file size
within a sizeable limit
Job Tracker
MapReduce master
Client submits the job to JobTracker
JobTracker talks to the NameNode to get the list of
blocks
Job Tracker locates the task tracker on the machine
where data is located
Data Localization
Job Tracker then first schedules the mapper tasks
Once all the mapper tasks are over it runs the reducer
tasks
Task Tracker
Responsible for running tasks (map or reduce tasks)
delegated by job tracker

For every task separate JVM process is spawned

Periodically sends heart beat signal to inform job

tracker
Regarding the available number of slots
Status of running tasks
Writing a file to HDFS
Reading a file from HDFS
Connects to NN

Ask NN to give the list of data nodes that is hosting the

replicas of the block of file

Client then directly read from the data nodes without

contacting again to NN

Along with the data, check sum is also shipped for verifying
the data integrity.
If the replica is corrupt client intimates NN, and try to get the data
from other DN
HDFS API
Accessing the file system

Require 3 things:
Configuration object
Path object
FileSystem instance
Hadoops Configuration
Encapsulates client and server configuration
Use Configuration class to access the file system.
Configuration object requires how you want to access
the Hadoop cluster
Local File System
Pseudo Mode
Cluster Mode
Hadoops Path
File on HDFS is represented using Hadoops Path
object
Path is similar to HDFS URI such as
hdfs://localhost:54310/user/dev/sample.txt
FileSystem API
General file system Api
public static FileSystem get(Configuration conf) throws IOException
public static FileSystem get(URI uri, Configuration conf) throws
IOException
Accessing the file system contd..

Step1 : Create a new configuration object

Configuration conf = new Configuration();

Step2 : Setting the name node path

conf.set(fs.default.name,hdfs://localhost:54310);

Step 3: Get the filesystem instance

FileSystem fs = FileSystem.get(conf);
Accessing the file system contd..

Step1 : Create a new configuration object

Configuration conf = new Configuration();

Step2 : Setting the name node path

conf.set(fs.default.name,hdfs://localhost:54310);
Create a configuration object
Using this configuration object, you will connect to the Hadoop
Step 3: Getcluster
the filesystem instance
FileSystem fs =need
You FileSystem.get(conf);
to tell this configuration object where is your NameNode
and Job Tracker running
Accessing the file system contd..

Step1 : Create a new configuration object

Configuration conf = new Configuration();

Step2 : Setting the name node path

conf.set(fs.default.name,hdfs://localhost:54310);

Step 3: Get the filesystem instance

Settingfsthe
FileSystem property fs.default.name, which tells the address of the
= FileSystem.get(conf);
name node
Accessing the file system contd..

Step1 : Create a new configuration object

Configuration conf = new Configuration();

Step2 : Setting the name node path

conf.set(fs.default.name,hdfs://localhost:54310);

Step 3: Get the filesystem instance

FileSystem fs = FileSystem.get(conf);

Finally getting the file system instance

Once you get the file system you can do any kind of file
operations on HDFS
Hands On
Refer Hands-on document
Module 1
In this module you will learn
What is MapReduce job?
What is input split?
What is mapper?
What is reducer?
What is MapReduce job?
Its a framework for processing the data residing on HDFS
Distributes the task (map/reduce) across several machines
Consist of typically 5 phases:
Map
Partitioning
Sorting
Shuffling
Reduce
A single map task works typically on one block of data
(dfs.block.size)
No of blocks / input split = No of map tasks
After all map tasks are completed the output from the map
is passed on to the machines where reduce task will run
MapReduce Terminology
What is job?
Complete execution of mapper and reducers over the entire data set
What is task?
Single unit of execution (map or reduce task)
Map task executes typically over a block of data (dfs.block.size)
Reduce task works on mapper output
What is task attempt?
Instance of an attempt to execute a task (map or reduce task)
If task is failed working on particular portion of data, another task
will run on that portion of data on that machine itself
If a task fails 4 times, then the task is marked as failed and entire
job fails
Make sure that atleast one attempt of task is run on different
machine
Terminology continued
How many tasks can run on portion of data?
Maximum 4
If speculative execution is ON, more task will run
What is failed task?
Task can be failed due to exception, machine failure etc.
A failed task will be re-attempted again (4 times)
What is killed task?
If task fails 4 times, then task is killed and entire job fails.
Task which runs as part of speculative execution will also be
marked as killed
Input Split
Portion or chunk of data on which mapper operates
Input split is just a reference to the data
Typically input size is equal to one block of data
(dfs.block.size)
128MB

64MB 64MB

Here there are 2 blocks, so there will be two input split

Input Split contd
Each mapper works only on one input split
Input split size can be controlled.
Useful for performance improvement
Generally input split is equal to block size (64MB)
What if you want mapper to work only on 32 MB of a
block data?
Controlled by 3 properties:
Mapred.min.split.size( default 1)
Mapred.max.split.size (default LONG.MAX_VALUE)
Dfs.block.size ( default 64 MB)
Input Split contd
Max(minSplitSize,min(maxSplitSize,blockSize)

Min Split Max Split block size Split size

Size Size taken
1 LONG.MAX_ 64 64
VALUE
1 ---- 128 128
128 ---- 128 128
1 32 64 32
What is Mapper?

Mapper is the first phase of MapReduce job

Works typically on one block of data (dfs.block.size)

MapReduce framework ensures that map task is run

closer to the data to avoid network traffic
Several map tasks runs parallel on different machines
and each working on different portion (block) of data

Mapper reads key/value pairs and emits key/value pair

Mapper contd

Map (in_key,in_val) ------- out_key,out_val

Mapper can use or can ignore the input key (in_key)

Mapper can emit
Zero key value pair
1 key value pair
n key value pair
Mapper contd

Map function is called for one record at a time

Input Split consist of records
For each record in the input split, map function will be
called
Each record will be sent as key value pair to map
function
So when writing map function keep ONLY one record in
mind
It does not keep the state of whether how many records it has
processed or how many records will appear
Knows only current record
What is reducer?
Reducer runs when all the mapper tasks are completed
After mapper phase , all the intermediate values for a given
intermediate keys is grouped together and form a list

KEY ,(Val1,Val2,Val3.Valn)

This list is given to the reducer

Reducer operates on Key, and List of Values
When writing reducer keep ONLY one key and its list of value
in mind
Reduce operates ONLY on one key and its list of values at a
time
Reducer contd
NOTE all the values for a particular intermediate key
goes to one reducer

There can be Zero, one or n reducer.

For better load balancing you should have more than
one reducer
Module 2
In this module you will learn
Hadoop primitive data types
What are the various input formats, and what kind of
key values they provide to the mapper function
Seeing TextInputFormat in detail
How input split is processed by the mapper?
Understanding the flow of word count problem
Hadoop Primitive Data types
Hadoop has its own set of primitive data types for
representing its variables
For efficient serialization over the network
While implementing mapper and reducer functionality
you need to emit/write ONLY Hadoop Primitive data
types OR your custom class extending from Writable or
WritableComparable interface( More on this later)
Hadoop Primitive Data types contd
Java Primitive(Box) Data types Hadoop Primitive (Box) Data Types

Integer IntWritable

Long LongWritable

Float FloatWritable

Byte ByteWritable

String Text

Double DoubleWritable
Input Format
Before running the job on the data residing on HDFS,
you need to tell what kind of data it is?
Is data is textual data?
Is data is binary data?

Specify InputFormat of the data

While writing mapper function, you should keep input
format of data in mind, since input format will provide
input key value pair to map function
Input Format contd
Base class is FileInputFormat
Input Format Key Value

Text Input Format Offset of the line within a Entire line till \n as
file value

Key Value Text Input Part of the record till the Remaining record after
Format first delimiter the first delimiter

Sequence File Input Key needs to be Value needs to be

Format determined from the determined from the
header header
Input Format contd
Input Format Key Data Type Value Data Type

Text Input Format LongWritable Text

Key Value Text Input Text Text

Format

Sequence File Input ByteWritable ByteWritable

Format
Text Input Format
Efficient for processing text data

Example:
Text Input Format contd
Internally every line is associated with an offset

This offset is treated as key. The first column is offset

For simplicity line number are given
Text Input Format contd

Key Value
0 Hello, how are you?
1 Hey I am fine?How about you?
2 This is plain text
3 I will be using Text Input Format
How Input Split is processed by mapper?
Input split by default is the block size (dfs.block.size)

Each input split / block comprises of records

A record is one line in the input split terminated by \n (new
line character)

Every input format has RecordReader

RecordReader reads the records from the input split
RecordReader reads ONE record at a time and call the map
function.
If the input split has 4 records, 4 times map function will be called,
one for each record
It sends the record to the map function in key value pair
Word Count Problem
Counting the number of times a word has appeared in
a file
Example:
Word Count Problem contd

Output of the Word Count problem

Key Value
Hello 2
you 2
I 2
Word Count Mapper
Assuming one input split

Input format is Text Input Format

Key = Offset which is of type LongWritable
Value = Entire Line as Text
Remember map function will be called 3times
Since there are only 3 records in this input split
Word Count Mapper contd
Map function for this record

(Hello,1)
(how,1)
Map(1,Hello, how are you?) ===> (are,1)
(you,1)
Input to the map function

Intermediate key value pairs from

the mapper
Word Count Mapper contd
Map function for this record

(Hello,1)
Map(2, Hello, I am fine? How about you?)====> (I,1)
(am,1)
Input to the mapper :
:
:

Intermediate key value pair from the

mapper
Word Count Mapper contd

Pseudo Code

Map (inputKey,InputValue)
{
Break the inputValue into individual words;
For each word in the individual words
{
write (word ,1)
}
}
Word Count Reducer
Reducer will receive the intermediate key and its list of
values

If a word Hello has been emitted 4 times in the map

phase, then input to the reducer will be
(Hello,{1,1,1,1})

To count the number of times the word Hello has

appeared in the file, just add the number of 1s
Word Count Reducer contd
Pseudo Code

Reduce (inputKey, listOfValues)

{
sum = 0;
for each value in the listOfValues
{
sum = sum + value;
}
write(inputKey,sum)
}
MapReduce Flow-1

Number of Input Splits = No . of mappers

Observe same key can be generated from

different mapper running on different
machine

But when reducers runs, the keys and its

intermediate values are grouped and fed to
the reducer in sorted order of keys
MapReduce Flow-2
MapReduce Flow - 3
MapReduce Flow- Word Count
Input Split 1 Input Split 2

Hello Hello
M World World
A
P
P
E (Hello ,1) (Hello ,1)
R
(World,1) (World,1)

Hello (1,1) Partitioning,

World (1,1) Shuffling,
Sorting

Reducer Hello = 1+1 = 2

World = 1 +1 = 2
 MapReduce Data Flow
Map 1 Map 2 Map 3

Partitioni Partitioning
ng and Partitioning and Sorting
Sorting on and Sorting on IKV
IKV on IKV

shuffling

Reduce 1

Sorting
Grouping
Important feature of map reduce job
Intermediate output key and value generated by the
map phase is stored on local file system of the machine
where it is running
Intermediate key-value is stored as sequence file (Binary
key value pairs)

Map tasks always runs on the machine where the data

is present
For data localization
For reducing data transfer over the network
Features contd
After the map phase is completed, the intermediate
key-value pairs are copied to the local file system of
the machine where reducers will run
If only ONE reducer is running, then all the
intermediate key-value pairs will be copied to the
machine
Reducer can be very slow if there are large number of
output key value pairs are generated
So its better to run more than one reducer for load balancing
If more than one reducers are running, PARTITIONER decides
which intermediate key value pair should go to which reducer
Features contd

Data localization is not applicable for reducer

Intermediate key-value pairs are copied

Keys and its list of values are always given to the

reducer in SORTED order with respect to key

SORTING on keys happens both after the mapper

phase and before the reducer phase
Sorting at mapper phase is just an optimization
Module 3
In this module you will learn
How will you write mapper class?
How will you write reducer class?
How will you write driver class?
How to use ToolRunner?
Launching your MapReduce job
Writing Mapper Class
public class WordCountMapper extends Mapper<LongWritable, Text, Text,
IntWritable>{

@Override
public void map(LongWritable inputKey,Text inputVal,Context context)
{
String line = inputVal.toString();
String[] splits = line.split(\\W+");
for(String outputKey:splits) {
context.write(new Text(outputKey), new IntWritable(1));
}
}

}
Writing Mapper Class
public class WordCountMapper extends Mapper<LongWritable, Text, Text,
IntWritable>{

@Override
Your Mapper class should extend from Mapper class
public void map(LongWritable inputKey,Text inputVal,Context context)
{
Mapper<LongWritable,Text,Text,IntWritable>
String line = value.toString();
First splits
String[] TypeDef : Input key Type given by input format you use
= line.split("//W+");
for(String outputKey:splits)
{ Second TypeDef: Input value Type given by input format
output.write(new Text(outputKey), new IntWritable(1));
} Third TypeDef: Output key Type which you emit from mapper
} Fourth TypeDef: Output value Type which you emit from mapper

}
Writing Mapper Class
public class WordCountMapper extends Mapper<LongWritable, Text, Text,
IntWritable>{

@Override
public void map(LongWritable inputKey,Text inputVal,Context context)
{

Override the map function

First argument: Input key to your mapper

Second argument: Input value to your mapper

Third argument: Using this context object you will emit output key
value pair
 Writing Mapper Class
public class WordCountMapper extends Mapper<LongWritable, Text, Text,
IntWritable>{

@Override
public void map(LongWritable inputKey,Text value,Context context)
{
String line = value.toString();
String[] splits = line.split(\\W+");
for(String outputKey : splits) {
context.write(new Text(outputKey), new IntWritable(1));
}
}
Step 1: Take the String object from the input value
}

Step 2:Splitting the string object obtained in step 1, split them into individual
words and take them in array

Step 3: Iterate through each words in the array and emit individual word as
key and emit value as 1, which is of type IntWritable
Writing Reducer Class
public class WordCountReducer extend Reducer <Text, IntWritable, Text,
IntWritable > {

public void reduce(Text key, Iterable<IntWritable> values, Context output)

throws IOException, InterruptedException {

int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}

output.write(key, new IntWritable(sum));

}
}
Writing Reducer Class
public class WordCountReducer extends Reducer< Text, IntWritable, Text,
IntWritable> {

Your
public void Reducer class
reduce(Text key,should extend from values,
Iterable<IntWritable> Reducer classoutput) throws IOException,
Context
InterruptedException {
Reducer<Text,IntWritable,Text,IntWritable>
int sum = 0;
First TypeDef : Input key Type given by the output key of map
output
for (IntWritable val : values) {
sum += val.get();
}
Second TypeDef: Input value Type given by output value of map
outputnew IntWritable(sum));
output.write(key,

}
}
Third TypeDef: Output key Type which you emit from reducer

Fourth TypeDef: Output value Type which you emit from reducer
Writing Reducer Class
public class WordCountReducer extends Reducer <Text, IntWritable, Text,
IntWritable> {

public void reduce(Text key, Iterable<IntWritable> values, Context output)

throws IOException, InterruptedException {
int sum = 0;

for(IntWritable
Reducer will get key{ and
val : values) list of values
sum += val.get();
}
Example: Hello {1,1,1,1,1,1,1,1,1,1}
output.write(key, new IntWritable(sum));

}
}
Writing Reducer Class
public class WordCountReducer extend Reducer <Text, IntWritable, Text,
IntWritable > {

public void reduce(Text key, Iterable<IntWritable> values, Context output)

throws IOException, InterruptedException {

int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}

output.write(key, new IntWritable(sum));

Iterate through list of values and
} add the values
}
Writing Driver Class
Step1 :Get the configuration object, which tells you where the namenode and job tracker are
running

Step2 :Create the job object

Step3: Specify the input format. by default it takes the TextInputFormat

Step4: Set Mapper and Reducer class

Step5:Specify the mapper o/p key and o/pvalue class

i/p key and value to mapper is determined by the input format. So NO need to specify

Step6: Specify the reducer o/p key and o/p value class
i/p key and value to reducer is determined by the map o/p key and map o/p value class
respectively. So NO need to specify

Step7: Provide the input and output paths

Step8: Submit the job

Driver Class contd

Job job = new Job(getConf(),"Basic Word Count Job");

job.setJarByClass(WordCountDriver.class);

job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);

job.setInputFormatClass(TextInputFormat.class);

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
Driver Class contd

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

job.waitForCompletion(true);
Usage of Tool Runner
Allows to specify the configuration options from the
command line
Can specify distributed cache properties
Can be used for tuning map reduce jobs
Flexibility to increase or decrease the reducer tasks
without changing the code
Can run the existing map reduce job to utilize local file
system instead of HDFS

It internally uses GenericOptionParser class

Tool Runner contd
For using Tool Runner your driver class should
extends from Configured and implements Tool

public class MyDriver extends Configured implements Tool

{

}
Tool Runner contd
After the previous step, you have to override the run
method
The run method is where actual driver code goes

The main function should call ToolRunner.run

method to invoke the driver

Job Configuration and everything should go in run

method
Tool Runner contd
public class MyDriver extends Configured implements Tool
{
public static void main(String[] args)
{
int exitCode = ToolRunner.run(new MyDriver(), args);
}
Note the getConf() inside the job
@Override object
public int run(String[] args) It gets the default configuration
from the classpath
{
//Your actual driver code goes here
Job job = new Job(getConf(),Basic Word Count Job);
}
}
Specifying Command Line Option using Tool
Runner
Use D flag for specifying properties
Properties could be Hadoop configuration properties
like mapred.reduce.tasks,
fs.default.name,mapred.job.tracker
Properties could be your custom key value pair which
you would like to process in your mapper (More on this
later)
Overrides the default properties in the configurtion
Doesnt change the properties set in the driver code
 Specifying command line Running on Local file system

hadoop jar WordCount.jar WordCountDriver -D fs.default.name=file:/// -D

mapred.job.tracker=local <local_file_path> <output path>
Specifying command line options contd
Example: Changing the reducer tasks

hadoop jar WordCount.jar WordCountDriver -D mapred.reduce.tasks=10

inputPath outputPath

Notice the space between D and property name

Beware if in the driver code if you have used

job.setNumReduceTasks(1) , running the above
command will still run ONLY 1 reducer tasks
Specifying command line options contd
Example: Running MapReduce job on local file system

hadoop jar WordCount.jar WordCountDriver D fs.default.name=file:/// -D

mapred.job.tracker=local inputPath outputPath

Another way of running on local file system

hadoop jar WordCount.jar WordCountDriver fs=file:/// -jt=local

inputPath outputPath
Specifying command line options contd
List of available options

-D property=value

-conf fileName [For overriding the default properties]

-fs uri [D fs.default.name=<NameNode URI> ]

-jt host:port [ -D mapred.job.tracker=<JobTracker URI>]

-files file1,file2 [ Used for distributed cache ]

-libjars jar1, jar2, jar3 [Used for distributed cache]

-archives archive1, archive2 [Used for distributed cache]

Launching the MapReduce Job
Create the jar file
Either from eclipse or from command line

Run the following command to launch

hadoop jar <jarName.jar> <DriverClassName> <input> <output>

<input> could be a file or directory consisting of files on HDFS

<output> Name should be different for every run.
If the <output> directory with the same name is present,
exception will be thrown
Its a directory which will be created on HDFS.
Output file name :part-r-00000 OR part-m-00000
Demo for map only jobs showing intermediate output
Hands ON
Refer the hands-on document
Older and newer API
Notes
All the programs in this training uses Newer API
Module 4
In this module you will learn
Word Co-Occurence problem
Average Word Length Problem
Inverted Index problem
Searching
Sorting
Hands on
Word Co-Occurrence Problem
Measuring the frequency with which two words
appearing in a set of documents

Used for recommendation like

You might like this also
People who choose this also choose that
Examples:
Shopping recommendations
Identifying people of interest
Similar to word count but two words at a time
Inverted Index Problem
Used for faster look up
Example: Indexing done for keywords in a book

Problem statement:
From a list of files or documents map the words to the
list of files in which it has appeared

Output
word = List of documents in which this word has
appeared
Indexing Problem contd
Output from Final Output
File A the mapper
This is cat This: File A
Big fat hen is:File A
cat:File A
Big:File A
fat:File A This:File A,File B
hen:File A
is:File A,File B
cat:File A
fat: File A,File B
File B
This is dog This:File B
My dog is fat is:File B
dog:File B
My: File B
dog:File B
is:File B
fat: File B
Indexing problem contd
Mapper
For each word in the line, emit(word,file_name)

Reducer
Remember for word , all the file names list will be
coming to the reducer
Emit(word,file_name_list)
Average Word Length Problem
Consider the record in a file

Problem Statement
Calculate the average word length for each character
Output:
Character Average word length
H (3+3+5) /3 = 3.66
I 2/1 = 2
T 5/1 =5
D 4/1 = 4
Average Word Length contd
Mapper
For each word in a line,
emit(firstCharacterOfWord,lengthOfWord)

Reducer
You will get a character as key and list of values
corresponding to length
For each value in listOfValue
Calculate the sum and also count the number of values
Emit(character,sum/count)
Hands On
Module 5
In this module you will learn

What is combiner?
setup/cleanup method in mapper / reducer
Passing the parameters to mapper and reducer
Distributed cache
Counters
Hands On
Combiner
Large number of mapper running will produce large
amounts of intermediate data
This data needs to be passed to the reducer over the
network
Lot of network traffic
Shuffling/Copying the mapper output to the machine
where reducer will run will take lot of time
Combiner contd
Similar to reducer
Runs on the same machine as the mapper task
Runs the reducer code on the intermediate output of the
mapper
Thus minimizing the intermediate key-value pairs
Combiner runs on intermediate output of each mapper

Advantages
Minimize the data transfer across the network
Speed up the execution
Reduces the burden on reducer
Combiner contd

Combiner has the same signature as reducer class

Can make the existing reducer to run as combiner, if
The operation is associative or commutative in nature
Example: Sum, Multiplication
Average operation cannot be used

Combiner may or may not run. Depends on the

framework
It may run more than once on the mapper machine
Combiner contd
Combiner contd
In the driver class specify
Job.setCombinerClass(MyReducer.class);
Important things about Combiner
Framework decides whether to run the combiner or
not

Combiner may run more than once on the same

mapper
Depends on two properties io.sort.factor and io.sort.mb
Setup/cleanup Method in Mapper and Reducer
3 functions can be overridden by your map/reduce class
setup method (Optional)
map/reduce method
cleanup method (Optional)

setup method is called only once before calling the map

function
If anything extra (parameter, files etc) is required while
processing data in map/reduce task, it should be initialized or
kept in memory in the setup method
Can be done in map function, but mapper will be called for
every record
setup/cleanup method contd
clean up method is called after the map/reduce
function is over
Can do the cleaning like, if you have opened some file in
the setup method, you can close the file
Setup/cleanup method contd
public class WordCountMapper extends Mapper <LongWritable, Text, Text,
Intwritable> {

String searchWord;
public void setup(Context context) {
searchWord = context.getConfiguration().get(WORD);
}

public void map(LongWritable inputKey,Text inputVal, Context context) {

//mapper logic goes here
}

public void cleanup() {

//clean up the things
}

}
Setup/cleanup method contd

public class WordCountMapper extends Mapper<LongWritable,Text,TextIntwritable>

{
String searchWord;
public void setup(Context context)
{
searchWord = context.getConfiguration().get(WORD);
}

public void map(LongWritable inputKey,Text inputVal, Context contex)

{ You can send the value of the property WORD from command
//mapper logic goes here
line as follows
} hadoop jar myJar MyDriver D WORD=hello
In the setup method you can access the value of the property
public void cleanup()
{
Setup/cleanup method contd

public class WordCountMapper extends Mapper<LongWritable,Text,TextIntwritable>

{ Once you get the searchWord in the setup method. You can use this
String
in searchWord;
your mapper
public void setup(Context context)
{
searchWord = context.getConfiguration().get(WORD);
}
public void map(LongWritable inputKey,Text inputVal, Context contex)
{
//mapper logic goes here
if(searchWord.equals(inputVal.toString())
{
//do something
}
}
Using Distributed Cache
Use Case:
Your map reduce application requires extra information
or data while executing the tasks (map or reduce )
For example list of stop words

While running map task on a data, you would like to

remove the stop words
You require list of stop words

For joining two data sets in mapper (Map Side Join.

More on this later)
Distributed Cache contd
One option is to read the data in setup method
You can put the file on HDFS and
Reading from HDFS in within 10o mapper could be very slow
Not scalable
Would be efficient if the files are read from local file system

DistributedCache facility provides caching of the files

(text,jar,zip,tar.gz,tgz etc) per JOB

MR framework will copy the files to the slave nodes on its local
file system before executing the task on that node
After the task is completed the file is removed from the local file
system.
Distributed Cache contd
Following can be cached
Text data
Jar files
.zip files
.tar.gz files
.tgz files
Distributed Cache contd
First Option: From the driver class
Second Option: From the command line
Using Distributed Cache-First Option

Configuring the distributed cache from the driver class

Place the files on HDFS

You cannot cache files present on local file system

This option requires your file has to be present on HDFS
Configuring Distributed Cache

Job job = new Job();

DistributedCache.addCacheFile(new URI("/myapp/lookup.dat"), job);
DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);
DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);
DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);
DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);
Configuring Distributed Cache contd
In setup of mapper/reducer , you can read the file

public class WordCountDistributedCacheReducer extends Reducer

<Text, IntWritable, Text, IntWritable> {

private URI[] files;

HashMap<String,Integer> weightedAverages = new
HashMap<String,Integer>();

@Override
public void setup(Context context) throws IOException {
this.files = DistributedCache.
getCacheFiles(context.getConfiguration());
Path path = new Path(files[0]);
//do something
}
Using Distributed Cache-Second Option
You can send the files from the command line
Your driver should have implemented ToolRunner

hadoop jar MyJar.jar MyDriver files file1,file2,file3

-files option is for text data

-libjars option is used for adding third party jar
-archives option is used for tar, tar.gz. Tgz file
These files automatically gets unarchived on the machines
where the tasks will run.
Advantage of second option
Files need not to be present on HDFS
Can cache local file

Once File is copied to the required machine

Read the file as if it is normal file
Counters
Counters provides a way for mapper and reducer to pass
aggregate values back to the driver code after the job has finished
Example: You can use counter to count the number of invalid and
valid ( good and bad) records

Counters are like name and value pair

Value can be incremented with in the code

Counters are collected into groups (enum)

For example: Lets say we have group of counters by name

RecordType
Names : validRecord, invalidRecord
Appropriate counter will be incremented as each record is
read
Counters Contd
Counters are also available from the job trackers web
UI
Hands On
Module 6
In this module you will learn
How to create your own custom keys and values?
How to create your own custom partitioner?
How to write custom input format?
Implementing custom keys and values
Working with Hadoop primitive data types does not
offer much flexibility over grouping and sorting
Custom keys/values will be useful for forming
composite keys or complex data structure

Example:
Data has 3 fields only lastName,firstName,and empId
You would like to group the data by lastName and
sorting should be done on both lastName and firstName
Writable and WritableComparable
Hadoop uses its own serialization mechanism for
transferring the intermediate data over the network
Fast and compact
Hadoop does not use Java serialization

Implement Writable interface for custom values

Implement WritableComparable interface for custom

keys
Since the keys will be compared during sorting phase, so
provide the implementation for compareTo() method
Implementing Custom Values
public class PointWritable implement Writable {
private IntWritable xcoord;//x coordinate
private IntWritable ycoord;//y coordinate

//Provide necessary constructors and getters and setters

@Override
public void readFields(DataInput in) throws IOException {
xcoord.readFields(in);
ycoord.readFields(in);
}
@Override
public void write(DataOutput out) throws IOException {
xcoord.write(out);
ycoord.write(out);
}
}
Implementing Custom Values contd
public class PointWritable implements Writable {
private IntWritable xcoord;//x coordinate
private IntWritable ycoord;//y coordinate

//Provide necessary constructors and getters and setters

@Override
Your
public voidcustom value class should
readFields(DataInput implement
in) throws Writable
IOException { interface
xcoord.readFields(in);
ycoord.readFields(in);
Provide the necessary constructors to initialize the member
} variables
@Override
public void write(DataOutput out) throws IOException {
Provide setters and getters method for member variables
xcoord.write(out);
ycoord.write(out);
Provide equals() and hashCode() method
}

}
Implementing Custom Values contd
public class PointWritable implements Writable {
private IntWritable xcoord;//x coordinate
private IntWritable ycoord;//y coordinate

//Provide necessary constructors and getters and setters

@Override
public void readFields(DataInput in) throws IOException {
xcoord.readFields(in);
ycoord.readFields(in);
} Read the fields in the same order you have defined
@Override
public void write(DataOutput out) throws IOException {
xcoord.write(out);
ycoord.write(out);
}
}
Implementing Custom Values contd
public class PointWritable implements Writable {
private IntWritable xcoord;//x coordinate
private IntWritable ycoord;//y coordinate

//Provide necessary constructors to initialize the member variables

@Override
public void readFields(DataInput in) throws IOException {
xcoord.readFields(in);
Write the fields in the same order you have defined
ycoord.readFields(in);
}
@Override
public void write(DataOutput out) throws IOException {
xcoord.write(out);
ycoord.write(out);

}
}
Implementing Custom Keys
public class Person implements WritableComparable<Person>
{
private Text lastName;
private Text firstName;

@Override
public void readFields(DataInput in) throws IOException {
lastName.readFields(in);
firstName.readFields(in);
}

@Override
public void write(DataOutput out) throws IOException {
lastName.write(out);
firstName.write(out);
}
Implementing Custom Keys contd

@Override
public int compareTo(Person other) {

int cmp = lastName.compareTo(other.getLastName());

if(cmp != 0) {
return cmp;
}

return firstName.compareTo(other.getFirstName());
}

}
Implementing Custom Keys contd
public class Person implements WritableComparable<Person> {
private Text lastName;
private Text firstName;

@Override
public void readFields(DataInput in) throws IOException {
.
}

@Override
public void write(DataOutput out) throws IOException {

}

Writable Comparable interface extends Writable interface

Must implement compareTo() method, because keys needs to be
compared during sorting phase
Developers should provide equals() and hashCode() method
Implementing Custom Keys contd
@Override
public int compareTo(Person other) {
int cmp = lastName.compareTo(other.getLastName());
if(cmp != 0) {
return cmp;
}

return firstName.compareTo(other.getFirstName());
}

Compare the fields

w you can compare single fields
Instead of comparing both the fields,
Partitioner
It is called after you emit your key value pairs from mapper
context.write(key,value)
Large number of mappers running will generate large
amount of data
And If only one reducer is specified, then all the intermediate
key and its list of values goes to a single reducer
Copying will take lot of time
Sorting will also be time consuming
Whether single machine can handle that much amount of
intermediate data or not?

Solution is to have more than one reducer

Partitioner contd

Partitioner divides the keys among the reducers

If more than one reducer running, then partitioner
decides which key value pair should go to which reducer

Default is Hash Partitioner

Calculates the hashcode and do the modulo operator
with total number of reducers which are running

Hash code of key %numOfReducer

The above operation returns between ZERO and

(numOfReducer 1)
Partitioner contd
Example:
Number of reducers = 3
Key = Hello
Hash code = 30 (lets say)
The key Hello and its list of values will go to 30 % 3 = 0th
reducer

Key = World
Hash Code = 31 (lets say)
The key world and its list of values will go to 31 % 3 = 1st
reducer
Implementing Custom Partitioner
Recap:
For better load balancing of key value pairs, you should
consider more than 1 reducer

Partitioner decides which key value pair should go to

which reducer

Default partitioner takes the entire key to decide which

key value pair should go to which reducer
Custom Partitioner contd
If using composite key(ex- firstName:lastName then it
would be difficult to achieve better load balancing

Custom key (Person) [ Last Name & First name ]

Smith John
Smith Jacob
Smith Bob
Smith Doug

What would you do if you want all the records having

the same lastName should be processed by single
reducer?
Custom Partitioner contd
Custom key (Person) [ Last Name & First name ]
Smith John
Smith Jacob
Smith Bob
Smith Doug

Default HashPartitioner will not work since it will take

the full KEY to determine which reducer it should go
Records having different first name will go to different
reducer
Smith John key is different than Smith Jacob
Custom Partitioner contd
Custom key (Person) [ Last Name & First name ]
Smith John
Smith Jacob
Smith Bob
Smith Doug

To solve this problem implement custom partitioner

which partitions the key w.r.t the lastName
All the records with same lastName will go to same
reducer
 Custom Partitioner contd
public class PersonPartitioner extends Partitioner<Person, Text>{

@Override
public int getPartition(Person outputKey, Text outputVal, int numOfReducer) {

//making sure that keys having same last name goes to the same reducer
//because partition is being done on last name

return Math.abs(outputKey.getLastName().hashCode()*127)%numOfReducer;
}
}
Custom Partitioner contd
public class PersonPartitioner extends Partitioner<Person, Text>{

@Override
Custom
public int getPartition(Person outputKey,
Partitioner should extend Text outputVal,
from Partitioner class int
numOfReducer) {
Input Arguments to Partitioner class represents mapper output key
//making sure that
and mapper keysvalue
output having same last name goes to the same
class
reducer
In thepartition
//because is being
current scenario thedone
mapon last name
output key is custom key
Custom key implements WritableComparable interface
return
Math.abs(outputKey.getLastName().hashCode()*127)%numOfReducer;
}

}
Custom Partitioner contd
public class PersonPartitioner extends Partitioner<Person, Text>{

@Override
public int getPartition(Person outputKey, Text outputVal, int numOfReducer) {

//making sure that keys having same last name goes to the same reducer
//because partition
Override is being done
the getPartition on last name
method
return
First argument is map output key
Math.abs(outputKey.getLastName().hashCode()*127)%numOfReducer;
} Second argument is map output value

Third argument is number of reducer which is set either through

job.setNumReduceTasks()
} Command line [ -D mapred.reduce.tasks ]
 Custom Partitioner contd
public class PersonPartitioner extends Partitioner<Person, Text>{

Note the partition is done on the lastName

@Override
public int getPartition(Person outputKey, Text outputVal, int numOfReducer) {
Doing this way, it will send all the keys with the same lastName to a single
reducer
//making sure that keys having same last name goes to the same reducer
//because partition is being done on last name

return Math.abs(outputKey.getLastName().hashCode()*127)
%numOfReducer;
}
}
Using Custom Partitioner in Job
Job.setPartitionerClass(CustomPartitioner.class)
Hands-On
Refer hands-on document
Assignment
Modify your WordCount MapReduce program to
generate 26 output files for each character
Each output file should consist of words starting with
that character only
Implementing Custom Input Format
In built input formats available
Text Input format
Key value text input format
Sequence file input format
Nline input format etc

In build input formats gives you pre-defined key value

pairs to your mapper

Custom input format provides better control over the

input key values processed by mapper
Custom Input Format contd
Input format is responsible for
Processing data splits (input splits)
Reading records from the input splits and sending
record by record to the map function in key value pair
form

Hence custom input format should implement

record reader
Implementing Custom Input Format
Assume the data is

Fields are separated

by space delimiter

Data is similar to wikipedia hourly data set

First column is project name
Second column is page name
Third column is page count
Fourth column is page size
Implementing custom input format contd
Lets assume that mapper should receive the records
only consisting of project name as en
Basically we are filtering the records consist of en
project name
You can run map reduce job to filter the records

Another option you can implement custom input

format
While reading the records implement your logic
Implementing custom input format contd
public class WikiProjectInputFormat extends
FileInputFormat<Text,IntWritable>{

@Override
public RecordReader<Text, IntWritable> createRecordReader(InputSplit
input, TaskAttemptContext arg1) throws IOException, InterruptedException
{
return new WikiRecordReader();
}
}
Implementing custom input format contd
public class WikiProjectInputFormat extends
FileInputFormat<Text,IntWritable>{

Extend your class from FileInputFormat

@Override
public RecordReader<Text,
FileInputFormat<Text,IntWritable>
IntWritable> createRecordReader(InputSplit input,
Text willarg1)
TaskAttemptContext throws
be the IOException,
input key given toInterruptedException
the map function
IntWritable will be the input value given to the map function
{

return new WikiRecordReader();

}

}
Implementing custom input format contd
public class WikiProjectInputFormat extends
FileInputFormat<Text,IntWritable>{

@Override
public RecordReader<Text, IntWritable> createRecordReader(InputSplit
input, TaskAttemptContext arg1) throws IOException,
InterruptedException

{ Over ride the createRecordReader method

You need to provide your own record reader to specify how to read
return new
the WikiRecordReader();
records
}
Note the InputSplit as one of the argument

}
Implementing custom input format contd
public class WikiProjectInputFormat extends
FileInputFormat<Text,IntWritable>{

@Override
WikiRecordReader
public is the
RecordReader<Text, custom record
IntWritable> reader which needs to be input,
createRecordReader(InputSplit
implemented arg1) throws IOException, InterruptedException
TaskAttemptContext

return new WikiRecordReader();

}
}
Implementing Record Reader
public class WikiRecordReader extends RecordReader<Text, IntWritable>{

private LineRecordReader lineReader;

private Text lineKey;
private IntWritable lineValue;

public WikiRecordReader() {
lineReader = new LineRecordReader();
}

@Override
public void initialize(InputSplit input, TaskAttemptContext context)
throws IOException, InterruptedException {
lineReader.initialize(input, context);
}
Implementing Record Reader contd
@Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if(!lineReader.nextKeyValue()) {
return false;
}

Text value = lineReader.getCurrentValue();

String[] splits = value.toString().split(" ");
if(splits[0].equals("en")) {
lineKey = new Text(splits[1]);
lineValue = new IntWritable(Integer.parseInt(splits[2]));
} else {
lineKey = null;
lineValue=null;
}
return true;
}
Implementing Record Reader contd
public class WikiRecordReader extends RecordReader<Text, IntWritable>{

private LineRecordReader lineReader;

private Text lineKey;
private IntWritable lineValue;
public WikiRecordReader()
{ Extend your class from RecordReader class
Using
lineReader = new LineRecordReader();
existing LineRecordReader class
}
Read line by line record
@Override Provides offset as key
initialize(InputSplit
public void And entire line as value
input, TaskAttemptContext context)
throws IOException, InterruptedException
{
lineKey will be the context);
lineReader.initialize(input, input key to the map function
} lineValue will be the input value to the map function
Implementing Record Reader contd
public class WikiRecordReader extends RecordReader<Text, IntWritable>{

private LineRecordReader lineReader;

private Text lineKey;
private IntWritable lineValue;

public WikiRecordReader()
Initialize the lineReader
{ lineReader takes the input split
lineReader = new LineRecordReader();
}

@Override
public void initialize(InputSplit input, TaskAttemptContext context)
throws IOException, InterruptedException
{
lineReader.initialize(input, context);
}
Implementing Record Reader contd
@Override
public boolean nextKeyValue() throws IOException, InterruptedException
{
if(!lineReader.nextKeyValue()){
return false;
} This function provides the input key values to mapper function
one at a time
Text value = lineReader.getCurrentValue();
String[] splits = value.toString().split(" ");
if(splits[0].equals("en")) {
lineKey = new Text(splits[1]);
lineValue = new IntWritable(Integer.parseInt(splits[2]));
} else {
lineKey = null;
lineValue=null;
}
return true;
}
Implementing Record Reader contd
@Override
public boolean nextKeyValue() throws IOException, InterruptedException
{
if(!lineReader.nextKeyValue()){
return false;
}

Text value = lineReader.getCurrentValue();

String[] splits = value.toString().split(" ");
if(splits[0].equals("en")) {
lineKey = new Text(splits[1]);
lineValue = new IntWritable(Integer.parseInt(splits[2]));
} else {
lineKey = null;
lineReader gives the offset as key and entire line as value
lineValue=null;
Once the value is taken
} It can be split on space
return true; the split[1] as key and split[2] as value
Make
}
Using Custom Input Format in Job
job.setInputFormat(CustomInputFormat.class)

Note while using custom input format, map input key

and value arguments should match with the key and
value the custom input format provides
Custom Input Format features
Can override isSplittable() function to return false
Then files will not be splitted
And entire data set will be operated by single mapper
Hands on
Refer hands-on document
Module 7
In this module you will learn

Implementing Custom comparator

Secondary sorting
Custom comparator
Comparable & Comparator are interfaces in java to
provide sorting in a collection.
Recall
MR framework does sorting of key-value pairs on keys at
Map phase as well as Reduce phase.
All keys must implement WritableComparable
WritableComparable
Hadoops equivalent interface as Comparable
Enable to define only one way of sorting Ex: Ascending
order.
How to define multiple sorting strategies?
Custom comparator contd..
Solution to above problem is use WritableComparator
Ex: secondary sort (sorted on the keys and within each key,
sorted on values)
Another advantage with custom comparators- compare()
method which deals in bytes :
compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2)

public class EmployeeSortComparator extends WritableComparator {

protected EmployeeSortComparator () {
super(Employee.class);
}
Custom comparator contd..
@Override
public int compare(WritableComparable o1, WritableComparable o2) {
Employee e1 = (Employee) o1;
Employee e2 = (Employee) o2;
int comparison =
e1.getFirstName().compareTo(e2.getFirstName());
if(comparison == 0) {
return e1.getLastName().compareTo(e2.getLastName());
}
return comparison;
}

job.setSortComparatorClass(EmployeeSortComparator .class);
Secondary Sorting
Motivation Sort the values for each of the keys.
Reminder Keys are by default sorted
So now values are also need to be sorted.
John Smith Bill Rod
John Rambo Gary Kirsten
Gary Kirsten John Andrews
John McMillan John McMillan
John Andrews John Rambo
Bill Rod John Smith
Tim Southee Tim Southee

Example use case Reduce side dataset join

Steps: Custom Type
SS means output is sorted on composite key ie combination of
natural key(first name) and value(last name).
So create a custom key type having both first name and last
name by extending WritableComparable.
public class Employee implements WritableComparable<Employee>{
private Text firstName;
private Text lastName;

public void readFields(DataInput arg0) throws IOException {

firstName.readFields(arg0);
lastName.readFields(arg0);
}
public void write(DataOutput arg0) throws IOException {
firstName.write(arg0);
lastName.write(arg0);
}
Steps: Custom Type contd..
@Override
public int compareTo(Employee o) {
int cmp = this.getFirstName().compareTo(o.getFirstName());
if(cmp != 0) {
return cmp;
}
return this.getLastName().compareTo(o.getLastName());
}

Why custom type ?

Sorting has to be done on key value combination, so the key Type must
have both fields. Hadoop data type is not sufficient !
Steps: Custom comparator
Next create a comparator so that new type can be sorted on
key & value combination.
Extends WritableComparator

Person class implement WritableComparable, which is

required for sorting, then why a WritableComparator ?
When WritableComparator is used, it gives compare() method on bytes, then
the framework compares the serialized values, which is faster.
Steps: Custom comparator contd
The New Comparator will sort the output from Mapper as well
as Reducer
Input:
(John, Smith) (John Rambo) (Gary Kirsten)

Mapper

Output :
(Gary#Kirsten:Kirsten) (John#Rambo:Rambo) (John#Smith:Smith)

Notice the key its the custom key (Employee)

Its just a representation of object
Steps: Custom comparator contd
public class SecondarySortComparator extends WritableComparator {

@Override
public int compare(WritableComparable o1, WritableComparable o2) {
Employee e1 = (Employee) o1;
Employee e2 = (Employee) o2;
int cmp = e1.getFirstName().compareTo(e2.getFirstName());
if(cmp != 0) {
return cmp;
}
return e1.getLastName().compareTo(e2.getLastName());
}

@Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
..
}
Steps :Custom Partitioner
Mappers emit <Custom Key, value>
All the keys having same natural key(first name) must
go the same reducer. So, create a custom Partitioner
which should partition on natural key.
public class SecondarySortPartitioner implements
Partitioner<Employee,Text>
{
@Override
public int getPartition(Employee emp, Text lastName, int
numOfReducer) {
return (emp.getFirstName().hashCode() & Integer.MAX_VALUE ) %
numOfReducer;
}
}
Steps: Grouping Comparator
Output from Mappers get collected to reducer.
Output is sorted on the composite key using comparator
defined earlier.
Example input for reducer phase

(Gary#Kirsten:Kirsten) (John#Rambo:Rambo) (John#Smith:Smith)

Remember reducer get key and list of values(reduce method

!)
If framework now try to group this input in key and lists of
values, it wont be correct as all keys are different.
Steps: Grouping Comparator contd..
To solve above problem, create a grouping Comparator
Implements WritableComparator. Make it to compare only
on the natural key(firstName) of Employee
public class GroupingComparator extendsWritableComparator {

@Override
public int compare(WritableComparable o1, WritableComparable o2) {
Employee e1 = (Employee) o1;
Employee e2 = (Employee) o2;
return e1.getFirstName().compareTo(e2.getFirstName());
}
Steps: Driver class
How to make use of these pieces in MR Job.

job.setInputFormat(KeyValueTextInputFormat.class);
job.setMapperClass(SecondarySortMapper.class);
job.setReducerClass(SecondarySortReducer.class);

job.setMapOutputKeyClass(Employee.class);
job.setMapOutputValueClass(Text.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);

job.setPartitionerClass(SecondarySortPartitioner.class);
job.setOutputKeyComparatorClass(SecondarySortComparator.class);
job.setOutputValueGroupingComparator(GroupingComparator.class);
Steps: Mapper
@Override
public void map(Text arg0, Text arg1, Context context) throws IOException {

Employee emp = new Employee();

emp.setFirstName(arg0);
Text lastName = new Text(arg1.toString());
emp.setLastName(lastName);
context.write(emp, lastName);

}
Steps: Reducer
@Override
public void reduce(Employee emp, Iterator<Text> values, Context context)
throws IOException {

Text firstName = emp.getFirstName();

while(values.hasNext()) {
context.write(firstName , values.next());
}
}
Hands On
Refer hands-on document
Joins
In this module you will learn
What are joins?
What is map side join?
What is reduce side join?
Hands on
What are joins?
Joining means joining two data sets on a common key

Employee Table Country Table

CountryCode Country Name
Name ID Country
Code AUS Australia
James 01 AUS
IN India
Siddharth 11 IN
Suman 23 US US United States
Joins contd
Your job is to replace the country code in the employee
table with country names

Output should be

Name ID Country
James O1 Australia
Siddharth 11 India
Suman 23 UnitedStates
Joins contd
MapReduce provides two types of join
Map Side join
Reduce Side join

Both kind of joins does the join but differs the way it is
done
Map Side join is done at mapper phase. No Reducer is
required
Reduce side join requires to have reducer
Map Side Join
Use this join, if one of the data sets you are joining can
be fit into memory
In the previous example if CountryTable can be fit into
memory, then do the joining during mapper phase

Map Side join is done in memory, and hence it is fast

Use Distributed Cache facility and cache the

CountryTable and use this in map function
Map Side Join contd

Public class MapSideMapper extends Mapper<>

{
public void setup(Context context)
{
Step1: Get the CountryTable from the distributed cache
Step2: You can populate a hash table where key is country code
and value is country name
}
public void map()
{
Step 1: Read the record and get the country code
Step 2: For the given country code get the country name from the
hash table you populated in set up method above and
replace in the record
Step 3: Write the modified record using context object
}

}
Map Side join contd
This join is faster
Hash table is in memory and for every record just a look
up is required
Use when dataset is small. (Can fit in memory)

It is not scalable
Memory is constraint
What if the CountryTable size is 100GB?
Reduce side join
Use case : Joining two datasets sharing one common
field in reducer phase.
Motivation : Map side join relies heavily on memory,
so when datasets are very large, Reduce side join is the
only option.
Example:
Employee Location

EMPID Name Location LocationId LocationName

42 John 13 13 New York

EMPID Name Location LocationName

42 John 13 New York
Reduce side join contd..
Steps:
Run mapper on both datasets and emit key which is the
common fields and value as the entire record. Ex:

M1 { 13 : 42 John 13 }
M2 { 13 : New York

Now, join the data in Reducer (all values for the same key will be
passed to the same reducer)
Is there any problem here ?
Reduce side join contd..
Problem: For larger datasets one value need to be
joined with several other values(and values are not
sorted!). Ex: location with employee

Hint use secondary sort to make sure that the

joining key appears first on reducer
Reduce Side Join contd
Hands On
Refer hands-on document
Practical development
In this module you will learn

Deciding number of mappers and reducers

Map only jobs
Compression
Deciding Number of Mappers
By default total number of mappers running will be
equal to total number of blocks

If you have 1000 blocks then 100o map task will run
If you have 50 machines and each machine is dual core
then total of 100 ( 50 machines * 2 cores) tasks can run
parallely.
Scheduling 1000 maps will be time consuming
You can merge the files and make it a bigger files

NOTE: Lot of smaller files means lot of tasks

Deciding Number of Reducer
By default only one reducer 1

Running one reducer could be very performance

bottleneck
Copying will take time
Sorting will take time
Reducer will be slow

Can run more than one reduce, but the output will not
Compression
Compressing the intermediate key value pairs from the
mapper

Use LZO / Snappy compresion

Data Mining
In this module you will learn

Data Mining on Wikipedia data set

Understanding Wikipedia data set
Download wikipedia data set from
http://dammit.lt/wikistats/

Its an hourly data set

Structure data with four columns

Project Name
Page Name
Page Count
Page size
Wikipedia Data set contd
Sample data in file

Page Name Appears only

once in a file
Example:
Yahoo page has been
watched 12 times in
that hour
First column is project name
Second column is page name
Third column is page count
Fourth column is page size
Analyzing Wikipedia data set- Part 1
Find Top 10 English project from a set of Wikipedia
files
Analyzing Wikipedia data set- Part 1 contd
Approach:
Use 2 MapReduce job

Map Reduce Job 1

Use Text Input Format
In mapper filter the project belonging to english project
In reducer calculate the sum of page count

Map Reduce job 2

Take the output of previous map reduce job
Do the sorting (secondary sorting) and arrange in
descending order
Analyzing Wikipedia data set-Part 2
Find top-10 English project sites?
Analyzing Wikipedia data set- Part 2 contd
Approach:
Use 2 MapReduce job

Map Reduce Job 1

Use Custom input format and send only English project to
your mapper
From mapper send page name and page hit/count
In reducer calculate the sum of page count

Map Reduce job 2

Take the output of previous map reduce job
Do the sorting (secondary sorting) and arrange in
descending order
In this module you will learn
What is the motivation for Hive and Pig?
Hive basics
Pig basics
In this module you will learn
What is the motivation for Hive and Pig?
Hive basics
Pig basics
Motivation of using Hive and Pig
Most of MapReduce programming is done in java

So one who is solving big data problems must

Have good Java programming back ground
Understand MapReduce and its concepts very well

Solving a word count problems needs 100 lines of code

Solving data mining problem (on wikipedia data set,

which we have seen earlier) require two MapReduce
jobs to be implemented and run sequentially one after
the other
Motivation contd
Map Reduce in Java is very low level
Needs to deal with all the intricacies

Not only programmer, many non programmers like

Business Analyst, Data Scientist wants to analyze the
data

Requirement:
Need of higher level abstraction on top of MapReduce
Not to deal with low level stuff involved in MapReduce
In this module you will learn
What is the motivation for Hive and Pig?
Hive basics
Pig basics
Hive
Data ware housing tool on top of Hadoop

SQL like interface

Provides SQL like language to analyze the data stored

on HDFS

Can be used by people who know SQL

Not all traditional SQL capabilities are supported

Example: Sub queries are not supported in hive
Hive contd
Under the hood hive queries are executed as
MapReduce jobs

No extra work is required

Hive Components
MetaStore
Its a data base consisting of table definitions and other
metadata
By default stored on the local machine on derby data
base
It can be kept on some shared machine like relational
data base if multiple users are using
Hive components contd
Query Engine
Hive-QL which gives SQL like query
Internally Hive queries are run as map reduce job
Installing Hive
Download Hive (0.10.0)

Create following environment in .bashrc file

export HIVE_HOME=<path to your hive home directory>
export HIVE_CONF_DIR=$HIVE_HOME/conf
export HIVE_LIB=$HIVE_HOME/lib
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:$HIVE_LIB
Installing Hive
Create hive-site.xml (if not present) under
$HIVE_HOME/conf folder
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310/</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>
</configuration>
Installing Hive
Create two directories /tmp and /user/hive/warehouse on
HDFS and give them full permission
hadoop fs mkdir /tmp
hadoop fs -mkdir /user/hive/warehouse
hadoop fs chmod g+w /tmp
hadoop fs chmod g+w /user/hive/warehouse
Launching Hive Shell
Type hive command
Hive Data Models
Hive forms or layers table definitions on top of data
residing on HDFS
Databases
Name space that separates tables from other units from
naming confliction
Table
Homogenous unit of data having same schema
Partition
Determines how the data is stored
Virtual columns, not part of data but derived from load
Buckets / Cluster
Data in each partition can be further divided into buckets
Efficient for sampling the data
Hive data types
Primitive types
TINYINT (1byte)
SMALLINT (2 byte)
INT (4 bytes)
BIGINT (8 bytes)
FLOAT
BOOLEAN
DOUBLE
STRING
BINARY ( recently included)
TIMESTAMP(recently included)
!Type constructors:
ARRAY < primitive-type >
MAP < primitive-type, data-type >
STRUCT < col-name : data-type, ... >
Basic Queries

hive> SHOW TABLES;

hive>SHOW DATABASES;

hive> CREATE TABLE sample(firstName STRING, lastName

STRING, id INT)
ROW FORMAT
DELIMITED FIELDS
TERMINATED BY
LINES TERMINATED BY \n
STORED AS TEXTFILE

hive> DESCRIBE sample;

Loading Data Into Table From HDFS
Assuming data is already present on HDFS

LOAD DATA INPATH sample_data INTO TABLE sample;

The sample_data on HDFS will be moved to

/user/hive/warehouse/sample folder on HDFS
/user/hive/warehouse is metastore
Loading Data Into Table From Local File System
Give the path to the data present on your local file
system

LOAD DATA LOCAL INPATH sample_local_data INTO TABLE

sample;
Basic Select Queries
SELECT * FROM sample;

SELECT * FROM sample WHERE id >100

ORDER BY id ASC
LIMIT 20

The first query does not run map reduce job

Just queries the meta data and displays the entire data

Second query executes map reduce job

Joins in Hive
Joins are complex and trickier in map reduce job

SELECT A.id, B.id,A.firstName FROM

sample A
JOIN otherTable B
ON
(A.id = B.id)
Storing Results
Create a new table and store the results into this table

CREATE TABLE output(sum int);

INSERT OVERWRITE INTO TABLE output

SELECT sum(id) FROM TABLE sample;

Results are stored in the table output

Result is just an output file which can be used in

subsequent queries or in map reduce jobs
Data Mining on Wikipedia datasets
Design a schema

CREATE EXTERNAL TABLE wiki(projectname STRING,pagename

STRING,pageview INT,pagesize INT)
ROW FORMAT
DELIMITED FIELDS
TERMINATED BY ' '
LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION /user/dev/wiki';

External table will not move the data to

/user/hive/warehouse folder
It just point to that location

Data Mining on Wikipedia datasets

Create another table to store the output

CREATE TABLE wiki1(pagename STRING,sum INT);

Data Mining on Wikipedia datasets
Fire the actual query to calculate the sum of the page
count

INSERT OVERWRITE TABLE wiki1

SELECT pagename,sum(pageview) FROM wiki
WHERE projectname='en' group by pagename;
Data Mining on Wikipedia datasets
View the results on console in descending order

Select * from wiki1 order by sum DESC limit 20

Hive Limitations
All the standard SQL queries are not supported
Sub queries are not supported

No Support for UPDATE and DELETE operation

Cannot insert single rows

Hands On
Refer hands-on document
Installing Pig
Download Pig (0.10.0)
Create following environment variables in .bashrc file
export PIG_HOME=<path to pig directory>
export PATH = $PATH:$PIG_HOME/bin
export PIG_HADOOP_VERSION=1 (Since we are using hadoop
version of 1.0.3)
export PIG_CLASSPATH=$HADOOP_HOME/conf/
Specify fs.default.name & mapred.job.tracker in pig.properties file
under $PIG_HOME/conf directory
No need to install the pig in all the nodes.
Requires only where the namenode and job tracker are running
Launching Grunt Shell
Run pig command
Pig basics
High level platform for MapReduce programs
Uses PigLatin Language
Simple Syntax
PigLatin queries are converted into map reduce jobs

No shared meta data is required as in Hive

Pig Terminology

Tuple:
Rows / Records
Bags:
Unordered Collection of tuples
Relation:
Pig Operator.
Generally the output of operator is stored in a relation

PigLatin scripts generally starts by loading one or more

data sets in bag, and then creates new bags by modifying
Loading Data
LOAD keyword

TAB is the default delimiter

If input file is not tab delimited, even though loading will not be a
problem but subsequent operations will be erroneous

Other delimiters can be specified by using PigStorage( )

In this case space is the delimiter
Loading Data contd

inputData = LOAD 'wiki/input1'using PigStorage(' ') as

(projectName:chararray,pageName:chararray,pageCount:int,pageSize:int);

Data wiki/input1 is stored on HDFS

Here the fields are separated by space which is defined by USING

PigStorage( )

Note when you are defining the column name, you need to specify the
data types of the column,else NULL will be stored

inputData will consist of the output of the loading

Filtering
Use Filter Keyword

Output = FILTER records by projectName==en

records is the data set and pageName is the column

name defined in the records
Grouping
Use GROUP key word

grouped_records = GROUP filtered_records by pageName;

Here grouping is done on the column pageName

Grouping contd

grouped_records = GROUP filtered_records by pageName;

Here grouping is done on the column pageName

Each Tuple in grouped_records has two parts
group element which is column name on which you are
grouping
filtered_records is itself is a bag containing all the tuples
from filtered_records with matching pageName
Joins
Use JOIN keyword

result = JOIN persons BY personId, orders BY personId;

Default is INNER JOIN

Above example joins persons and orders data on

personId column
Joins contd
Left outer join

result = JOIN persons BY personId LEFT OUTER, orders BY personId;

For doing left outer join use LEFT OUTER key word
Joins contd

Right outer join

result = JOIN persons BY personId RIGHT OUTER, orders BY

personId;
Joins contd
Full outer join

result = JOIN persons BY personId FULL OUTER, orders BY personId;

FOREACHGENERATE
Iterate over tuples in a bag

empID=FOREACH employee GENERATE id;

Other useful commands

DESCRIBE

EXPLAIN

ILLUSTRATE

REGISTER
User Defined Functions (UDF)
DEFINE
DataMining: Top 10 English sites from wiki data

records = LOAD 'wiki/input1'using PigStorage(' ') as

(projectName:chararray,pageName:chararray,pageCount:int,pageSize:int);

filtered_records = FILTER records by projectName == 'en';

grouped_records = GROUP filtered_records by pageName;

results = FOREACH grouped_records generate group,

SUM(filtered_records.pageCount);

sorted_result = ORDER results by $1 desc;

STORE sorted_result INTO 'wikiOP ;

Using Pig Scripts
Can combine the pig statements in a script
Script name should end with .pig extension

Can run the pig script by one of the three ways as

described as follows:
pig script_name.pig
From the grunt shell
run script_name.pig
exec script_name.pig
From java (Embedded Pig)
Hands On
Refer hands-on document
Big Data Is No More BIG