Cannot execute libexec/hdfs-config.sh or libexec/hadoop-config.sh — Hadoop Single Node Cluster problem

Hi there, today we will learn about solving the most common problem while installing Hadoop on a single node cluster.

The main reason why we get the error “Cannot execute:hadoop/libexec/hdfs-config.sh or hadoop/libexec/hdfs-config.sh” is because of the permissions not available for Hadoop to run. Whenever we install Hadoop in localhost, we generally configure SSH to avoid entering passwords or to avoid providing root permissions to Hadoop. In a simple way, we configure Hadoop to run in non-root mode.

Probable reasons why you get “cannot execute libexec/hadoop-config or hdfs-config.sh” error

1. Check whether the permissions from where you are running Hadoop is properly set to run without root permissions.
2. Check JAVA_HOME is properly set in ~/.bashrc. Sometimes you need to add JAVA_HOME to /etc/profile
3. Check the HADOOP_HOME environmental variables in ~/.bashrc file. I have provided the list of necessary environmental vars for Hadoop below.
4. Check where the logs of Namenode and Datanode are created. Make sure that folder has non-root permissions else Namenode and Datanode can’t be started when we type sbin/start-dfs.sh

I will follow a simple and compact installation of Hadoop on a single cluster or localhost eliminating all unnecessary confusion.

Step1: Download Hadoop 3.x

Go to the official Hadoop website and download hadoop tar.gz. Once you download your file should be as “hadoop-3.x.x.tar.gz”

Apache Download Mirrors

Let's say the tar file is present in your Downloads directory. For convenience, we will move it to /usr/local/

For this, you would definitely require “su” permissions as /usr/ belongs to root. Execute the following command from the Downloads directory on your terminal.

hemachandra@hemachandra:~/Downloads$ sudo mv hadoop-3.1.2.tar.gz /usr/local/

Here “hemachandra” is my user name and I belong to a group “hemachandra”. My hadoop tar file is present in the Downloads directory of user “hemachandra”. For moving this file from my user account to the root account I need administrator rights. For this, we are using “sudo”.

Step2: Move to /usr/local directory and extract the tar file

Type the following command and press enter.

sudo tar -xzvf hadoop-3.1.2.tar.gz

Extraction of tar file

List the files in /usr/local/

Step3: Rename Hadoop folder and set permissions

For ease, we will rename extracted Hadoop folder from hadoop-3.1.2 to hadoop-inst.

Renaming hadoop-3.1.2 to hadoop-inst and listing files in /usr/local

We will now change the permissions of the hadoop-inst folder such that it can be executed with any root permissions. As any folder that is created in /usr/local belong to root user we have to change the ownership of the folder. For this, we have to run “chown” command to change the owner of the folder to our user name(here: hemachandra)

sudo chown -R YOUR_USER_NAME:YOUR_USER_GROUP hadoop-inst/

owning permissions of hadoop-inst folder to our user and listing files

Here first “hemachandra” is my user name and the second “hemachandra” after “:” is user group. “-R” indicates recursively change permissions of all subdirectories and folders of hadoop-inst folder.

Now that you are good to go. You have perfectly set the permissions for your Hadoop now. We will now start configuring JAVA, SSH, and Hadoop.

Step4: Install and set environmental variables for JAVA

This is pretty straight forward and you can find a lot of articles over the internet. We will use OpenJDK 8 instead of standalone JAVA8. Open your terminal and type the following.

sudo apt update
sudo apt install openjdk-8

If you are getting “unable to locate package error” then refer here.

After the commands are successful we can find JAVA installed at /usr/lib/jvm/

Setting environmental variables for JAVA

We have to add the JAVA_HOME variable to ~/.bashrc.

sudo nano ~/.bashrc

Type the above command and then enter the following lines at the end of the file.

export JAVA_HOME=”/usr/lib/jvm/java-1.8.0-openjdk-amd64"
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export PATH

Make sure your JAVA_HOME value is exactly correct. If required go to the /usr/lib/jvm/ path and verify the exact name of JDK. After this execute the following command for the changes to take effect without rebooting.

source ~/.bashrc

Step5: Setting up SSH(Secured Shell)

Hadoop requires SSH access to all the nodes configured in the cluster. We need to configure SSH access to the localhost. Run the following to install ssh.

sudo apt-get install ssh

sudo apt-get install pdsh

Setting up passwordless SSH

We have to generate public and private SSH key pairs for our computer. The secure communication between client and server happens using these SSH keys. Open a terminal and execute the following commands.

ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Open and add the following line at the end of ~/.bashrc

$ sudo nano ~/.bashrc

export PDSH_RCMD_TYPE=ssh

check if you can SSH to the localhost. Type the following in a terminal.

ssh localhost

ssh localhost on ZORIN os

You should be seeing something similar to the above image output when you ssh to localhost. As I am running ZORIN os(flavor of Ubuntu) the output corresponds to ZORIN. You should be seeing Ubuntu or any other distro which you installed.

Step6: Configuring Hadoop config files

Configuring hadoop config files involves in editing the following files present in /usr/local/hadoop-inst/etc/hadoop/

1. hadoop-env.sh
2. core-site.xml
3. yarn-site.xml
4. hdfs-site.xml
5. mapred-site.xml

Add JAVA_HOME and HADOOP_HOME path in hadoop-env.sh

Go to /usr/local/hadoop-inst/etc/hadoop/ and open hadoop-env.sh using a text editor or you can directly open the file from a terminal. Now that you changed the permissions of the folder, you can edit using GUI without any special permissions.

Get Hemachandra’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Get to 54 line and un-comment the line and add the JAVA_HOME path which you added in ~/.bashrc file as shown below. Similarly, in the same file please un-comment line 58 and add the home path of your hadoop-inst.

NOTE: Make sure you add double quotes and also remove ‘/’ at the end

adding JAVA_HOME and HADOOP_HOME

Adding Hadoop Core configuration in core-site.xml

Navigate to /usr/local/hadoop-inst/etc/hadoop (the same path where hadoop-env.sh). Find and open core-site.xml and add the following. Make sure you remove <configuration></configuration> tags and add the following.

<configuration>
  <property>       
      <name>hadoop.tmp.dir</name>
      <value>/usr/local/hadoop-inst/hdfs-store/tmp</value>     
  </property>    
  <property>      
      <name>fs.defaultFS</name>     
      <value>hdfs://localhost:9000</value>    
  </property>
</configuration>

NOTE: We will configure the /hdfs-store/tmp while configuring hdfs-site.xml

Configure ResourseManager and NodeManager in yarn-site.xml

Similar to core-site.xml add the following to yarn-site.xml

<configuration>
   <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
   </property>
   <property>
        <name>yarn.nodemanager.aux-  services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

Configuring hdfs-site.xml for hadoop master node

This file holds configuration settings for HDFS daemons, the Namenode, the secondary Namenode, and the data nodes.

Both the Namenode and Datanode write some logs when they get started. For storing these logs we need to create a folder. Let us create the log folder in the same hadoop-inst folder as the permissions are already set on the folder. Follow the commands:

cd /usr/local/hadoop-inst

mkdir hdfs-store

cd hdfs-store

We will now create separate folders for Namenode, Datanode, and for other logs. Make sure you are in hdfs-store folder.

mkdir namenode

mkdir datanode

mkdir tmp

Now, configure these paths in hdfs-site.xml file present in /usr/local/hadoop-inst/etc/hadoop

$ cd /usr/local/hadoop-inst/etc/hadoop/

<configuration>
    <property>
       <name>dfs.replication</name>
       <value>1</value>
    </property>
    <property>
       <name>dfs.name.dir</name>
       <value>/usr/local/hadoop-inst/hdfs-store/namenode</value>
    </property>
    <property>
       <name>dfs.data.dir</name>
       <value>/usr/local/hadoop-inst/hdfs-store/datanode</value>
    </property>
</configuration>

Configure MapReduce settings in mapred-site.xml

<configuration>
    <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
    </property>
</configuration>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=/usr/local/hadoop-inst</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=/usr/local/hadoop-inst</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=/usr/local/hadoop-inst</value>
    </property>

Step7: Adding HADOOP environmental variables to bashrc

Open /.bashrc and add the below respective Hadoop environmental variables at the end.

sudo nano ~/.bashrc

export HADOOP_HOME="/usr/local/hadoop-inst"
export HADOOP_CONF_DIR="/usr/local/hadoop-inst/etc/hadoop"
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

Step8: Formatting Namenode

This is the most important step. You have to make sure this step works perfectly else you have done something wrong in the above steps.

Namenode is used to specify the default file system and also the defaults of your local file system. When we format the Namenode, the original Datanode is not affected but the MetaData related to it gets deleted. So, we can use this space for new Data.

Open a fresh terminal and type the following:

ssh localhost

cd /usr/local/hadoop-inst/

Format the Namenode now:

bin/hdfs namenode -format

the output of namenode -format

Make sure you get the output similar to the above image This is the most important step. If you see any of the errors in the terminal output, I highly recommend you to check the XML file configurations and environmental variables are properly set in the ~/.bashrc.

Step9: Starting dfs and yarn services

Now that you have formatted the namenode. We will proceed ahead and start the dfs and yarn services now to verify whether everything is working fine. Run the below commands one after the other and compare the output.

sbin/start-dfs.sh

Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [hemachandra]

sbin/start-yarn.sh

Starting resourcemanager
Starting nodemanagers

To verify type jps

jps

13233 Jps
12801 NodeManager
11937 NameNode
12354 SecondaryNameNode
12649 ResourceManager

Now that all of your HDFS and YARN services are successfully switched on. And you are done.

For stoping the services, you can:

sbin/stop-all.sh

WARNING: Stopping all Apache Hadoop daemons as hemachandra in 10 seconds.
WARNING: Use CTRL-C to abort.
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [hemachandra]
Stopping nodemanagers
Stopping resourcemanager

For more courses, you can visit my personal website.