HDFS Commands

Well they do just require a Prefix 😗

After, we have installed Cloudera Quickstart VM we are good to start trying and firing some basic commands on the terminal and try to do some basic stuff like renaming a file, creating a file etc. If you still haven’t installed Quickstart VM on your machine then do follow my last article for installation on windows (I don’t have a mac 💻until now but will surely write installation steps for it when I get my hands on one).

To open a terminal just click on the terminal icon on top

Before jumping to code let’s understand the basic file structure of hdfs.

There are two kinds of home directories available to us one is local and other is HDFS the local just work like your any other directory in windows or Linux and there is not distributed processing whereas in HDFS there is distributed processing going on.

Home Directory for a user in Local : /home/cloudera

Home Directory for a user in HDFS: /user/cloudera

How does HDFS commands work?

😶😶

All HDFS commands will interact with the name node(example : ls). However, some commands will go to the data nodes to read the data from files (example : cat, tail)

Some basic symbols to help further

1. root folder /
2. home ~
3. current .
4. parent ..
5. previous -

Some HDFS commands

commands are just like commands in Linux but we just have hadoop fs - or hdfs dfs -prefix in every command so command becomes.

That’s how HDFS will handle big data🌋🌋

hadoop fs - <command name>
hdfs dfs - <command name>

Note :- to get help on any command hadoop fs -help <command name>

ls command

To see list of files inside a directory

hadoop fs -ls /user/cloudera
hadoop fs -ls (it will display all the files and folders in current directory)

sort files by time

hadoop fs -ls -t -r /

Here, t stands for time and r stands for reverse so oldest file will be displayed first

Get Prikshit Singla’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

sort by size (by default largest is on top)

hadoop fs -ls -S /
hadoop fs -ls -S -h / (size will be displayed now in human readable form)

search in the listed files using grep

hadoop fs - ls /user | grep cloudera

We will firstly look for files inside /user directory and then we will look for cloudera in that list. Pipe operator ( | ) is used when we need to use one command after another.

mkdir command

It is used to create directory

hadoop fs -mkdir /user/cloudera/testing

rm command

rm can be used to remove files

hadoop fs -rm /user/cloudera/file1.txt (will just remove files not directories)
hadoop fs -rm -R /user/cloudera/testing (will remove directory)

Copying files or folder from local to hdfs

The (copyFromLocal or put )command is used to copy local files to hdfs.

hadoop fs -copyFromLocal <local_path> <hdfs_path>
hadoop fs -copyFromLocal Desktop/file1.txt /data
hadoop fs -put Desktop/file1.txt /data

Copying files or folder from hdfs to local

The (copyToLocal or get)command is used to copy hdfs files to local.

hadoop fs -copyToLocal <hdfs_path> <local_path>
hadoop fs -get <hdfs_path> <local_path>

cp command

to copy one location from one hdfs location to another

hadoop fs -cp <hdfs_path1> <hdfs_path2>

mv command

This command is used to move a file from one location to another. This simply means we are renaming a file. So just metadata is needs to be updated and this is very quick

hadoop fs -mv <hdfs_path1> <hdfs_path2>

Note :- For cp and mv command the condition is we can just move/copy it within hdfs or local. The cross movement between local and hdfs or vice versa is not possible with these commands

df command

df command is used to check free disk space

hadoop fs -df -h /user/cloudera (h is for human readable form)

du command

du command is used to check disk usage

hadoop fs -du -h /user/cloudera

Some Important and Useful Commands

Dynamically change the replication factor

for changing the replication factor for data nodes that is by default 3 we need to set the below property

dfs.replication=5

fsck command

This command is used to see metadata in hdfs. fsck stands for file system check

hdfs  fsck  /user/data.csv -files  -blocks  -locations

The above command will give the block information and replication information along with the ip address of datanodes where the block are kept.

How my parents and siblings see me when I type these commands 🤣🤣

I think that’s a lot to digest for today. I know it may seem like a long list of commands but you don’t actually need to learn them you can actually see that the command name actually tells a lot about it and as we will be practicing these commands we will get a grasp on them. So, don’t worry and chill 🧊. We will be having a lot of fun going forward and the theory 📝 part will be followed by a lot of practical stuff.

Next, time we meet we will start our adventure 🎢 with Map Reduce and you will see the actual power ⚡ of Hadoop then. Believe me it’s awesome guys! We have stretched up long enough for today.

Let’s end this blog 🤗 for now!

Until next time guys 🥰🥰

Thanks! for reading .

If you loved this blog please do respond with a clap 👏

Follow..☺️ me for more such insightful articles✍️

Share your thoughts in comments and please share the feedback for the part which needs to be improved so that I can improve next time.

Have a Good Day Everyone!