10 Basic HDFS Commands
Top 10 basic Hadoop HDFS operations managed through shell commands which are useful to manage files on HDFS clusters; for testing purposes, you can invoke this commands using either some of the VMs from Cloudera, Hortonworks, etc. or if you have your own setup of pseudo distributed cluster.
- Create a directory in HDFS at given path(s).
Usage: hadoop fs -mkdir <paths>
Example: hadoop fs -mkdir /user/bose/dir1 /user/bose/dir2
2. List the contents of a directory.
Usage : hadoop fs -ls <args>
Example: hadoop fs -ls /user/bose
3. Upload and download a file in HDFS.
Upload: hadoop fs -put: Copy single src file, or multiple src files from local file system to the Hadoop data file system
Usage: hadoop fs -put <localsrc> … <HDFS_dest_Path>
Example: hadoop fs -put /home/bose/Samplefile.txt /user/bose/dir3/
Get Bhaskar Berry’s stories in your inbox
Join Medium for free to get updates from this writer.
Download:hadoop fs -get: Copies/Downloads files to the local file system
Usage: hadoop fs -get <hdfs_src> <localdst>
Example: hadoop fs -get /user/bose/dir3/Samplefile.txt /home/
4. See contents of a file. Same as unix cat command:
Usage: hadoop fs -cat <path[filename]>
Example: hadoop fs -cat /user/bose/dir1/abc.txt
5. Copy a file from source to destination
This command allows multiple sources as well in which case the destination must be a directory.
Usage: hadoop fs -cp <source> <dest>
Example: hadoop fs -cp /user/bose/dir1/abc.txt /user/bose/dir2
6. Copy a file from/To Local file system to HDFS
copyFromLocal
Usage: hadoop fs -copyFromLocal <localsrc> URI
Example: hadoop fs -copyFromLocal /home/bose/abc.txt /user/bose/abc.txt
Similar to put command, except that the source is restricted to a local file reference.
copyToLocal
Usage: hadoop fs -copyToLocal [-ignorecrc] [-crc] URI <localdst>
Similar to get command, except that the destination is restricted to a local file reference.
7. Move file from source to destination.
Note:- Moving files across filesystem is not permitted.
Usage : hadoop fs -mv <src> <dest>
Example: hadoop fs -mv /user/bose/dir1/abc.txt /user/bose/dir2
8. Remove a file or directory in HDFS.
Remove files specified as argument. Deletes directory only when it is empty
Usage : hadoop fs -rm <arg>
Example: hadoop fs -rm /user/bose/dir1/abc.txt
Recursive version of delete.
Usage : hadoop fs -rmr <arg>
Example: hadoop fs -rmr /user/bose/
9. Display last few lines of a file.
Similar to tail command in Unix.
Usage : hadoop fs -tail <path[filename]>
Example: hadoop fs -tail /user/bose/dir1/abc.txt
10. Display the aggregate length of a file.
Usage : hadoop fs -du <path>
Example: hadoop fs -du /user/bose/dir1/abc.txt
