Sunday, 18 September 2016

Hadoop Cluster Practice Commands

How to Start / Stop Hadoop Process:

Name Node: start namenode stop namenode

Data Node: start datanode stop datanode

Secondary Name Node: start secondarynamenode stop secondarynamenode

Job Tracker: start jobtracker stop jobtracker

Task Tracker: start tasktracker stop tasktracker

Resource Manager: start resourcemanager stop resourcemanager

Node Manager: start nodemanager stop nodemanager

Job History Server: start historyserver stop historyserver

Hadoop Required URLS
Name Node :                http://orienit1:50070
Resource Manager :     http://orienit2:8088

1. Create a new folder with your hostname (i.e orienit1 / orienit2 / orienit3) using below command.

hadoop fs -mkdir /orienit

2. Put some files from Local File System to HDFS using below command

hadoop fs -put <local file system path> <hdfs path>

hadoop fs -put /etc/hosts /orienit/hosts

3. Read the Data from HDFS using below command

hadoop fs -cat /orienit/hosts

4. Change the Replication factor using below commands

Increase the replication number:

hadoop fs -setrep 5 /orienit/hosts

Decrease the replication number:

hadoop fs -setrep 3 /orienit/hosts

5. Transfer the data from one cluster to other cluster using below command

hadoop distcp hdfs://nn1:8020/<src path> hdfs://nn2:8020/<dst path>

where nn1 is first cluster namenode ip or hostname

where nn2 is second cluster namenode ip or hostname

6. Commissioning and Decommissioning the Nodes in Hadoop Cluster

In Name Node machine modify below changes

1. create include file in /home/kalyan/work folder

2. create exclude file in /home/kalyan/work folder

3. update hdfs-site.xml with below configurations



4. execute below command to reflect the hdfs changes

hadoop dfsadmin -refreshNodes

5. update yarn-site.xml with below configurations



6. execute below command to reflect the mr changes

yarn rmadmin -refreshNodes

7. verify the changes in browser

Share this article with your friends.

No comments :

Post a Comment

Related Posts Plugin for WordPress, Blogger...