An IT technologist building the bridge between strategy and execution...
November 14, 2010

When Installing a Jobtracker of any Hadoop cluster, be careful on how these two files interact each other:

  • /etc/hosts
  • $HADOOP_HOME/conf/mapred-site.xml
Hadoop cluster at Kaukana

To make a long story short,  this is very wrong:

$>cat /etc/hosts
127.0.0.1           localhost   tophat
192.168.3.60   tophat

And, if you resolve to hostname in mapred-site.xml

$>cat $HADOOP_HOME/conf/mapred-site.xml
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>tophat:50300</value>
</property>
<property>
:::::

Why this is all wrong? Because if the host’s hostname is resolved to localhost the Jobtracker will bind to 127.0.0.1 and it will not be visible by those other services using external resolving!!

An indication you incur in this problem can be investigated in Jobtracker’s logs, if one finds the following message:

2010-11-14 11:29:53,771 FATAL org.apache.hadoop.mapred.JobTracker: java.net.BindException: Problem binding to tophat/127.0.1.1:50300 : Address already in use

Then a healthy check with Netstat should give the final clue:

$>sudo netstat -na

And check if sockets listening on JobTracker’s port are bound on localhost.

Leave a Reply

Name Required:

Email Required:

Website

Comment Required: