Tuesday, May 22, 2012

-libjars not working in custom mapreduce code, How to debug

Mostly application developers bump into this issue. They ship their custom jars to map reduce job but when the classes in those are referred by code it throws a Class not found exception.

For -libjars to work your main class should satisfy the following two conditions.
 
1) Main Class should implement the Tool interface


 //wrong usage - Tool Interface not implemented
public class WordCount extends Configured {

//right usage
public class WordCount extends Configured implements Tool {
 
2) Main Class should get the existing configuration using getConf() method rather than creating anew configuration instance.


//wrong usage - creating anew instance of Conf 
public int run(String[] args) throws Exception {
   Configuration conf = new Configuration();
 
//right usage 
 public int run(String[] args) throws Exception {
    Configuration conf = getConf();

16 comments:

  1. It is really nice to see the best blog for hadoop tutorials here. Thanks for every thing which has been shared by you. It helped us to build our knowledge on hadoop. And we have shared some info here

    Hadoop Tutorial

    ReplyDelete
  2. My project is not working. I followed the steps given above, but i still get

    Exception in thread "main" java.lang.ClassNotFoundException:
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:249)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:201).

    Seems even the mainclass has not been found yet.
    i use the following command:
    hadoop jar ../lib/train_javaexample.jar train.examples.app.TypeSum -libjars=${LIBJAR} /user/whoiam/in /user/whoiam/out

    proj code:
    public class TypeSum extends Configured implements Tool{
    ...
    public final int run(final String[] args) throws Exception {
    Job job = new Job(super.getConf());
    job.setJarByClass(TypeSum.class);
    job.setMapperClass(TypeSumMapper.class);
    job.setCombinerClass(TypeSumReducer.class);
    job.setReducerClass(TypeSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    //FileInputFormat.addInputPath(job, new Path(args[0]));//???????????
    //FileOutputFormat.setOutputPath(job, new Path(args[1]));

    return (job.waitForCompletion(true)?0:1);
    }

    }
    Could you help please?

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Hadoop is really a good booming technology in coming days. And good path for the people who are looking for the Good Hikes and long career. We also provide Hadoop online training

    ReplyDelete
  5. Replies
    1. This comment has been removed by the author.

      Delete
    2. Thanks The information which you provided is very much useful for Pega Training v

      Delete
  6. I was reading your blog this morning and noticed that you have a awesome resource page. I actually have a similar blog that might be helpful or useful to your audience.

    Regards
    sap sd and crm online training
    sap online tutorials
    sap sd tutorial
    sap sd training in ameerpet
    sap crm training tutorial

    ReplyDelete
  7. Thanks The information which you provided is very much useful for Hybris Training v

    ReplyDelete
  8. Thank you,so thank of this tutorial! Hope you have more the lession like this!AWS Training
    Thank agains!

    ReplyDelete
  9. Thanks for sharing this information .You may also refer http://www.s4techno.com/hadoop-training-in-pune/

    ReplyDelete