Hadoop-WordCount的结果未在输出文件上写入

debugcn 发表于 Dev

巴尔旺·库玛·辛格（Balwant kumar singh）

我正在尝试通过执行此链接中给出的步骤来运行一个程序来计算单词数及其频率：http : //developer.yahoo.com/hadoop/tutorial/module3.html

我已经加载了一个名为input的目录，其中包括三个文本文件。

我能够正确配置所有内容。现在，在运行WordCount.java时，在输出目录中的part-00000文件中没有看到任何内容。

Mapper的Java代码为：

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordCountMapper extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {

private final IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(WritableComparable key, Writable value,
  OutputCollector output, Reporter reporter) throws IOException {

String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase());
while(itr.hasMoreTokens()) {
  word.set(itr.nextToken());
  output.collect(word, one);
}
}

@Override
public void map(LongWritable arg0, Text arg1,
    OutputCollector<Text, IntWritable> arg2, Reporter arg3)
     throws IOException {
// TODO Auto-generated method stub

 }

}

减少代码为：

public class WordCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterator values,
  OutputCollector output, Reporter reporter) throws IOException {

int sum = 0;
while (values.hasNext()) {
    //System.out.println(values.next());
  IntWritable value = (IntWritable) values.next();
  sum += value.get(); // process value
}

output.collect(key, new IntWritable(sum));
 }
 }

Word计数器的代码是：

public class Counter {

public static void main(String[] args) {
    JobClient client = new JobClient();
    JobConf conf = new JobConf(com.example.Counter.class);

    // TODO: specify output types
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    // TODO: specify input and output DIRECTORIES (not files)
    conf.setInputPath(new Path("src"));
    conf.setOutputPath(new Path("out"));

    // TODO: specify a mapper
    conf.setMapperClass(org.apache.hadoop.mapred.lib.IdentityMapper.class);

    // TODO: specify a reducer
    conf
                   .setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);

    client.setConf(conf);
    try {
        JobClient.runJob(conf);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

}

在控制台中，我得到以下日志：

13/09/10 10:09:20 WARN mapred.JobClient: Use GenericOptionsParser for parsing the       arguments. Applications should implement Tool for the same.
13/09/10 10:09:20 INFO mapred.FileInputFormat: Total input paths to process : 3
13/09/10 10:09:20 INFO mapred.FileInputFormat: Total input paths to process : 3
13/09/10 10:09:20 INFO mapred.JobClient: Running job: job_201309100855_0012
13/09/10 10:09:21 INFO mapred.JobClient:  map 0% reduce 0%
13/09/10 10:09:25 INFO mapred.JobClient:  map 25% reduce 0%
13/09/10 10:09:26 INFO mapred.JobClient:  map 75% reduce 0%
13/09/10 10:09:27 INFO mapred.JobClient:  map 100% reduce 0%
13/09/10 10:09:35 INFO mapred.JobClient: Job complete: job_201309100855_0012
13/09/10 10:09:35 INFO mapred.JobClient: Counters: 15
13/09/10 10:09:35 INFO mapred.JobClient:   File Systems
13/09/10 10:09:35 INFO mapred.JobClient:     HDFS bytes read=54049
13/09/10 10:09:35 INFO mapred.JobClient:     Local bytes read=14
13/09/10 10:09:35 INFO mapred.JobClient:     Local bytes written=214
13/09/10 10:09:35 INFO mapred.JobClient:   Job Counters 
13/09/10 10:09:35 INFO mapred.JobClient:     Launched reduce tasks=1
13/09/10 10:09:35 INFO mapred.JobClient:     Launched map tasks=4
13/09/10 10:09:35 INFO mapred.JobClient:     Data-local map tasks=4
13/09/10 10:09:35 INFO mapred.JobClient:   Map-Reduce Framework
13/09/10 10:09:35 INFO mapred.JobClient:     Reduce input groups=0
13/09/10 10:09:35 INFO mapred.JobClient:     Combine output records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map input records=326
13/09/10 10:09:35 INFO mapred.JobClient:     Reduce output records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map output bytes=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map input bytes=50752
13/09/10 10:09:35 INFO mapred.JobClient:     Combine input records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map output records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Reduce input records=0