두 번째 정렬을 수행하는 동안 hadoop.im의 초보자가 null 포인터 예외를 얻었습니다.
이것은 내 매퍼 클래스입니다.
public void map(LongWritable key, Text value,
OutputCollector<Text, Employee> outputCollector, Reporter reporter)
throws IOException {
// TODO Auto-generated method stub
String employeeId = value.toString().split(",")[0];
String employeeName= value.toString().split(",")[1];
String employeeDept= value.toString().split(",")[2];
String employeejoinDate= value.toString().split(",")[3];
String employeSalary= value.toString().split(",")[4];
//System.out.println(employeSalary);
Employee employee=new Employee(Integer.parseInt(employeeId),employeeName,employeeDept,employeejoinDate,Integer.parseInt(employeSalary));
outputCollector.collect(new Text(employeeName),employee);
}
이것은 내 감속기입니다
public void reduce(Text arg0, Iterator<Employee> arg1,
OutputCollector<NullWritable,IntWritable> arg2, Reporter arg3)
throws IOException {
// TODO Auto-generated method stub
System.out.println("inside reducer");
while(arg1.hasNext()){
arg2.collect(NullWritable.get(),new IntWritable(arg1.next().getEmployeeSalary()));
}
이것은 내 직원 클래스입니다
public class Employee implements WritableComparable<Employee>{
private int employeeId;
private String employeeName;
private String employeeDept;
private String employeeJoinDt;
private int employeeSalary;
public Employee(int employeeId,String employeeName,String employeeDept,String employeeJoinDt,int employeeSalary){
this.employeeId=employeeId;
this.employeeName=employeeName;
this.employeeDept=employeeDept;
this.employeeJoinDt=employeeJoinDt;
this.employeeSalary=employeeSalary;
}
public int getEmployeeId() {
return employeeId;
}
public void setEmployeeId(int employeeId) {
this.employeeId = employeeId;
}
public String getEmployeeName() {
return employeeName;
}
public void setEmployeeName(String employeeName) {
this.employeeName = employeeName;
}
public String getEmployeeDept() {
return employeeDept;
}
public void setEmployeeDept(String employeeDept) {
this.employeeDept = employeeDept;
}
public String getEmployeeJoinDt() {
return employeeJoinDt;
}
public void setEmployeeJoinDt(String employeeJoinDt) {
this.employeeJoinDt = employeeJoinDt;
}
public int getEmployeeSalary() {
return employeeSalary;
}
public void setEmployeeSalary(int employeeSalary) {
this.employeeSalary = employeeSalary;
}
@Override
public void readFields(DataInput input) throws IOException {
// TODO Auto-generated method stubt
this.employeeId=input.readInt();
this.employeeName=input.readUTF();
this.employeeDept=input.readUTF();
this.employeeJoinDt=input.readUTF();
this.employeeSalary=input.readInt();
}
@Override
public void write(DataOutput output) throws IOException {
// TODO Auto-generated method stub
output.writeInt(this.employeeId);
output.writeUTF(this.employeeName);
output.writeUTF(this.employeeDept);
output.writeUTF(this.employeeJoinDt);
output.writeInt(this.employeeSalary);
}
public int compareTo(Employee employee) {
// TODO Auto-generated method stub
if(this.employeeSalary>employee.getEmployeeSalary())
return 1;
else if(this.employeeSalary<employee.getEmployeeSalary())
return -1;
else
return 0;
}
}
이것은 내 정렬 비교기 클래스입니다
public class SecondarySortComparator extends WritableComparator {
public SecondarySortComparator(){
super(Employee.class);
System.out.println("sort");
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
// TODO Auto-generated method stub
Employee employee1 = (Employee)a;
Employee employee2 = (Employee)b;
int i = employee1.getEmployeeSalary()>employee2.getEmployeeSalary()?1:-1;
return i;
}
이것은 내 그룹 비교기 클래스입니다
public class SecondarySortGroupingComparator extends WritableComparator{
public SecondarySortGroupingComparator(){
super(Employee.class,true);
System.out.println("group");
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
// TODO Auto-generated method stub
Employee employee1 = (Employee)a;
Employee employee2 = (Employee)b;
return employee1.getEmployeeName().compareTo(employee2.getEmployeeName());
}
}
이것은 오류입니다.
13/09/01 19:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/01 19:13:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/01 19:13:47 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/09/01 19:13:47 INFO mapred.FileInputFormat: Total input paths to process : 1
13/09/01 19:13:47 INFO mapred.JobClient: Running job: job_local_0001
13/09/01 19:13:47 INFO util.ProcessTree: setsid exited with exit code 0
13/09/01 19:13:47 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b3f8f6
13/09/01 19:13:47 INFO mapred.MapTask: numReduceTasks: 1
13/09/01 19:13:47 INFO mapred.MapTask: io.sort.mb = 100
13/09/01 19:13:48 INFO mapred.JobClient: map 0% reduce 0%
13/09/01 19:13:48 INFO mapred.MapTask: data buffer = 79691776/99614720
sort13/09/01 19:13:48 INFO mapred.MapTask: record buffer = 262144/327680
1
1
1
1
13/09/01 19:13:49 INFO mapred.MapTask: Starting flush of map output
13/09/01 19:13:49 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
13/09/01 19:13:49 INFO mapred.JobClient: Job complete: job_local_0001
13/09/01 19:13:49 INFO mapred.JobClient: Counters: 0
13/09/01 19:13:49 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at secondarysort.JobRunner.main(JobRunner.java:31)
이 문제를 해결하는 방법에 대한 제안이 있습니까?
미리 감사드립니다
이 선이 문제를 일으키는 것 같습니다.
context.write(new Text(employeeName), employee);
직원 개체 (Employee 유형)를 키가 아닌 값으로 내보내고 SecondarySortComparator 및 SecondarySortGroupingComparator 모두 값이 아닌 키에 대해 작동합니다.
따라서 주요 문제는 텍스트를 키로 전달하고 문제를 일으키는 것입니다. 두 비교기가 실제로 작동하도록 직원 개체를 텍스트 대신 키로 전달하는 것을 고려할 수 있습니다.
Employee 클래스에 기본 생성자를 넣을 수도 있습니다.
public Employee() { }
이 기사는 인터넷에서 수집됩니다. 재 인쇄 할 때 출처를 알려주십시오.
침해가 발생한 경우 연락 주시기 바랍니다[email protected] 삭제
몇 마디 만하겠습니다