Scala MapReduce Framework giving Type Mismatch

J Calbreath

I have a MapReduce framework in Scala that is based on several o fthe org.apache.hadoop libraries. It works great with a simple wordcount program. However, I want to apply it to something useful and am hitting a roadblock. I want to take a csv file (or any delimiter really) and pass whatever is in the 1st column as the key and subsequently count the incidence of keys.

The mapper code looks like this

class WordCountMapper extends Mapper[LongWritable, Text, Text, LongWritable] with HImplicits {
  protected override def map(lnNumber: LongWritable, line: Text, context: Mapper[LongWritable, Text, Text, LongWritable]#Context): Unit = {
  line.split(",", -1)(0) foreach (context.write(_,1))  //Splits data
  }
}

The problem comes in the 'line.split' code. When I try to compile it, I get an error that says:

found: char required:org.apache.hadoop.io.Text.

line.split... should return a string that is being passed to the _ in the write(_,1), but for soem reason itis thinking it is a char. I've even added .toString to explicitly make it a string but that didn't work either.

Any ideas are appreciated. Let me know what additional details I can provide.

Update:

Here is the list of imports:

import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapreduce.{Reducer, Job, Mapper}
import org.apache.hadoop.conf.{Configured}
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
import scala.collection.JavaConversions._
import org.apache.hadoop.util.{ToolRunner, Tool}

Here is the build.sbt code:

import AssemblyKeys._ // put this at the top of the file

assemblySettings

organization := "scala"

name := "WordCount"

version := "1.0"

scalaVersion:= "2.11.2"

scalacOptions ++= Seq("-no-specialization", "-deprecation")

libraryDependencies ++= Seq("org.apache.hadoop" % "hadoop-client" % "1.2.1",
                        "org.apache.hadoop" % "hadoop-core" % "latest.integration" exclude ("hadoop-core", "org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.class") ,
                        "org.apache.hadoop" % "hadoop-common" % "2.5.1",
                        "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.5.1",
                        "commons-configuration" % "commons-configuration" % "1.9",
                        "org.apache.hadoop" % "hadoop-hdfs" % "latest.integration")


 jarName in assembly := "WordCount.jar"

 mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {case s if s.endsWith(".class") => MergeStrategy.last
case s if s.endsWith(".xsd") => MergeStrategy.last
case s if s.endsWith(".dtd") => MergeStrategy.last
case s if s.endsWith(".xml") => MergeStrategy.last
case s if s.endsWith(".properties") => MergeStrategy.last
case x => old(x)
  }
}
J Calbreath

I actually solved this by not using the _ notation and just directly specifying the value in context.write. So instead of:

line.split(",", -1)(0) foreach (context.write(_,1))

I used:

context.write(line.split(",", -1)(0), 1)

I found a item online that said sometime Scala gets confused on data types when using the _, and recommending just explicitly defining the value in place. Not sure if that is true, but it solved the problem in this case.

이 기사는 인터넷에서 수집됩니다. 재 인쇄 할 때 출처를 알려주십시오.

침해가 발생한 경우 연락 주시기 바랍니다[email protected] 삭제

에서 수정
0

몇 마디 만하겠습니다

0리뷰
로그인참여 후 검토

관련 기사

분류에서Dev

scala 2.10: why there is a type mismatch?

분류에서Dev

Scala - type mismatch; found Int, required String

분류에서Dev

MapReduce or aggregation framework?

분류에서Dev

Type mismatch in criteria expression

분류에서Dev

Type mismatch: cannot convert from Scanner to boolean

분류에서Dev

Type mismatch when calling a function in qtp

분류에서Dev

Declare and use Array - Getting Type Mismatch

분류에서Dev

What is "Type mismatch" and how do I fix it?

분류에서Dev

ActionBarActivity cannot be resolved to a type, Jar mismatch : Eclipse

분류에서Dev

MS Access 2013 SQL Type mismatch error

분류에서Dev

Scala type filtering

분류에서Dev

Firebase and Play Framework (Scala) is it possible?

분류에서Dev

Using Jersey Test Framework in Scala

분류에서Dev

error : Type mismatch: cannot convert from Object to JSONObject

분류에서Dev

Run - Time Error '13': Type Mismatch - excel user form

분류에서Dev

Installer Custom Action Error - Type Mismatch: [string: "C:\"]

분류에서Dev

Type mismatch: cannot convert from Object to Class object

분류에서Dev

Color.RGBToHSV Type mismatch: cannot convert from void to float[]

분류에서Dev

Why does ternary operator fail with a type mismatch error?

분류에서Dev

Access Yes/No Field Data Type Mismatch with Boolean Value

분류에서Dev

Using parameterized query is causing data type mismatch. Any ideas?

분류에서Dev

VBA Application.Index with array causes type mismatch error 13

분류에서Dev

VBA .Formula Vlookup with Wildcard Returning Type Mismatch (Simple)

분류에서Dev

Scala Number type pattern matching

분류에서Dev

Play Framework: Inheritance sort by type

분류에서Dev

Scala Play framework 2.1 derived classes

분류에서Dev

Scala Play framework dependency in clean sbt project

분류에서Dev

Stackless Scala play framework runtime error

분류에서Dev

scala play framework reverse routing and controllers

Related 관련 기사

  1. 1

    scala 2.10: why there is a type mismatch?

  2. 2

    Scala - type mismatch; found Int, required String

  3. 3

    MapReduce or aggregation framework?

  4. 4

    Type mismatch in criteria expression

  5. 5

    Type mismatch: cannot convert from Scanner to boolean

  6. 6

    Type mismatch when calling a function in qtp

  7. 7

    Declare and use Array - Getting Type Mismatch

  8. 8

    What is "Type mismatch" and how do I fix it?

  9. 9

    ActionBarActivity cannot be resolved to a type, Jar mismatch : Eclipse

  10. 10

    MS Access 2013 SQL Type mismatch error

  11. 11

    Scala type filtering

  12. 12

    Firebase and Play Framework (Scala) is it possible?

  13. 13

    Using Jersey Test Framework in Scala

  14. 14

    error : Type mismatch: cannot convert from Object to JSONObject

  15. 15

    Run - Time Error '13': Type Mismatch - excel user form

  16. 16

    Installer Custom Action Error - Type Mismatch: [string: "C:\"]

  17. 17

    Type mismatch: cannot convert from Object to Class object

  18. 18

    Color.RGBToHSV Type mismatch: cannot convert from void to float[]

  19. 19

    Why does ternary operator fail with a type mismatch error?

  20. 20

    Access Yes/No Field Data Type Mismatch with Boolean Value

  21. 21

    Using parameterized query is causing data type mismatch. Any ideas?

  22. 22

    VBA Application.Index with array causes type mismatch error 13

  23. 23

    VBA .Formula Vlookup with Wildcard Returning Type Mismatch (Simple)

  24. 24

    Scala Number type pattern matching

  25. 25

    Play Framework: Inheritance sort by type

  26. 26

    Scala Play framework 2.1 derived classes

  27. 27

    Scala Play framework dependency in clean sbt project

  28. 28

    Stackless Scala play framework runtime error

  29. 29

    scala play framework reverse routing and controllers

뜨겁다태그

보관