java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to packagename.MyRecord

Markon

I am trying to use Spark 1.5.1 (with Scala 2.10.2) to read some .avro files from HDFS (with spark-avro 1.7.7) , in order to do some computation on them.

Now, starting with the assumption that I have already searched thoroughly the web to find a solution ( and the best link so far is this one that suggests to use a GenericRecord, while this one reports the same issue, and this one just does not work for me, because it gives almost the same code I have used ), I ask here, because it might be that someone had the same. This is the code :

import org.apache.avro.mapred.{AvroInputFormat, AvroWrapper} 
import org.apache.hadoop.io.NullWritable 
import org.apache.spark.{SparkConf, SparkContext}

object SparkPOC {

  def main(args: Array[String]): Unit ={

    val conf = new SparkConf()
      .setAppName("SparkPOC")
      .set("spark.master", "local[4]")
    val sc = new SparkContext(conf)
    val path = args(0)
    val profiles = sc.hadoopFile(
      path,
      classOf[AvroInputFormat[MyRecord]],
      classOf[AvroWrapper[MyRecord]],
      classOf[NullWritable]
    )

    val timeStamps = profiles.map{ p => p._1.datum.getTimeStamp().toString}
    timeStamps.foreach(print)

}

And I get the following message:

java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to packagename.MyRecord
    at packagename.SparkPOC$$anonfun$1.apply(SparkPOC.scala:24)
    at packagename.SparkPOC$$anonfun$1.apply(SparkPOC.scala:24)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:890)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:890)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

Does anybody have a clue? I was also considering the possibility of using spark-avro, but they don't support reading from multiple files at the same time (while .hadoopFile supports wildcards). Otherwise, it seems that I have to go for the GenericRecord and use the .get method, losing the advantage of the coded schema (MyRecord).

Thanks in advance.

Markon

The problem has gone after I have set KryoSerializer and a spark.kryo.registrator class, as follows:

val config = new SparkConf()
  .setAppName(appName)
  .set("spark.master", master)
  .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .set("spark.kryo.registrator", "com.mypackage.AvroKryoRegistrator")

where AvroKryoRegistrator is something like this.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

java.lang.ClassCastException: com.sun.faces.facelets.compiler.UIInstructions cannot be cast to org.primefaces.model.menu.MenuElement

From Dev

java.lang.ClassCastException: org.hibernate.ejb.HibernatePersistence cannot be cast to javax.persistence.spi.PersistenceProvider in WebLogic

From Dev

java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast

From Dev

parse.com java.lang.ClassCastException: java.util.HashMap cannot be cast to org.json.JSONObject

From Dev

Greenfoot: java.lang.ClassCastException: Obstacle cannot be cast to BallProjectile

From Dev

java.lang.ClassCastException: org.glassfish.jersey.servlet.ServletContainer cannot be cast to javax.servlet.Servlet

From Dev

java.lang.ClassCastException: cannot be cast to java.lang.Object

From Dev

java.lang.ClassCastException: JedisConnectionFactory cannot be cast to ConnectionFactory

From Dev

java.lang.ClassCastException: org.apache.commons.dbcp.BasicDataSource cannot be cast to org.apache.commons.dbcp.BasicDataSource

From Dev

java.lang.ClassCastException: org.apache.xml.dtm.ref.DTMManagerDefault cannot be cast to org.apache.xml.dtm.DTMManager

From Dev

java.lang.ClassCastException: org.springframework.security.core.userdetails.User cannot be cast to model.User

From Dev

Caused by: java.lang.ClassCastException: libcore.reflect.ParameterizedTypeImpl cannot be cast to java.lang.Class when use Generic Type

From Dev

wildfly 10: java.lang.ClassCastException: org.dom4j.DocumentFactory cannot be cast to org.dom4j.DocumentFactory

From Dev

java.lang.ClassCastException: org.apache.derby.client.am.ClientPreparedStatement42 cannot be cast to org.apache.derby.iapi.sql.PreparedStatement

From Dev

java.lang.ClassCastException: DTOObject cannot be cast to DTOObject

From Dev

java.lang.ClassCastException: java.lang.String cannot be cast to org.json.JSONObject

From Dev

java.lang.ClassCastException: CLASS/Activity cannot be cast to MainActivity

From Dev

java.lang.ClassCastException: org.hibernate.ejb.HibernatePersistence cannot be cast to javax.persistence.spi.PersistenceProvider in WebLogic

From Dev

java.lang.ClassCastException: org.json.JSONObject$1 cannot be cast to org.json.JSONObject while calling ParseCloud function

From Dev

Exception in thread "main" java.lang.ClassCastException: org.springframework.jdbc.datasource.DriverManagerDataSource cannot be cast

From Dev

java.lang.ClassCastException: org.apache.tomcat.dbcp.dbcp.PoolableConnection cannot be cast to oracle.jdbc.OracleConnection

From Dev

java.lang.ClassCastException: org.apache.derby.client.am.ClientPreparedStatement42 cannot be cast to org.apache.derby.iapi.sql.PreparedStatement

From Dev

Error: java.lang.ClassCastException: wordCountTest.WordCountTest cannot be cast to org.apache.hadoop.mapreduce.Mapper

From Dev

ClassCastException: JSONArray cannot be cast to java.lang.String[]

From Dev

How to apply UDF to collected Rows? (fails with "java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.sql.Column")

From Dev

Getting the Error - java.lang.ClassCastException: org.openqa.selenium.chrome.ChromeDriver cannot be cast to com.initialization.DriverInitialization

From Dev

How to resolve ClassCastException: java.lang.String cannot be cast exception

From Dev

java.lang.ClassCastException: org.openqa.selenium.By$ById cannot be cast to org.openqa.selenium.WebElement

From Dev

java.lang.ClassCastException (Cannot cast class to same class)

Related Related

  1. 1

    java.lang.ClassCastException: com.sun.faces.facelets.compiler.UIInstructions cannot be cast to org.primefaces.model.menu.MenuElement

  2. 2

    java.lang.ClassCastException: org.hibernate.ejb.HibernatePersistence cannot be cast to javax.persistence.spi.PersistenceProvider in WebLogic

  3. 3

    java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast

  4. 4

    parse.com java.lang.ClassCastException: java.util.HashMap cannot be cast to org.json.JSONObject

  5. 5

    Greenfoot: java.lang.ClassCastException: Obstacle cannot be cast to BallProjectile

  6. 6

    java.lang.ClassCastException: org.glassfish.jersey.servlet.ServletContainer cannot be cast to javax.servlet.Servlet

  7. 7

    java.lang.ClassCastException: cannot be cast to java.lang.Object

  8. 8

    java.lang.ClassCastException: JedisConnectionFactory cannot be cast to ConnectionFactory

  9. 9

    java.lang.ClassCastException: org.apache.commons.dbcp.BasicDataSource cannot be cast to org.apache.commons.dbcp.BasicDataSource

  10. 10

    java.lang.ClassCastException: org.apache.xml.dtm.ref.DTMManagerDefault cannot be cast to org.apache.xml.dtm.DTMManager

  11. 11

    java.lang.ClassCastException: org.springframework.security.core.userdetails.User cannot be cast to model.User

  12. 12

    Caused by: java.lang.ClassCastException: libcore.reflect.ParameterizedTypeImpl cannot be cast to java.lang.Class when use Generic Type

  13. 13

    wildfly 10: java.lang.ClassCastException: org.dom4j.DocumentFactory cannot be cast to org.dom4j.DocumentFactory

  14. 14

    java.lang.ClassCastException: org.apache.derby.client.am.ClientPreparedStatement42 cannot be cast to org.apache.derby.iapi.sql.PreparedStatement

  15. 15

    java.lang.ClassCastException: DTOObject cannot be cast to DTOObject

  16. 16

    java.lang.ClassCastException: java.lang.String cannot be cast to org.json.JSONObject

  17. 17

    java.lang.ClassCastException: CLASS/Activity cannot be cast to MainActivity

  18. 18

    java.lang.ClassCastException: org.hibernate.ejb.HibernatePersistence cannot be cast to javax.persistence.spi.PersistenceProvider in WebLogic

  19. 19

    java.lang.ClassCastException: org.json.JSONObject$1 cannot be cast to org.json.JSONObject while calling ParseCloud function

  20. 20

    Exception in thread "main" java.lang.ClassCastException: org.springframework.jdbc.datasource.DriverManagerDataSource cannot be cast

  21. 21

    java.lang.ClassCastException: org.apache.tomcat.dbcp.dbcp.PoolableConnection cannot be cast to oracle.jdbc.OracleConnection

  22. 22

    java.lang.ClassCastException: org.apache.derby.client.am.ClientPreparedStatement42 cannot be cast to org.apache.derby.iapi.sql.PreparedStatement

  23. 23

    Error: java.lang.ClassCastException: wordCountTest.WordCountTest cannot be cast to org.apache.hadoop.mapreduce.Mapper

  24. 24

    ClassCastException: JSONArray cannot be cast to java.lang.String[]

  25. 25

    How to apply UDF to collected Rows? (fails with "java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.sql.Column")

  26. 26

    Getting the Error - java.lang.ClassCastException: org.openqa.selenium.chrome.ChromeDriver cannot be cast to com.initialization.DriverInitialization

  27. 27

    How to resolve ClassCastException: java.lang.String cannot be cast exception

  28. 28

    java.lang.ClassCastException: org.openqa.selenium.By$ById cannot be cast to org.openqa.selenium.WebElement

  29. 29

    java.lang.ClassCastException (Cannot cast class to same class)

HotTag

Archive