尝试编写自己的Spark应用程序时,我开始遇到此问题,最后退而去,只是尝试使示例运行-我一直无法做到。
我跑了bin/run-example CassandraTest localhost localhost 9160
,遇到以下错误。(我bin/run-example CassandraTest localhost 9160
在那之前跑了,但是得到了ArrayOutOfBoundsException
,所以我只是把东西放到了第一位)
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)
at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:90)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.FlatMappedRDD.getPartitions(FlatMappedRDD.scala:30)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:59)
at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:370)
at org.apache.spark.examples.CassandraTest$.main(CassandraTest.scala:100)
at org.apache.spark.examples.CassandraTest.main(CassandraTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
我的配置:
sbt
或构建文件)。我没有对该文件夹中的任何文件进行任何更改。sudo bin/cassandra start
从我下载的未压缩二进制包开始使用)。我也尝试过1.2.6和2.0.6。我希望有人可以帮助我,这样我就可以克服这个错误,并开始在Cassandra中使用Spark。
您在这里有两个版本的hadoop。一种是带有二进制发行版Spark的软件包,另一种是嵌入在Cassandra中的软件包:org.apache.cassandra.hadoop...
您需要同时匹配两个版本才能拥有一个正常的系统。我不确定Cassandra支持哪个hadoop版本,但是鉴于您spark-1.0.0-bin-hadoop2
遇到的问题,我建议您尝试Hadoop v1的Spark发行版。
本文收集自互联网,转载请注明来源。
如有侵权,请联系[email protected] 删除。
我来说两句