How to run a spark example program in Intellij IDEA

StephenBoesch

First on the command line from the root of the downloaded spark project I ran

mvn package

It was successful.

Then an intellij project was created by importing the spark pom.xml.

In the IDE the example class appears fine: all of the libraries are found. This can be viewed in the screenshot.

However , when attempting to run the main() a ClassNotFoundException on SparkContext occurs.

Why can Intellij not simply load and run this maven based scala program? And what can be done as a workaround?

As one can see below, the SparkContext is looking fine in the IDE: but then is not found when attempting to run: enter image description here

The test was run by right clicking inside main():

enter image description here

.. and selecting Run GroupByTest

It gives

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkContext
    at org.apache.spark.examples.GroupByTest$.main(GroupByTest.scala:36)
    at org.apache.spark.examples.GroupByTest.main(GroupByTest.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkContext
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

Here is the run configuration:

enter image description here

Yuriy

Spark lib isn't your class_path.

Execute sbt/sbt assembly,

and after include "/assembly/target/scala-$SCALA_VERSION/spark-assembly*hadoop*-deps.jar" to your project.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Running Apache Spark Example Application in IntelliJ Idea

From Dev

Run configuration with command line program in IntelliJ IDEA

From Dev

Different program arguments for 'run' and 'debug' in Intellij Idea

From Dev

How to work with Apache Spark using Intellij Idea?

From Dev

BEX error when trying to compile/run a program using Intellij IDEA

From Dev

how to run IntelliJ IDEA with JetBrains Runtime

From Dev

How to split the Run Tool Window in IntelliJ IDEA?

From Dev

How run all tests for class in Intellij Idea?

From Dev

How to run a Scala script within IntelliJ IDEA?

From Dev

Intellij IDEA how to run a class not the entire project?

From Dev

How to run Java scratch file in IntelliJ IDEA

From Dev

How run all tests for class in Intellij Idea?

From Dev

How to run Java program through the terminal in IntelliJ

From Dev

How to run my program in mac terminal (Intellij)

From Dev

How to run Spark scala application inside Intellij

From Dev

IntelliJ IDEA: How to run gradle application plugin run task with --args

From Dev

How to set multi-reducers for Hadoop Program in IntelliJ IDEA?

From Dev

How to set up logging level for Spark application in IntelliJ IDEA?

From Dev

IntelliJ IDEA program arguments (shortcut)

From Dev

How to run sbt-assembly tasks from within IntelliJ IDEA?

From Dev

How do I run an Applet within Intellij IDEA?

From Dev

How to build and Run Storm Topology within Intellij IDEA

From Dev

How to run all Specs2 tests under IntelliJ IDEA?

From Dev

How to run WildFly with standalone-full.xml from IntelliJ IDEA?

From Dev

How to show run tool only when there is an error in IntelliJ Idea?

From Dev

How to select the database to run an .sql file in IntelliJ IDEA?

From Dev

How can I setup and run/debug ElasticSearch in IntelliJ IDEA

From Dev

How can I setup and run/debug ElasticSearch in IntelliJ IDEA

From Dev

How to set a run/debug configuration in IntelliJ IDEA using a Jar file?

Related Related

  1. 1

    Running Apache Spark Example Application in IntelliJ Idea

  2. 2

    Run configuration with command line program in IntelliJ IDEA

  3. 3

    Different program arguments for 'run' and 'debug' in Intellij Idea

  4. 4

    How to work with Apache Spark using Intellij Idea?

  5. 5

    BEX error when trying to compile/run a program using Intellij IDEA

  6. 6

    how to run IntelliJ IDEA with JetBrains Runtime

  7. 7

    How to split the Run Tool Window in IntelliJ IDEA?

  8. 8

    How run all tests for class in Intellij Idea?

  9. 9

    How to run a Scala script within IntelliJ IDEA?

  10. 10

    Intellij IDEA how to run a class not the entire project?

  11. 11

    How to run Java scratch file in IntelliJ IDEA

  12. 12

    How run all tests for class in Intellij Idea?

  13. 13

    How to run Java program through the terminal in IntelliJ

  14. 14

    How to run my program in mac terminal (Intellij)

  15. 15

    How to run Spark scala application inside Intellij

  16. 16

    IntelliJ IDEA: How to run gradle application plugin run task with --args

  17. 17

    How to set multi-reducers for Hadoop Program in IntelliJ IDEA?

  18. 18

    How to set up logging level for Spark application in IntelliJ IDEA?

  19. 19

    IntelliJ IDEA program arguments (shortcut)

  20. 20

    How to run sbt-assembly tasks from within IntelliJ IDEA?

  21. 21

    How do I run an Applet within Intellij IDEA?

  22. 22

    How to build and Run Storm Topology within Intellij IDEA

  23. 23

    How to run all Specs2 tests under IntelliJ IDEA?

  24. 24

    How to run WildFly with standalone-full.xml from IntelliJ IDEA?

  25. 25

    How to show run tool only when there is an error in IntelliJ Idea?

  26. 26

    How to select the database to run an .sql file in IntelliJ IDEA?

  27. 27

    How can I setup and run/debug ElasticSearch in IntelliJ IDEA

  28. 28

    How can I setup and run/debug ElasticSearch in IntelliJ IDEA

  29. 29

    How to set a run/debug configuration in IntelliJ IDEA using a Jar file?

HotTag

Archive