In a recent popular sbt app https://github.com/databricks/reference-apps, I found a line that required me to
import AssemblyKeys._
This line doesn't compile in SBT or in my IntelliJ IDEA.
What is the import used for and why is it necessary?
The other answer by @mfirry pretty much answers what part of the build (definition) brings import AssemblyKeys._. It's the sbt-assembly plugin that (quoting the docs of the plugin):
Create a fat JAR of your project with all of its dependencies.
It's needed by the plugin to do its job.
You may ask yourself why I would need the plugin at all.
Since you didn't reference the application that requires the import and hence the plugin, and I didn't review the examples, either, I can only guess by the company Databricks that's the commercial entity behind Apache Spark the examples use.
In order to deploy an application onto a cluster of Apache Spark you need to assemble the entire application and configure the workers so they can access the binaries (with the necessary dependencies, namely jars). That's the aim of the sbt-assembly plugin to pack all up and offer a single application jar so Spark can use. See Standalone Applications in the documentation of Apache Spark to read about the standalone case (no cluster environment):
For sbt to work correctly, we’ll need to layout SimpleApp.scala and simple.sbt according to the typical directory structure. Once that is in place, we can create a JAR package containing the application’s code, then use the spark-submit script to run our program.
You may want to read Cluster Mode Overview to have an overview of how to deploy a Spark application onto a Spark cluster.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments