How to exclude transitive dependency in Sbt ( in context of assembly plugin )?

Julias Published at Dev

Julias

I have two Sbt projects, my-commons and my-service.

my-commons

with the dependencies

libraryDependencies ++= Seq(
  "nz.ac.waikato.cms.weka" % "attributeSelectionSearchMethods" % "1.0.7",
  "de.bwaldvogel" % "liblinear" % "1.95"
  "io.dropwizard.metrics" % "metrics-graphite" % "3.1.2",
  "com.github.nscala-time" %% "nscala-time" % "2.2.0",
  "org.apache.hive" % "hive-jdbc" % "1.1.0-cdh5.4.5",
  "org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.4.5",
  "org.apache.hadoop" % "hadoop-hdfs" % "2.6.0-cdh5.4.5"
)

my-service:

with the dependencies

libraryDependencies ++= {
  Seq(
    "ch.qos.logback" % "logback-classic" % "1.0.13",
    "io.spray" %% "spray-httpx" % "1.3.3",
    "io.spray" %% "spray-json" % "1.3.2",
    "io.spray" %% "spray-can" % "1.3.3",
    "io.spray" %% "spray-routing" % "1.3.3",
    "io.spray" %% "spray-testkit" % "1.3.3" % "test",
    "com.typesafe.akka" %% "akka-actor" % "2.3.9",
    "com.typesafe.akka" %% "akka-testkit" % "2.3.9" % "test",
    "org.specs2" %% "specs2-core" % "2.3.11" % "test",
    "org.json4s" %% "json4s-native" %  "3.2.11",
    "org.json4s" %% "json4s-ext" %  "3.2.11",
    "org.mockito" % "mockito-all" % "1.8.4" % "test",
    "com.mycommon.projects" % "my-commons" % "1.0.+"
  )

I'm using the assembly sbt plugin

   addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

I'm getting error on sbt assembly:

at java.lang.Thread.run(Thread.java:745)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /home/me/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.7.jar:META-INF/maven/org.slf4j/slf4j-api/pom.properties
[error] /home/me/.ivy2/cache/com.twitter/parquet-hadoop-bundle/jars/parquet-hadoop-bundle-1.5.0-cdh5.4.5.jar:META-INF/maven/org.slf4j/slf4j-api/pom.properties

I've tried to exclude those libs from build but with no success.

libraryDependencies ~= { _ map {
  case m if m.organization.startsWith("org.apache") || m.organization.startsWith("com.twitter") || m.name.contains("parquet") =>
   m.exclude("org.slf4j","slf4j-api").
   exclude("org.slf4j","slf4j-log4j12")
   case m => m
 }}

Probably I'm doing something wrong... How can I resolve this dependency hell?

Martin Senne

In your case, files exist with different content and thus can't be merged with SBT assembly plugin.

Two approaches possible

Approach 1:

Use the merge filter of the assembly plugin to specify, which files to keep, e.g.

val sharedMergeStrategy: (String => MergeStrategy) => String => MergeStrategy =
  (old: (String) => MergeStrategy) => {
    case x if x.startsWith("META-INF/ECLIPSEF.RSA") => MergeStrategy.last
    case x if x.startsWith("META-INF/mailcap") => MergeStrategy.last
    case x if x.endsWith("plugin.properties") => MergeStrategy.last
    case x => old(x)
  }