sbt / sbt-assembly

Deploy über-JARs. Restart processes. (port of codahale/assembly-sbt)

sbt-assembly

Deploy über JARs. Restart processes.

sbt-assembly is a sbt plugin originally ported from codahale's assembly-sbt, which I'm guessing was inspired by Maven's assembly plugin. The goal is simple: Create a über JAR of your project with all of its dependencies.

Requirements

• sbt
• The burning desire to have a simple deploy procedure.

Reporting Issues & Contributing

Before you email me, please read Issue Reporting Guideline carefully. Twice. (Don't email me)

Setup

Using Published Plugin

Add sbt-assembly as a dependency in project/plugins.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "x.y.z")

Starting in sbt-assembly 1.2.0, sbt 0.13.x has been deprecated. Please use 1.1.0 if this is required.

Usage

Since sbt-assembly is now an auto plugin that's triggered for all projects with JvmPlugin, it shouldn't require extra setup to include assembly task into your project. See migration guide for details on how to upgrade from older sbt-assembly.

Applying the plugin to multi-project build.sbt

For example, here's a multi-project build.sbt:

ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / organization := "com.example"
ThisBuild / scalaVersion := "2.13.11"

lazy val app = (project in file("app"))
  .settings(
    assembly / mainClass := Some("com.example.Main"),
    // more settings here ...
  )

lazy val utils = (project in file("utils"))
  .settings(
    assembly / assemblyJarName := "utils.jar",
    // more settings here ...
  )

In the above example, both the app project and the utils project do not run tests during assembly. The app project sets a main class whereas the utils project sets the name of its jar file.

assembly task

Now you'll have an awesome new assembly task which will compile your project, run your tests, and then pack your class files and all your dependencies into a single JAR file: target/scala_X.X.X/projectname-assembly-X.X.X.jar.

> assembly

If you specify a assembly / mainClass in build.sbt (or just let it autodetect one) then you'll end up with a fully executable JAR, ready to rock.

Here is the list of the keys you can rewire that are scoped to current subproject's assembly task:

assemblyJarName               test                          mainClass
assemblyOutputPath            assemblyOption

And here is the list of the keys you can rewite that are scoped globally:

assemblyAppendContentHash     assemblyCacheOutput           assemblyShadeRules
assemblyExcludedJars          assemblyMergeStrategy         assemblyRepeatableBuild

Keys scoped to the subproject should be placed in .settings(...) whereas the globally scoped keys can either be placed inside of .settings(...) or scoped using ThisBuild / to be shared across multiple subprojects.

For example the name of the jar can be set as follows in build.sbt:

lazy val app = (project in file("app"))
  .settings(
    assembly / assemblyJarName := "something.jar",
    // more settings here ...
  )

To set an explicit main class,

lazy val app = (project in file("app"))
  .settings(
    assembly / mainClass := Some("com.example.Main"),
    // more settings here ...
  )

To run the test during assembly,

lazy val app = (project in file("app"))
  .settings(
    assembly / test := (Test / test).value,
    // more settings here ...
  )

Excluding an explicit main class from your assembly requires something a little bit different though

lazy val app = (project in file("app"))
  .settings(
    assembly / packageOptions ~= { pos =>
      pos.filterNot { po =>
        po.isInstanceOf[Package.MainClass]
      }
    },
    // more settings here ...
  )

Merge Strategy

If multiple files share the same relative path (e.g. a resource named application.conf in multiple dependency JARs), the default strategy is to verify that all candidates have the same contents and error out otherwise. This behavior can be configured on a per-path basis using either one of the following built-in strategies or writing a custom one:

• MergeStrategy.deduplicate is the default described above
• MergeStrategy.first picks the first of the matching files in classpath order
• MergeStrategy.last picks the last one
• MergeStrategy.singleOrError bails out with an error message on conflict
• MergeStrategy.concat simply concatenates all matching files and includes the result. There is also an overload that accepts a line separator for formatting the result
• MergeStrategy.filterDistinctLines also concatenates, but leaves out duplicates along the way. There is also an overload that accepts a Charset for reading the lines
• MergeStrategy.rename renames the files originating from jar files
• MergeStrategy.discard simply discards matching files
• MergeStrategy.preferProject will choose the first project file over library files if present. Otherwise, it works like MergeStrategy.first

The mapping of path names to merge strategies is done via the setting assemblyMergeStrategy which can be augmented as follows:

ThisBuild / assemblyMergeStrategy := {
  case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
  case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
  case "application.conf"                            => MergeStrategy.concat
  case "unwanted.txt"                                => MergeStrategy.discard
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

NOTE:

• Actually, a merge strategy serves two purposes:

  • To merge conflicting files
  • To transform a single file (despite the naming), such as in the case of a MergeStrategy.rename. Sometimes, the transformation is a pass-through, as in the case of a MergeStrategy.deduplicate if there are no conflicts on a target path.

• ThisBuild / assemblyMergeStrategy expects a function. You can't do ThisBuild / assemblyMergeStrategy := MergeStrategy.first!

• Some files must be discarded or renamed otherwise to avoid breaking the zip (due to duplicate file name) or the legal license. Delegate default handling to (ThisBuild / assemblyMergeStrategy) as the above pattern matching example.

• Renames are processed first, since renamed file targets might match more merge patterns afterwards. By default, LICENSEs and READMEs are renamed before applying every other merge strategy. If you need a custom logic for renaming, create a new rename merge strategy so it is processsed first, along with the custom logic. See how to create custom MergeStrategys in a later section of this README.

• There is an edge case that may occasionally fail. If a project has a file that has the same relative path as a directory to be written, an error notification will be written to the console as shown below. To resolve this, create a shade rule or a new merge strategy.

    [error] Files to be written at 'shadeio' have the same name as directories to be written:
    [error]   Jar name = commons-io-2.4.jar, jar org = commons-io, entry target = shadeio/input/Tailer.class (from original source = org/apache/commons/io/input/Tailer.class)
    [error]   Project name = foo, target = shadeio

By the way, the first case pattern in the above using PathList(...) is how you can pick javax/servlet/* from the first jar. If the default MergeStrategy.deduplicate is not working for you, that likely means you have multiple versions of some library pulled by your dependency graph. The real solution is to fix that dependency graph. You can work around it by MergeStrategy.first but don't be surprised when you see ClassNotFoundException.

Here is the default:

  val defaultMergeStrategy: String => MergeStrategy = {
    case x if Assembly.isConfigFile(x) =>
      MergeStrategy.concat
    case PathList(ps @ _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
      MergeStrategy.rename
    case PathList("META-INF", xs @ _*) =>
      (xs map {_.toLowerCase}) match {
        case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
          MergeStrategy.discard
        case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
          MergeStrategy.discard
        case "plexus" :: xs =>
          MergeStrategy.discard
        case "services" :: xs =>
          MergeStrategy.filterDistinctLines
        case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
          MergeStrategy.filterDistinctLines
        case _ => MergeStrategy.deduplicate
      }
    case _ => MergeStrategy.deduplicate
  }

Creating a custom Merge Strategy (since 2.0.0)

Custom merge strategies can be plugged-in to the assemblyMergeStrategy function, for example:

...
ThisBuild / assemblyMergeStrategy := {
  case "matching-file" => CustomMergeStrategy("my-custom-merge-strat") { conflicts =>
    // NB! same as MergeStrategy.discard
    Right(Vector.empty)
  }
  case x   =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}
...

The CustomMergeStrategy accepts a name and a notifyIfGTE that affects how the result is reported in the logs. Please see the scaladoc for more details.

Finally, to perform the actual merge/transformation logic, a function has to be provided. The function accepts a Vector of Dependency, where you can access the target of type String and the byte payload of type LazyInputStream, which is just a type alias for () => InputStream.

The input Dependency also has two subtypes that you can pattern match on:

• Project represents an internal/project dependency
• Library represents an external/library dependency that also contains the ModuleCoordinate (jar org, name and version) it originated from

To create a merge result, a Vector of JarEntry must be returned wrapped in an Either.Right, or empty to discard these conflicts from the final jar. JarEntry only has two fields, a target of type String and the byte payload of type lazy InputStream.

To fail the assembly, return an Either.Left with an error message.

There is also a factory specifically for renames, so it gets processed first along with the built-in rename merge strategy, before other merge strategies, as mentioned in a previous section. It accepts a function Dependency -> String, so the Dependency can be inspected and a new target path returned.

Here is an example that appends a String to the original target path of the matched file.

...
case "matching-file" =>
  import sbtassembly.Assembly.{Project, Library}
  CustomMergeStrategy.rename {
    case dependency@(_: Project) => dependency.target + "_from_project"
    case dependency@(_: Library) => dependency.target + "_from_library"
  }
...

For more information/examples, see the scaladoc/source code in sbtassembly.Assembly and sbtassembly.MergeStrategy.

NOTE:

• The name parameter will be distinguished from a built-in strategy. For example, the name=First will execute its custom logic along with the built-in MergeStrategy.first. They cannot cancel/override one another. In fact, the custom merge strategy will be logged as First (Custom) for clarity.
• However, you should still choose a unique name for a custom merge strategy within the build. Even if all built-in and custom merge strategies are guaranteed to execute if they match a pattern regardless of their names, similarly-named custom merge strategies will have their log reports joined. YMMV, but you are encouraged to avoid duplicate names.

Third Party Merge Strategy Plugins

Support for special-case merge strategies beyond the generic scope can be provided by companion plugins, below is a non-exhaustive list:

• Log4j2 Plugin Caches: merge Log4j2Plugins.dat binaries

Shading

sbt-assembly can shade classes from your projects or from the library dependencies. Backed by Jar Jar Links, bytecode transformation (via ASM) is used to change references to the renamed classes.

ThisBuild / assemblyShadeRules := Seq(
  ShadeRule.rename("org.apache.commons.io.**" -> "shadeio.@1").inAll
)

Here are the shade rules:

• ShadeRule.rename("x.**" -> "y.@1", ...).inAll This is the main rule.
• ShadeRule.zap("a.b.c").inAll
• ShadeRule.keep("x.**").inAll

The main ShadeRule.rename rule is used to rename classes. All references to the renamed classes will also be updated. If a class name is matched by more than one rule, only the first one will apply. The rename rules takes a vararg of String pairs in <pattern> -> <result> format:

• <pattern> is a class name with optional wildcards. ** will match against any valid class name substring. To match a single package component (by excluding . from the match), a single * may be used instead.
• <result> is a class name which can optionally reference the substrings matched by the wildcards. A numbered reference is available for every * or ** in the <pattern>, starting from left to right: @1, @2, etc. A special @0 reference contains the entire matched class name.

Instead of .inAll, call .inProject to match your project source, or call .inLibrary("commons-io" % "commons-io" % "2.4", ...) to match specific library dependencies. inProject and inLibrary(...) can be chained.

ThisBuild / assemblyShadeRules := Seq(
  ShadeRule.rename("org.apache.commons.io.**" -> "shadeio.@1").inLibrary("commons-io" % "commons-io" % "2.4", ...).inProject
)

The ShadeRule.zap rule causes any matched class to be removed from the resulting jar file. All zap rules are processed before renaming rules.

The ShadeRule.keep rule marks all matched classes as "roots". If any keep rules are defined all classes which are not reachable from the roots via dependency analysis are discarded when writing the output jar. This is the last step in the process, after renaming and zapping.

To see the verbose output for shading:

lazy val app = (project in file("app"))
  .settings(
    assembly / logLevel := Level.Debug
    // more settings here ...
  )

Scala libraries

Scala classes contain an annotation which, among other things, contain all symbols referenced in that class. As of sbt-assembly XXX the rename rules will be applied to these annotations as well which makes it possible to compile or reflect against a shaded library.

This is currently limited to renaming packages. Renaming class names will not work and cause compiler errors when compiling against the shaded library.

Excluding JARs and files

If you need to tell sbt-assembly to ignore JARs, you're probably doing it wrong. assembly task grabs deps JARs from your project's classpath. Try fixing the classpath first.

% "provided" configuration

If you're trying to exclude JAR files that are already part of the container (like Spark), consider scoping the dependent library to "provided" configuration:

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "0.8.0-incubating" % "provided",
  "org.apache.hadoop" % "hadoop-client" % "2.0.0-cdh4.4.0" % "provided"
)

Maven defines "provided" as:

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

The dependency will be part of compilation and test, but excluded from the runtime. If you're using Spark and want to include "provided" dependencies back to run, @douglaz has come up with a one-liner solution on StackOverflow sbt: how can I add "provided" dependencies back to run/test tasks' classpath?:

Compile / run := Defaults.runTask(Compile / fullClasspath, Compile / run / mainClass, Compile / run / runner).evaluated

Exclude specific transitive deps

You might be thinking about excluding JAR files because of the merge conflicts. Merge conflict of *.class files indicate pathological classpath, often due to non-modular bundle JAR files or SLF4J, not the problem with assembly. Here's what happens when you try to create a über JAR with Spark included:

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/foo/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-2.5.0.v201103041518.jar:javax/servlet/SingleThreadModel.class
[error] /Users/foo/.ivy2/cache/org.mortbay.jetty/servlet-api/jars/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class

In the above case two separate JAR files javax.servlet-2.5.0.v201103041518.jar and servlet-api-2.5-20081211.jar are defining javax/servlet/SingleThreadModel.class! Similarly also conflicts on common-beanutils and EsotericSoftware/minlog. Here's how to evict specific transitive deps:

libraryDependencies ++= Seq(
  ("org.apache.spark" %% "spark-core" % "0.8.0-incubating").
    exclude("org.mortbay.jetty", "servlet-api").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-logging", "commons-logging").
    exclude("com.esotericsoftware.minlog", "minlog")
)

See sbt's Exclude Transitive Dependencies for more details.

Sometimes it takes a bit of detective work to figure out which transitive deps to exclude. Play! comes with dist task, so assembly is not needed, but suppose we wanted to run assembly. It brings in signpost-commonshttp4, which leads to commons-logging. This conflicts with jcl-over-slf4j, which re-implements the logging API. Since the deps are added via build.sbt and playScalaSettings, here's one way to work around it:

libraryDependencies ~= { _ map {
  case m if m.organization == "com.typesafe.play" =>
    m.exclude("commons-logging", "commons-logging").
      exclude("com.typesafe.play", "sbt-link")
  case m => m
}}

Excluding specific files

To exclude specific files, customize merge strategy:

ThisBuild / assemblyMergeStrategy := {
  case PathList("about.html") => MergeStrategy.rename
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

Splitting your project and deps JARs

To make a JAR file containing only the external dependencies, type

> assemblyPackageDependency

This is intended to be used with a JAR that only contains your project

lazy val app = (project in file("app"))
  .settings(
    assemblyPackageScala / assembleArtifact := false,
    assemblyPackageDependency / assembleArtifact := false,

    // or as follows
    assembly / assemblyOption ~= {
      _.withIncludeScala(false)
       .withIncludeDependency(false)
    },

    // more settings here ...
  )

NOTE: If you use -jar option for java, it will ignore -cp, so if you have multiple JAR files you have to use -cp and pass the main class: java -cp "jar1.jar:jar2.jar" Main

Excluding Scala library JARs

To exclude Scala library (JARs that start with scala- and are included in the binary Scala distribution) to run with scala command,

lazy val app = (project in file("app"))
  .settings(
    assemblyPackageScala / assembleArtifact := false,

    // or as follows
    assembly / assemblyOption ~= {
      _.withIncludeScala(false)
    },

    // more settings here ...
  )

assemblyExcludedJars

If all efforts fail, here's a way to exclude JAR files:

lazy val app = (project in file("app"))
  .settings(
    assembly / assemblyExcludedJars := {
      val cp = (assembly / fullClasspath).value
      cp filter {_.data.getName == "compile-0.1.0.jar"}
    },

    // more settings here ...
  )

Other Things

Content hash

You can also append SHA-1 fingerprint to the assembly file name, this may help you to determine whether it has changed and, for example, if it's necessary to deploy the dependencies,

ThisBuild / assemblyAppendContentHash := true

// or
lazy val app = (project in file("app"))
  .settings(
     assembly / assemblyOption ~= { _.withAppendContentHash(true) }
  )

Caching

Caching is implemented by checking all the input dependencies (class and jar files)' latest timestamp and some configuration changes from the build file.

In addition the über JAR is cached so its timestamp changes only when the input changes.

To disable caching:

ThisBuild / assemblyCacheOutput := false

// or
lazy val app = (project in file("app"))
  .settings(
     assembly / assemblyOption ~= { _.withCacheOutput(false) }
  )

NOTE:

• Unfortunately, using a custom MergeStrategy other than rename will create a function in which the plugin cannot predict the outcome. This custom function must always be executed if it matches a PathList pattern, and thus, will disable caching.

Jar assembly performance

By default, the setting key assemblyRepeatableBuild is set to true. This ensures that the jar entries are assembled in a specific order, resulting in a consistent hash for the jar.

There is actually a performance improvement to be gained if this setting is set to false, since jar entries will now be assembled in parallel. The trade-off is, the jar will not have a consistent hash, and thus, caching will not work.

To set the repeatable build to false:

ThisBuild / assemblyRepeatableBuild := false

If a repeatable build/consistent jar is not of much importance, one may avail of this feature for improved performance, especially for large projects.

Prepending a launch script

Your can prepend a launch script to the über jar. This script will be a valid shell and batch script and will make the jar executable on Unix and Windows. If you enable the shebang the file will be detected as an executable under Linux but this will cause an error message to appear on Windows. On Windows just append a ".bat" to the files name to make it executable.

import sbtassembly.AssemblyPlugin.defaultUniversalScript

ThisBuild / assemblyPrependShellScript := Some(defaultUniversalScript(shebang = false))

lazy val app = (project in file("app"))
  .settings(
     assembly / assemblyJarName := s"${name.value}-${version.value}"
  )

This will prepend the following shell script to the jar.

(#!/usr/bin/env sh)
@ 2>/dev/null # 2>nul & echo off & goto BOF
:
exec java -jar $JAVA_OPTS "$0" "$@"
exit

:BOF
@echo off
java -jar %JAVA_OPTS% "%~dpnx0" %*
exit /B %errorlevel%

You can also choose to prepend just the shell script to the über jar as follows:

import sbtassembly.AssemblyPlugin.defaultShellScript

ThisBuild / assemblyPrependShellScript := Some(defaultShellScript)

lazy val app = (project in file("app"))
  .settings(
     assembly / assemblyJarName := s"${name.value}-${version.value}"
  )

Publishing (Not Recommended)

We discourage you from publishing non-shaded über JARs beyond deployment. The moment your über JAR is used as a library, it becomes a parasitized über JAR, bringing in parasite libraries that can not be excluded or resolved. One might think non-modularity is convenience, but it turns into others' headache down the road.

Here are some example of parasitized über JARs:

• hive-exec 2.3.9 and 3.1.3 contain com.google.common, com.google.protobuf, io.airlift, org.apache.parquet, org.codehaus.jackson, org.joda.time, etc.

Q: Despite the concerned friends, I still want publish über JARs. What advice do you have?

Shade everything. Next, you would likely need to set up a front business to lie about what dependencies you have in pom.xml and ivy.xml. To do so, make a subproject for über JAR purpose only where you depend on the dependencies, and make a second cosmetic subproject that you use only for publishing purpose:

lazy val uberJar = project
  .enablePlugins(AssemblyPlugin)
  .settings(
    depend on the good stuff
    publish / skip := true
  )

lazy val cosmetic = project
  .settings(
    name := "shaded-something",
    Compile / packageBin := (uberJar / assembly).value
  )

License

Published under The MIT License, see LICENSE

Copyright e.e d3si9n, LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

sbt-assembly's Issues

license: file exists and is not a directory for licenses

I am switching from mvn assembly to sbt assembly and I get this error:

{file:/Users/tamir/work/jnomicss/jnomicss-cli/}default-5dc2c4/*:assembly: Could not create directory /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f/license: file exists and is not a directory.

Do you have a quick workaround? I don't need any licences now (its completely internal).

excluding them in build.sbt like this does not help:

excludedFiles in assembly := { (bases: Seq[File]) =>
  bases flatMap { base =>
    (base / "META-INF" * "*").get collect {
      case f if f.getName == "something" => f
      case f if f.getName.toLowerCase == "license" => f
      case f if f.getName.toLowerCase == "manifest.mf" => f
    }
  }}

the temporary folders actually don't extis:

127 [email protected] ~> ls /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f/license
ls: /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f/license: No such file or directory
1 [email protected] ~> ls /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f
ls: /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f: No such file or directory

Update README for SBT 0.12-RC

I'm using SBT 0.12.0-RC4, and in project/plugins.sbt it's enough to add this instruction:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.8.3")

Hence, the README should be updated. The instruction specified in the README for the -Beta2, instead, causes an AbstractMethodError - apparently the API changed.

NoClassDefFoundError: scala/App

Hi,

my generated assembly gives:

Exception in thread "main" java.lang.NoClassDefFoundError: scala/App
...
Caused by: java.lang.ClassNotFoundException: scala.App

What should I do to include Scala dependencies?

Thanks,
Etam.

Settings aren't being respected in full configuration

I'm having trouble with getting my settings used. For example, my mainClass setting appears to not be found.

Here's the configuration I'm using:

45     val yggdrasilSettings = serviceSettings ++ Seq(
46       resolvers ++= Seq(
47         "riptano" at "http://mvn.riptano.com/content/repositories/public",
48         "Scale7 Maven Repo" at "https://github.com/s7/mvnrepo/raw/master"
49       ),
50       libraryDependencies ++= Seq(
51         "org.scale7" % "scale7-pelops" % "1.2-0.8.x-SNAPSHOT",
52         "joda-time"               % "joda-time"    % "1.6.2",
53         "org.scalaz"              %% "scalaz-core" % "6.0.1"
54       ),
55       mainClass := Some("com.reportgrid.yggdrasil.Yggdrasil"),
56       jarName in Assembly <<= version apply { v => "yggdrasil-v" + v.substring(0, v.indexOf(".")) }
57     )
58 
59     val yggdrasil = Project("yggdrasil", file("yggdrasil"), settings = yggdrasilSettings) dependsOn(common) dependsOnAlt (blueeyes(base))

I've tried both mainClass and mainClass in Assembly, and neither seems to work:

[knuttycombe@floorshow yggdrasil (master)]$ java -jar target/yggdrasil-assembly-0.3.jar --configFile yggdrasil.conf 
Failed to load Main-Class manifest attribute from
target/yggdrasil-assembly-0.3.jar

As you can see, my jarName is also not being respected. Any ideas?

caching (sbt assembly touches target jar file even when source code doesn't change)

I want to use sbt assembly in conjunction with a bigger build process that is managed by make so it would nice if the timestamp on the jar that sbt-assembly produces would only change when necessary.

Provide (default?) merge strategy for META-INF/NOTICE and LICENSE

It would be cool if there would be a merge strategy which is similar to concat but which writes the name of the jar file where ithe file comes from above the actual content. Like this:

netty-x.x.x.jar

[license text]

httpclient.jar

[license text]

excluded files is not working correctly

We are trying to create Dropwizard-based (https://github.com/codahale/dropwizard) services, which uses Jersey and depends on providers defined under META-INF/services. It looks like these should be getting included based on the PathFinder in assemblyExcludedFiles, but it's not appearing to work for us. I have only been able to debug this far enough to see where things were breaking, but not so far as to find out what, in these settings, is causing things to not work correctly. This is with the 0.3 version of sbt-assembly, fwiw.

Just to make it really dead simple to replicate, I created a very simple, basic project to demonstrate the issue: https://github.com/tlockney/sbt-assembly-issue

packaging takes very long time

I am not sure if I am the only one who is facing this problem but
Packaging takes hell lot of time. May be I am missing some settings or something.
Last these three steps are the slowest, build stucks here for long long time.

[info] Merging 'org/apache/http/annotation/Immutable.class' with strategy 'deduplicate'
[info] SHA-1: WrappedArray(26, -125, -113, 64, 37, 94, -6, -79, -79, 41, 34, -92, 42, 7, -72, 31, -97, -57, 55, -2)
[info] Packaging ~/workspace/myproj/xyz/target/xyz-assembly-1.0.jar ..
[info] Done packaging.
[success] Total time: 724 s, completed 31 Jan, 2013 1:51:10 PM

Please let me know if anyone need more info ?

test task does not run tests for aggregated projects

It appears that including "assemblySettings" in a project's settings causes "test" to no longer tests the aggregated projects. There is a sbt groups thread about the issue here.

I've verified that running test:test does work, but it'd be nice if test behaved correctly.

sbt-assembly for 0.10.1

originally reported by @ijuma

Could you please deploy a version of sbt-assembly for sbt 0.10.1?

Add options for handling "provided" dependencies

According to http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope, which is VERY POORLY WRITTEN, "runtime" scoped dependencies are not required for compilation but are required for execution, so should be included in the assembly, but "provided" scope means that something outside your app will provide the jar. I can see multiple ways of handling provided:

• provided doesn't really mean anything when running in a fat jar, because you're supposed to have everything you need anyway, so include it if you can find it
• provided jars shouldn't go in the fat jar and it's up to the user to add them to the classpath at execution time

Both seem legitimate to me.

assembly-src task (a merged sources artifact)

@stephenh wrote:

I'm uploading an assembled artifact to our internal Maven repo, and would like to include a -sources.jar that was the merged sources of the main project + the dependencies that end up in the assembly jar.

assembly-src would be analogous to package-src as assembly is to package, except Scala library source should not be included by default. Likely it would expand AssemblyOption, which currently works like this:

    assemblyOption in assembly <<= (assembleArtifact in packageBin,
        assembleArtifact in packageScala, assembleArtifact in packageDependency, excludedFiles in assembly) {
      (includeBin, includeScala, includeDeps, exclude) =>   
      AssemblyOption(includeBin, includeScala, includeDeps, exclude) 
    },
    assemblyOption in packageDependency <<= (assemblyOption in assembly) { opt =>
      opt.copy(includeBin = false, includeScala = true, includeDependency = true)
    },
    assemblyOption in packageScala <<= (assemblyOption in assembly) { opt =>
      opt.copy(includeBin = false, includeScala = true, includeDependency = false)
    },

The build user should be able to control the default assembly behavior by setting:

assembleArtifact in packageSource := true

assembleArtifact in packageDependencySource := true

In other words, assemblySource would be a specialized version of assemblyOption:

    assemblyOption in assemblySource <<= (assemblyOption in assembly) { opt =>
      opt.copy(includeBin = false, includeScala = false, includeDependency = false,
        includeSource = true, includeDependencySource = true)
    },

subproject with no sources should not cause "No mapping" failure

I have a multi-module project with many subprojects. One of the subprojects has a src/test/java with some annotations and no src/main directory. Let's call it foo.

When I run assembly on a project (let's call it bar) that depends on foo, I got the following error:

[error] {file:-/home/ijuma/likecube/}bar/*:assembly: No mapping for /home/ijuma/likecube/foo/target/classes

Adding an empty Java file in src/main/java fixes the issue.

assembly doesn't pick up the jar file of project classes correctly

Here are the relevant parts build.sbt:

import AssemblyKeys._ //on the top

seq(assemblySettings: _*) //right at the end

Here is the project/plugins.sbt

 addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.7.1")

Invoking

> assembly-package-dependency

doesn't include the classes from the project and also doesn't include the scala classes. It seems like sbt 0.11 has changed the path where it compiles the jar file to.

Everything works with sbt 0.10 and version 0.6. If time permits i will try and create an example github project.

Documentation request - Deduplication

Hi, many of my projects have problems with clashes during de-duplication in class files. Could you add a section in the documentation on how to deal with duplicates that arise like this? For example:

[info] Merging 'javax/servlet/SingleThreadModel.class' with strategy 'deduplicate'
[trace] Stack trace suppressed: run 'last *:assembly' for the full output.
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /home/riri/.ivy2/cache/javax.servlet/servlet-api/jars/servlet-api-2.5.jar:javax/servlet/SingleThreadModel.class
[error] /home/riri/.ivy2/cache/org.mortbay.jetty/servlet-api-2.5/jars/servlet-api-2.5-6.1.14.jar:javax/servlet/SingleThreadModel.class
[error] /home/riri/.ivy2/cache/org.mortbay.jetty/servlet-api/jars/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class

The current documentation would suggest using the "concat" strategy on "SingleThreadModel.class", but in this case there are many duplicate files, which are only exposed one by one as more exceptions are added to the assembly settings so the configuration becomes quite long.

Windows: MANIFEST.MF doesn't seem to be ignored like it should

(using version 0.8.3)

I have to add this to make assembly ignore manifest.mf files:

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case n if n.toLowerCase.contains("manifest.mf") => MergeStrategy.discard
    case x => old(x)
  }
}

Output multiple jars?

How would I go about configuring sbt-assembly to produce multiple jars? I have a codebase, to which I want to adds multiple tools - actually a server process and now would like to have some CLI tools for it. Is it possible with sbt-assembly or should I look elsewhere?

Can you post an example of using AssemblyOption?

I'd like to exclude a particular dependency from the fat jar (bouncycastle) because it has jar signatures that don't work as part of a fat jar.

Provide a setting to exclude Scala library

If one plans to execute the jar with the scala command (instead of Java), then it's not necessary to include the Scala library.

Not an Issue but a request for new version

Hey Eugene, it has been a while since you pushed a new version, and I noticed you have a new "excludedJars" key/setting. I was wondering if you'd mind publishing a 0.7 build. I prefer static build to putting a dependency on your git repository.

Thanks,
D

multiple jars

Is it possible to configure sbt-assembly to create a separate jar file for each of a set of main classes?

Do not remove all LICESNSEs by default

Asking for lawsuits. We should instead aggregate licenses into a "dependent-LICENSES" file by default.

Repackaging a signed jar file

I get this:

Exception in thread "main" java.lang.SecurityException: no manifiest section for signature file entry org/bouncycastle/cms/CMSSign
edDataStreamGenerator$TeeOutputStream.class
        at sun.security.util.SignatureFileVerifier.verifySection(Unknown Source)
        at sun.security.util.SignatureFileVerifier.processImpl(Unknown Source)
        at sun.security.util.SignatureFileVerifier.process(Unknown Source)
        at java.util.jar.JarVerifier.processEntry(Unknown Source)
        at java.util.jar.JarVerifier.update(Unknown Source)

How do you deal with this issue?

missing META-INF artifacts

I've got a project that relies on some Spring jars, turns out that when I started using the assembly plugin (0.4) that the META-INF files required by Spring (*.handlers) were not being included in the assembled package. I followed the instructions posted here and haven't set any additional configuration options (though it would be nice to have a task that skipped tests as a short-cut)

feature request: add resource merging capability

Our shiny new config library relies upon reading all resources named reference.conf from the class path in order to obtain and merge the default values for all libraries using it (e.g. Akka, Play framework). sbt-assembly will include only the last such fragment from the class path into the fat jar, thereby losing all other defaults and making e.g. Akka crash upon initialization. The solution is to just concatenate those files (which can be done manually for example in a simple REPL session) and add that to the JAR, but sbt-assembly will not allow that because of a duplicate entry. Ignoring that resource will leave it out completely, so that it needs to be added manually afterwards (e.g. with zip in the shell).

A possible solution, which might also benefit other purposes, is the addition of merge capabilities to sbt-assembly. The build settings would include a transformation function Seq[File] -> Array[Byte] attached to a resource name, and then all resources with that name would be collected, transformed and the result added to the JAR.

Readme is misleading

I'm using sbt for a pure java project. Sources are in src/main/java and they get properly compiled and packaged by sbt's compile and package tasks. However, when I run assembly, only my dependencies (libraries) and up in the assembly jar, while the main project's classes are missing. Is this a bug? Can I work around this?

Caching of dependent project's libs avoidable?

If project A depends on project B, and project B uses a jar X, then the cached folders of both projects contains the expanded jar X.

For a large project, with tens of sub- projects and hundreds of jars, it creates a huge overhead, slowing down builds because of all the disk activity.

Is there a way to reuse the cached folders from B in A?

Thanks!

Configuration aware sbt-assembly

Hi,

Would it be possible to make the assembly commands aware of the configuration they are run in? For example I would like to package all my tests to jar (that contains also the test framework etc.), to make running the test set as easy as possible in any machine. This is what I do now:

mainClass in assembly <<= mainClass in Test

fullClasspath in assembly <<= fullClasspath in Test

dependencyClasspath in assembly <<= dependencyClasspath in Test

Well it works, but now it only builds me the package including tests. What I would like to do is to define something like

(mainClass in assembly) in Test <<= mainClass in Test

(fullClasspath in assembly) in Test <<= fullClasspath in Test

(dependencyClasspath in assembly) in Test <<= dependencyClasspath in Test

and run "test:assembly" to get my package with tests or run "assembly" to get my normal package. I use test just as an example, but making the assembly fully aware of the Configuration scope, would allow easy way to make different kind of far Jars from the same code base.

"Duplicated entry" error when I run assembly twice

I recently set up Spark (http://github.com/mesos/spark) to use the sbt-assembly plugin, and it works great the first time I run sbt assembly, but if I run this again afterwards I get errors like this:

[error] {file:/Users/matei/workspace/spark/}core/*:assembly: java.util.zip.ZipException: duplicate entry: spark/Partitioner.class

If I do sbt clean followed by assembly again though, it works fine. Any idea why this is happening? Am I doing something wrong in my project? Here is my SBT script: https://github.com/mesos/spark/blob/master/project/SparkBuild.scala.

tag for 0.8.4 and 0.8.5 missing

How to add Class-Path to manifest in resulting assembly?

So I could get:

Manifest-Version: 1.0
Main-Class: Hi
Class-Path: lib\scala-library.jar

in resulting hi.jar

Simplify merging of typesafe config 'reference.conf' files

This issue builds on issue #32 that you very promptly got resolved, thanks for that!
Your fix plus the code that Roland from the Akka team published in this blog post solve the issue of merging typesafe config 'reference.conf's.

The only remaining question is: How do we best package the little bit of code shown in the blog post, so we make it as easy and DRY for users to apply to their SBT builds?

For copypasting it's a bit too much code and we'd loose control over it, making potential updates and fixes hard. So we thought about producing a wrapper-plugin around sbt-assembly that "decorates" it with the missing bits. That would work, but it appears like a lot of overhead for such little logic and it would probably also add initial confusion as to what exactly the difference would be between sbt-assembly and "sbt-assembly-plus".

So, after all, it seems the best solution would be if the logic for merging the "reference.conf"s would live inside of sbt-assembly. Maybe it shouldn't be "chained in" by default, but we can't really see a better place for it than inside of your plugin.
How about the following cascade of interfaces/hooks to incrementally add more specific "merging logic"?

1. assembledMappings (which you already put in) allows me to add/remove/patch the list of files that go into the fat jar. That's the lowest level "interface" for messing with far jar content. By default it would do whatever it does now plus the support of collisionResolution.
2. collisionResolution (a new TaskKey[Seq[File] => Either[String, File]]) could allow me to hook in custom logic for resolving conflicts, where several files from the merged JARs would be mapped to the same file in the fat jar. The default implementation could be mergeReferenceConfs or "log a warning and return Right(seq.head)" or Left(<collision warning>), which would abort the assembly with an error message.
3. mergeReferenceConfs (a new TaskKey[Seq[File] => Either[String, File]]) would return Right(<concatenated contents>) if the name is "reference.conf". It could be the default implementation of collisionResolution or (if you prefer) simply be available for use by anyone wanting it to apply to collisionResolution without actually being chained in by default.

I personally would prefer everything to be set up in a way that doesn't require any additional configuration from a user in order to have libs using the typesafe config be properly merged, but I could understand, if you wanted to keep sbt-assembly as "clean" as possible by default.

What do you think?

Where does the posterousNotesVersion setting key come from?

I'm trying to use the version directly from github at the moment, and SBT is complaining about the posterousNotesVersion setting key:

[knuttycombe@floorshow services (master)]$ sbt
[info] Compiling 1 Scala source to /Users/knuttycombe/reportgrid/services/project/plugins/project/target/scala_2.8.1/classes...
/Users/knuttycombe/.sbt/staging/23219d1aa6e43f4668ba/build.sbt:7: error: not found: value posterousNotesVersion
posterousNotesVersion := "0.3"
^
/Users/knuttycombe/.sbt/staging/23219d1aa6e43f4668ba/build.sbt:7: error: reassignment to val
posterousNotesVersion := "0.3"
                      ^

Current project's LICENSE file is removed from assembled jar

Very suspicious to find your own LICENSE file gone. I think the default should allow your license file, if no others.

don't override the settings

steps

1. put sbt-assembly into deps.

problem

it automatically introduces itself into k-v.

expectation

let the build user choose.

Fix jar naming for cross-building

Cross-builds seem to work but overwrite each previous language version built:

> +assembly
Setting version to 2.8.1
[info] Set current project to util (in build file:/Users/peter/Projects/Miogiro/Code/util/)
...
[info] Packaging /Users/peter/Projects/Miogiro/Code/util/target/util-assembly-0.1.jar ...
[info] Done packaging.
[success] Total time: 3 s, completed Nov 9, 2011 4:14:44 PM
Setting version to 2.9.1
[info] Set current project to util (in build file:/Users/peter/Projects/Miogiro/Code/util/)
... 
[info] Packaging /Users/peter/Projects/Miogiro/Code/util/target/util-assembly-0.1.jar ...
[info] Done packaging.
[success] Total time: 3 s, completed Nov 9, 2011 4:14:48 PM
Setting version to 2.9.1
[info] Set current project to util (in build file:/Users/peter/Projects/Miogiro/Code/util/)

The solution is to include the Scala version in the jar name. Basically, I would follow the sbt naming convention.

Publish Commands

It'd be great to have versions of the publish commands for the assembled jar.

Possible to run multiple packaging tasks?

Is it possible in a Build.scala to somehow invoke assembly multiple times?

I wish to generate platform specific jars, e.g. target/MyProject-Mac.jar, target/MyProject-Linux.jar, and target/MyProject-Windows.jar, each of which would require separate excludeFiles / excludeJars settings.

How would I do this?

Publish release for SBT 0.11.3

The subject says it all. Thanks!

Type mismatch in full configuration

Hi Eugene!

When I try to use sbt-assembly in a full configuration, I get this:

[error] /Users/psnively/vmware/git/vcib/vcaf-db/project/Build.scala:71: type mismatch;
[error] found : scala.Seq[sbt.Project.Setting[]]
[error] required: Seq[sbt.Project.Setting[$1(in lazy value creation)]] where type _$1(in lazy value creation)
[error] Error occurred in an application involving default arguments.
[error] settings = buildSettings ++ Seq (libraryDependencies := commonDeps, resolvers := cseResolvers) ++ Seq (sbtassembly.Plugin.assemblySettings: _*)

I'm sure I'm being dense, but I don't see how those types don't match. :-)

Any thoughts?

Thanks!
Paul

sbt-assembly doesn't seem to respect sbt's `exclude` and `excludeAll` settings

For example, with the following dependencies, assembly will include those jars specified by 'excludeAll' clause.

val appDependencies = Seq(
"com.twitter" % "finagle-core_2.9.1" % "3.0.0",
"com.twitter" % "finagle-ostrich4_2.9.1" % "3.0.0",
"com.twitter" % "finagle-thrift_2.9.1" % "3.0.0",
"com.twitter" % "parrot" % "0.4.5" intransitive, // pulls in non-2.9.1 finagle
"com.twitter.common.zookeeper" % "server-set" % "0.0.8" excludeAll(
ExclusionRule(organization="com.twitter", name="finagle-core"),
ExclusionRule(organization="com.twitter", name="finagle-thrift"),
ExclusionRule(organization="com.twitter", name="util-core"),
ExclusionRule(organization="com.twitter", name="util-collection"),
ExclusionRule(organization="com.twitter", name="util-hashing")
),
"mysql" % "mysql-connector-java" % "5.1.18"
)

Provide a task to generate a jar for all the dependencies

In other words, the source code for the current project would not be included in the jar.

Spark (https://github.com/mesos/spark) overrides a bunch of methods in sbt-assembly for sbt 0.7.x in order to achieve this, but it's not as easy to do the same in sbt-assembly for sbt 0.10.x. In any case, it's useful because the dependencies don't change as often as the code for the project and time is saved building the deployment jars.

package-dependency still packages project classes.

Project classes are still being packaged into assemblied jar.

Ability to prevent Caching?

Is there a way to prevent caching in the assembly task?

I am maintaining a big and complex project with many dependencies. The generated cache grows to a total of 2GB!

Worse, I don't really need the cache because when we do assemble the jars, it is for deployment, and this always happens from a clean checkout.

Is there a way to disable the cache completely?

Thanks!

Can't figure out how to specify a main class.

I'm sure I must be doing something wrong, but I can't for the life of me figure out what. I've stripped it down to as simple as possible, but it still doesn't work.

I keep getting this error when I try to run the jar files:

Error: Could not find or load main class runner.runner

The build.sbt and plugins.sbt files were exactly as in your example at:
https://github.com/sbt/sbt-assembly
except build.sbt is:

import AssemblyKeys._
assemblySettings
jarName in assembly := "MyScalaUtils.jar"
mainClass in assembly := Some("runner.runner")

Here's my src directory layout:

$ ls -R src
src:
helloWorldTest  runner

src/helloWorldTest:
helloWorld.scala

src/runner:
runner.scala

runner.scala is just:

package runner

class runner {
  def main(args: Array[String]){
    println("Starting run.");
    helloWorldTest.helloWorld.run(args);
    println("Done.");
  }
}

What the heck is going wrong here?

don't unzip the jars (maven-assembly-plugin mode)

originally reported by @rollinsruss

steps

1. add deps to "org.jboss.netty" % "netty" % "3.2.3.Final, spring etc.
2. assembly

problem

the license folder, *.handlers, etc gets wiped out under META-INF.

expectation

That being said, I keep coming back to thinking about the Maven assembly plugin that applies an alternative approach to unzipping everything and simply shoves the jars into an embedded lib directory and updates the manifest so the classpath is complete--which avoids this problem altogether and maintains pristine isolation. Perhaps that approach could be a future alternative task on this plugin (wish I had time to help, as it has me quite intrigued).

0.8.3 crashes under windows

I was using sbt-assembly as a git dependency, until the scripted-plugin stopped being available for sbt 0.11.3, so I switched to using 0.8.3 as a jar. But now assembly crashes under windows, and I can't figure out the cause:

[info] Including geotrellis_2.9.2-0.7.0-RC1.jar
[error] {file:/C:/Users/Administrator/projects/gt-hunchlab/}root/*:assembly: Err
or extracting zip entry 'META-INF/LICENSE' to 'C:\Users\ADMINI1\AppData\Local\T
emp\sbt_26b9328f\638941e23157ea788a0a1dea22ac3affe26e10b4\META-INF\LICENSE': jav
a.io.FileNotFoundException: C:\Users\ADMINI1\AppData\Local\Temp\sbt_26b9328f\63
8941e23157ea788a0a1dea22ac3affe26e10b4\META-INF\LICENSE (Access is denied)
[error] Total time: 112 s, completed Jul 26, 2012 1:17:06 PM

Need Some() to define mainClass

To define the main class for my assembly I need to put

mainClass in assembly := Some("main.ClassName")

in my build.sbt file.

If I don't use Some() then I get the following error when starting sbt in the project directory

$ sbt
[info] Loading global plugins from /Volumes/Users/Home/.sbt/plugins
[info] Loading project definition from /Volumes/Users/Home/Workspace/Proj/project
/Volumes/Users/Home/Workspace/Proj/build.sbt:11: error: type mismatch;
 found   : java.lang.String("main.ClassName")
 required: Option[String]
mainClass in assembly := "main.ClassName"
                         ^
[error] Type error in expression
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? 

I'm using sbt-assembly 0.7.2 with scala 2.9.1

jarname in assembly should be jarname in Assembly?

I'm using sbt-assembly 0.6 with sbt 0.11.0 and to my surprise putting

jarName in assembly := "foo.jar"

in build.sbt did not work: the jar produced was called foo-assembly-1.0.0.jar.

Now I'm still in the cargo-cult stage of sbt programming, but with trial and error I found that this works:

jarName in Assembly := "foo.jar"

Scala star pattern varargs matching error

I am not sure if this is a scala error or the PathList class needs changes, But i get this for matching PathList with a start pattern like in the README example

[error] ...../project/Build.scala:17: star patterns must correspond with varargs parameters
[error] Error occurred in an application involving default arguments.
[error]               case PathList("javax","servlet",xs @ _*) => MergeStrategy.first
[error]                                                     ^
[error] one error found

sbt 0.11.3

scala 2.9.2