sbt / sbt-assembly Goto Github PK

It'd be great to have versions of the publish commands for the assembled jar.

I have a multi-module project with many subprojects. One of the subprojects has a src/test/java with some annotations and no src/main directory. Let's call it foo.

When I run assembly on a project (let's call it bar) that depends on foo, I got the following error:

[error] {file:-/home/ijuma/likecube/}bar/*:assembly: No mapping for /home/ijuma/likecube/foo/target/classes

Adding an empty Java file in src/main/java fixes the issue.

It would be cool if there would be a merge strategy which is similar to concat but which writes the name of the jar file where ithe file comes from above the actual content. Like this:

netty-x.x.x.jar

[license text]

httpclient.jar

[license text]

I'm having trouble with getting my settings used. For example, my mainClass setting appears to not be found.

Here's the configuration I'm using:

45     val yggdrasilSettings = serviceSettings ++ Seq(
46       resolvers ++= Seq(
47         "riptano" at "http://mvn.riptano.com/content/repositories/public",
48         "Scale7 Maven Repo" at "https://github.com/s7/mvnrepo/raw/master"
49       ),
50       libraryDependencies ++= Seq(
51         "org.scale7" % "scale7-pelops" % "1.2-0.8.x-SNAPSHOT",
52         "joda-time"               % "joda-time"    % "1.6.2",
53         "org.scalaz"              %% "scalaz-core" % "6.0.1"
54       ),
55       mainClass := Some("com.reportgrid.yggdrasil.Yggdrasil"),
56       jarName in Assembly <<= version apply { v => "yggdrasil-v" + v.substring(0, v.indexOf(".")) }
57     )
58 
59     val yggdrasil = Project("yggdrasil", file("yggdrasil"), settings = yggdrasilSettings) dependsOn(common) dependsOnAlt (blueeyes(base))

I've tried both mainClass and mainClass in Assembly, and neither seems to work:

[knuttycombe@floorshow yggdrasil (master)]$ java -jar target/yggdrasil-assembly-0.3.jar --configFile yggdrasil.conf 
Failed to load Main-Class manifest attribute from
target/yggdrasil-assembly-0.3.jar

As you can see, my jarName is also not being respected. Any ideas?

We are trying to create Dropwizard-based (https://github.com/codahale/dropwizard) services, which uses Jersey and depends on providers defined under META-INF/services. It looks like these should be getting included based on the PathFinder in assemblyExcludedFiles, but it's not appearing to work for us. I have only been able to debug this far enough to see where things were breaking, but not so far as to find out what, in these settings, is causing things to not work correctly. This is with the 0.3 version of sbt-assembly, fwiw.

Just to make it really dead simple to replicate, I created a very simple, basic project to demonstrate the issue: https://github.com/tlockney/sbt-assembly-issue

I recently set up Spark (http://github.com/mesos/spark) to use the sbt-assembly plugin, and it works great the first time I run sbt assembly, but if I run this again afterwards I get errors like this:

[error] {file:/Users/matei/workspace/spark/}core/*:assembly: java.util.zip.ZipException: duplicate entry: spark/Partitioner.class

If I do sbt clean followed by assembly again though, it works fine. Any idea why this is happening? Am I doing something wrong in my project? Here is my SBT script: https://github.com/mesos/spark/blob/master/project/SparkBuild.scala.

Is it possible in a Build.scala to somehow invoke assembly multiple times?

I wish to generate platform specific jars, e.g. target/MyProject-Mac.jar, target/MyProject-Linux.jar, and target/MyProject-Windows.jar, each of which would require separate excludeFiles / excludeJars settings.

How would I do this?

Here are the relevant parts build.sbt:

import AssemblyKeys._ //on the top

seq(assemblySettings: _*) //right at the end

Here is the project/plugins.sbt

 addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.7.1")

Invoking

> assembly-package-dependency

doesn't include the classes from the project and also doesn't include the scala classes. It seems like sbt 0.11 has changed the path where it compiles the jar file to.

Everything works with sbt 0.10 and version 0.6. If time permits i will try and create an example github project.

So I could get:

Manifest-Version: 1.0
Main-Class: Hi
Class-Path: lib\scala-library.jar

in resulting hi.jar

I am not sure if I am the only one who is facing this problem but
Packaging takes hell lot of time. May be I am missing some settings or something.
Last these three steps are the slowest, build stucks here for long long time.

[info] Merging 'org/apache/http/annotation/Immutable.class' with strategy 'deduplicate'
[info] SHA-1: WrappedArray(26, -125, -113, 64, 37, 94, -6, -79, -79, 41, 34, -92, 42, 7, -72, 31, -97, -57, 55, -2)
[info] Packaging ~/workspace/myproj/xyz/target/xyz-assembly-1.0.jar ..
[info] Done packaging.
[success] Total time: 724 s, completed 31 Jan, 2013 1:51:10 PM

Please let me know if anyone need more info ?

I've got a project that relies on some Spring jars, turns out that when I started using the assembly plugin (0.4) that the META-INF files required by Spring (*.handlers) were not being included in the assembled package. I followed the instructions posted here and haven't set any additional configuration options (though it would be nice to have a task that skipped tests as a short-cut)

According to http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope, which is VERY POORLY WRITTEN, "runtime" scoped dependencies are not required for compilation but are required for execution, so should be included in the assembly, but "provided" scope means that something outside your app will provide the jar. I can see multiple ways of handling provided:

• provided doesn't really mean anything when running in a fat jar, because you're supposed to have everything you need anyway, so include it if you can find it
• provided jars shouldn't go in the fat jar and it's up to the user to add them to the classpath at execution time

Both seem legitimate to me.

To define the main class for my assembly I need to put

mainClass in assembly := Some("main.ClassName")

in my build.sbt file.

If I don't use Some() then I get the following error when starting sbt in the project directory

$ sbt
[info] Loading global plugins from /Volumes/Users/Home/.sbt/plugins
[info] Loading project definition from /Volumes/Users/Home/Workspace/Proj/project
/Volumes/Users/Home/Workspace/Proj/build.sbt:11: error: type mismatch;
 found   : java.lang.String("main.ClassName")
 required: Option[String]
mainClass in assembly := "main.ClassName"
                         ^
[error] Type error in expression
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? 

I'm using sbt-assembly 0.7.2 with scala 2.9.1

I'd like to exclude a particular dependency from the fat jar (bouncycastle) because it has jar signatures that don't work as part of a fat jar.

Our shiny new config library relies upon reading all resources named reference.conf from the class path in order to obtain and merge the default values for all libraries using it (e.g. Akka, Play framework). sbt-assembly will include only the last such fragment from the class path into the fat jar, thereby losing all other defaults and making e.g. Akka crash upon initialization. The solution is to just concatenate those files (which can be done manually for example in a simple REPL session) and add that to the JAR, but sbt-assembly will not allow that because of a duplicate entry. Ignoring that resource will leave it out completely, so that it needs to be added manually afterwards (e.g. with zip in the shell).

A possible solution, which might also benefit other purposes, is the addition of merge capabilities to sbt-assembly. The build settings would include a transformation function Seq[File] -> Array[Byte] attached to a resource name, and then all resources with that name would be collected, transformed and the result added to the JAR.

(using version 0.8.3)

I have to add this to make assembly ignore manifest.mf files:

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case n if n.toLowerCase.contains("manifest.mf") => MergeStrategy.discard
    case x => old(x)
  }
}

@stephenh wrote:

I'm uploading an assembled artifact to our internal Maven repo, and would like to include a -sources.jar that was the merged sources of the main project + the dependencies that end up in the assembly jar.

assembly-src would be analogous to package-src as assembly is to package, except Scala library source should not be included by default. Likely it would expand AssemblyOption, which currently works like this:

    assemblyOption in assembly <<= (assembleArtifact in packageBin,
        assembleArtifact in packageScala, assembleArtifact in packageDependency, excludedFiles in assembly) {
      (includeBin, includeScala, includeDeps, exclude) =>   
      AssemblyOption(includeBin, includeScala, includeDeps, exclude) 
    },
    assemblyOption in packageDependency <<= (assemblyOption in assembly) { opt =>
      opt.copy(includeBin = false, includeScala = true, includeDependency = true)
    },
    assemblyOption in packageScala <<= (assemblyOption in assembly) { opt =>
      opt.copy(includeBin = false, includeScala = true, includeDependency = false)
    },

The build user should be able to control the default assembly behavior by setting:

assembleArtifact in packageSource := true

assembleArtifact in packageDependencySource := true

In other words, assemblySource would be a specialized version of assemblyOption:

    assemblyOption in assemblySource <<= (assemblyOption in assembly) { opt =>
      opt.copy(includeBin = false, includeScala = false, includeDependency = false,
        includeSource = true, includeDependencySource = true)
    },

Hi,

my generated assembly gives:

Exception in thread "main" java.lang.NoClassDefFoundError: scala/App
...
Caused by: java.lang.ClassNotFoundException: scala.App

What should I do to include Scala dependencies?

Thanks,
Etam.

Hi, many of my projects have problems with clashes during de-duplication in class files. Could you add a section in the documentation on how to deal with duplicates that arise like this? For example:

[info] Merging 'javax/servlet/SingleThreadModel.class' with strategy 'deduplicate'
[trace] Stack trace suppressed: run 'last *:assembly' for the full output.
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /home/riri/.ivy2/cache/javax.servlet/servlet-api/jars/servlet-api-2.5.jar:javax/servlet/SingleThreadModel.class
[error] /home/riri/.ivy2/cache/org.mortbay.jetty/servlet-api-2.5/jars/servlet-api-2.5-6.1.14.jar:javax/servlet/SingleThreadModel.class
[error] /home/riri/.ivy2/cache/org.mortbay.jetty/servlet-api/jars/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class

The current documentation would suggest using the "concat" strategy on "SingleThreadModel.class", but in this case there are many duplicate files, which are only exposed one by one as more exceptions are added to the assembly settings so the configuration becomes quite long.

I'm trying to use the version directly from github at the moment, and SBT is complaining about the posterousNotesVersion setting key:

[knuttycombe@floorshow services (master)]$ sbt
[info] Compiling 1 Scala source to /Users/knuttycombe/reportgrid/services/project/plugins/project/target/scala_2.8.1/classes...
/Users/knuttycombe/.sbt/staging/23219d1aa6e43f4668ba/build.sbt:7: error: not found: value posterousNotesVersion
posterousNotesVersion := "0.3"
^
/Users/knuttycombe/.sbt/staging/23219d1aa6e43f4668ba/build.sbt:7: error: reassignment to val
posterousNotesVersion := "0.3"
                      ^

I'm sure I must be doing something wrong, but I can't for the life of me figure out what. I've stripped it down to as simple as possible, but it still doesn't work.

I keep getting this error when I try to run the jar files:

Error: Could not find or load main class runner.runner

The build.sbt and plugins.sbt files were exactly as in your example at:
https://github.com/sbt/sbt-assembly
except build.sbt is:

import AssemblyKeys._
assemblySettings
jarName in assembly := "MyScalaUtils.jar"
mainClass in assembly := Some("runner.runner")

Here's my src directory layout:

$ ls -R src
src:
helloWorldTest  runner

src/helloWorldTest:
helloWorld.scala

src/runner:
runner.scala

runner.scala is just:

package runner

class runner {
  def main(args: Array[String]){
    println("Starting run.");
    helloWorldTest.helloWorld.run(args);
    println("Done.");
  }
}

What the heck is going wrong here?

Project classes are still being packaged into assemblied jar.

I was using sbt-assembly as a git dependency, until the scripted-plugin stopped being available for sbt 0.11.3, so I switched to using 0.8.3 as a jar. But now assembly crashes under windows, and I can't figure out the cause:

[info] Including geotrellis_2.9.2-0.7.0-RC1.jar
[error] {file:/C:/Users/Administrator/projects/gt-hunchlab/}root/*:assembly: Err
or extracting zip entry 'META-INF/LICENSE' to 'C:\Users\ADMINI1\AppData\Local\T
emp\sbt_26b9328f\638941e23157ea788a0a1dea22ac3affe26e10b4\META-INF\LICENSE': jav
a.io.FileNotFoundException: C:\Users\ADMINI1\AppData\Local\Temp\sbt_26b9328f\63
8941e23157ea788a0a1dea22ac3affe26e10b4\META-INF\LICENSE (Access is denied)
[error] Total time: 112 s, completed Jul 26, 2012 1:17:06 PM

I am switching from mvn assembly to sbt assembly and I get this error:

{file:/Users/tamir/work/jnomicss/jnomicss-cli/}default-5dc2c4/*:assembly: Could not create directory /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f/license: file exists and is not a directory.

Do you have a quick workaround? I don't need any licences now (its completely internal).

excluding them in build.sbt like this does not help:

excludedFiles in assembly := { (bases: Seq[File]) =>
  bases flatMap { base =>
    (base / "META-INF" * "*").get collect {
      case f if f.getName == "something" => f
      case f if f.getName.toLowerCase == "license" => f
      case f if f.getName.toLowerCase == "manifest.mf" => f
    }
  }}

the temporary folders actually don't extis:

127 [email protected] ~> ls /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f/license
ls: /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f/license: No such file or directory
1 [email protected] ~> ls /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f
ls: /var/folders/2h/2hhAWmGeGYufjuse2p1fE+++krQ/-Tmp-/sbt_1cdeab2f: No such file or directory

Hi Eugene!

When I try to use sbt-assembly in a full configuration, I get this:

[error] /Users/psnively/vmware/git/vcib/vcaf-db/project/Build.scala:71: type mismatch;
[error] found : scala.Seq[sbt.Project.Setting[]]
[error] required: Seq[sbt.Project.Setting[$1(in lazy value creation)]] where type _$1(in lazy value creation)
[error] Error occurred in an application involving default arguments.
[error] settings = buildSettings ++ Seq (libraryDependencies := commonDeps, resolvers := cseResolvers) ++ Seq (sbtassembly.Plugin.assemblySettings: _*)

I'm sure I'm being dense, but I don't see how those types don't match. :-)

Any thoughts?

Thanks!
Paul

How would I go about configuring sbt-assembly to produce multiple jars? I have a codebase, to which I want to adds multiple tools - actually a server process and now would like to have some CLI tools for it. Is it possible with sbt-assembly or should I look elsewhere?

Is it possible to configure sbt-assembly to create a separate jar file for each of a set of main classes?

I want to use sbt assembly in conjunction with a bigger build process that is managed by make so it would nice if the timestamp on the jar that sbt-assembly produces would only change when necessary.

If one plans to execute the jar with the scala command (instead of Java), then it's not necessary to include the Scala library.

originally reported by @rollinsruss

steps

1. add deps to "org.jboss.netty" % "netty" % "3.2.3.Final, spring etc.
2. assembly

problem

the license folder, *.handlers, etc gets wiped out under META-INF.

expectation

That being said, I keep coming back to thinking about the Maven assembly plugin that applies an alternative approach to unzipping everything and simply shoves the jars into an embedded lib directory and updates the manifest so the classpath is complete--which avoids this problem altogether and maintains pristine isolation. Perhaps that approach could be a future alternative task on this plugin (wish I had time to help, as it has me quite intrigued).

I'm using sbt-assembly 0.6 with sbt 0.11.0 and to my surprise putting

jarName in assembly := "foo.jar"

in build.sbt did not work: the jar produced was called foo-assembly-1.0.0.jar.

Now I'm still in the cargo-cult stage of sbt programming, but with trial and error I found that this works:

jarName in Assembly := "foo.jar"

It appears that including "assemblySettings" in a project's settings causes "test" to no longer tests the aggregated projects. There is a sbt groups thread about the issue here.

I've verified that running test:test does work, but it'd be nice if test behaved correctly.

steps

1. put sbt-assembly into deps.

problem

it automatically introduces itself into k-v.

expectation

let the build user choose.

I'm using sbt for a pure java project. Sources are in src/main/java and they get properly compiled and packaged by sbt's compile and package tasks. However, when I run assembly, only my dependencies (libraries) and up in the assembly jar, while the main project's classes are missing. Is this a bug? Can I work around this?

For example, with the following dependencies, assembly will include those jars specified by 'excludeAll' clause.

val appDependencies = Seq(
"com.twitter" % "finagle-core_2.9.1" % "3.0.0",
"com.twitter" % "finagle-ostrich4_2.9.1" % "3.0.0",
"com.twitter" % "finagle-thrift_2.9.1" % "3.0.0",
"com.twitter" % "parrot" % "0.4.5" intransitive, // pulls in non-2.9.1 finagle
"com.twitter.common.zookeeper" % "server-set" % "0.0.8" excludeAll(
ExclusionRule(organization="com.twitter", name="finagle-core"),
ExclusionRule(organization="com.twitter", name="finagle-thrift"),
ExclusionRule(organization="com.twitter", name="util-core"),
ExclusionRule(organization="com.twitter", name="util-collection"),
ExclusionRule(organization="com.twitter", name="util-hashing")
),
"mysql" % "mysql-connector-java" % "5.1.18"
)

The subject says it all. Thanks!

Hi,

Would it be possible to make the assembly commands aware of the configuration they are run in? For example I would like to package all my tests to jar (that contains also the test framework etc.), to make running the test set as easy as possible in any machine. This is what I do now:

mainClass in assembly <<= mainClass in Test

fullClasspath in assembly <<= fullClasspath in Test

dependencyClasspath in assembly <<= dependencyClasspath in Test

Well it works, but now it only builds me the package including tests. What I would like to do is to define something like

(mainClass in assembly) in Test <<= mainClass in Test

(fullClasspath in assembly) in Test <<= fullClasspath in Test

(dependencyClasspath in assembly) in Test <<= dependencyClasspath in Test

and run "test:assembly" to get my package with tests or run "assembly" to get my normal package. I use test just as an example, but making the assembly fully aware of the Configuration scope, would allow easy way to make different kind of far Jars from the same code base.

I'm using SBT 0.12.0-RC4, and in project/plugins.sbt it's enough to add this instruction:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.8.3")

Hence, the README should be updated. The instruction specified in the README for the -Beta2, instead, causes an AbstractMethodError - apparently the API changed.

Cross-builds seem to work but overwrite each previous language version built:

> +assembly
Setting version to 2.8.1
[info] Set current project to util (in build file:/Users/peter/Projects/Miogiro/Code/util/)
...
[info] Packaging /Users/peter/Projects/Miogiro/Code/util/target/util-assembly-0.1.jar ...
[info] Done packaging.
[success] Total time: 3 s, completed Nov 9, 2011 4:14:44 PM
Setting version to 2.9.1
[info] Set current project to util (in build file:/Users/peter/Projects/Miogiro/Code/util/)
... 
[info] Packaging /Users/peter/Projects/Miogiro/Code/util/target/util-assembly-0.1.jar ...
[info] Done packaging.
[success] Total time: 3 s, completed Nov 9, 2011 4:14:48 PM
Setting version to 2.9.1
[info] Set current project to util (in build file:/Users/peter/Projects/Miogiro/Code/util/)

The solution is to include the Scala version in the jar name. Basically, I would follow the sbt naming convention.

In other words, the source code for the current project would not be included in the jar.

Spark (https://github.com/mesos/spark) overrides a bunch of methods in sbt-assembly for sbt 0.7.x in order to achieve this, but it's not as easy to do the same in sbt-assembly for sbt 0.10.x. In any case, it's useful because the dependencies don't change as often as the code for the project and time is saved building the deployment jars.

Very suspicious to find your own LICENSE file gone. I think the default should allow your license file, if no others.

originally reported by @ijuma

Could you please deploy a version of sbt-assembly for sbt 0.10.1?

Hey Eugene, it has been a while since you pushed a new version, and I noticed you have a new "excludedJars" key/setting. I was wondering if you'd mind publishing a 0.7 build. I prefer static build to putting a dependency on your git repository.

Thanks,
D

Is there a way to prevent caching in the assembly task?

I am maintaining a big and complex project with many dependencies. The generated cache grows to a total of 2GB!

Worse, I don't really need the cache because when we do assemble the jars, it is for deployment, and this always happens from a clean checkout.

Is there a way to disable the cache completely?

Thanks!

I am not sure if this is a scala error or the PathList class needs changes, But i get this for matching PathList with a start pattern like in the README example

[error] ...../project/Build.scala:17: star patterns must correspond with varargs parameters
[error] Error occurred in an application involving default arguments.
[error]               case PathList("javax","servlet",xs @ _*) => MergeStrategy.first
[error]                                                     ^
[error] one error found

sbt 0.11.3

scala 2.9.2

If project A depends on project B, and project B uses a jar X, then the cached folders of both projects contains the expanded jar X.

For a large project, with tens of sub- projects and hundreds of jars, it creates a huge overhead, slowing down builds because of all the disk activity.

Is there a way to reuse the cached folders from B in A?

Thanks!

Asking for lawsuits. We should instead aggregate licenses into a "dependent-LICENSES" file by default.

This issue builds on issue #32 that you very promptly got resolved, thanks for that!
Your fix plus the code that Roland from the Akka team published in this blog post solve the issue of merging typesafe config 'reference.conf's.

The only remaining question is: How do we best package the little bit of code shown in the blog post, so we make it as easy and DRY for users to apply to their SBT builds?

For copypasting it's a bit too much code and we'd loose control over it, making potential updates and fixes hard. So we thought about producing a wrapper-plugin around sbt-assembly that "decorates" it with the missing bits. That would work, but it appears like a lot of overhead for such little logic and it would probably also add initial confusion as to what exactly the difference would be between sbt-assembly and "sbt-assembly-plus".

So, after all, it seems the best solution would be if the logic for merging the "reference.conf"s would live inside of sbt-assembly. Maybe it shouldn't be "chained in" by default, but we can't really see a better place for it than inside of your plugin.
How about the following cascade of interfaces/hooks to incrementally add more specific "merging logic"?

1. assembledMappings (which you already put in) allows me to add/remove/patch the list of files that go into the fat jar. That's the lowest level "interface" for messing with far jar content. By default it would do whatever it does now plus the support of collisionResolution.
2. collisionResolution (a new TaskKey[Seq[File] => Either[String, File]]) could allow me to hook in custom logic for resolving conflicts, where several files from the merged JARs would be mapped to the same file in the fat jar. The default implementation could be mergeReferenceConfs or "log a warning and return Right(seq.head)" or Left(<collision warning>), which would abort the assembly with an error message.
3. mergeReferenceConfs (a new TaskKey[Seq[File] => Either[String, File]]) would return Right(<concatenated contents>) if the name is "reference.conf". It could be the default implementation of collisionResolution or (if you prefer) simply be available for use by anyone wanting it to apply to collisionResolution without actually being chained in by default.

I personally would prefer everything to be set up in a way that doesn't require any additional configuration from a user in order to have libs using the typesafe config be properly merged, but I could understand, if you wanted to keep sbt-assembly as "clean" as possible by default.

What do you think?

I get this:

Exception in thread "main" java.lang.SecurityException: no manifiest section for signature file entry org/bouncycastle/cms/CMSSign
edDataStreamGenerator$TeeOutputStream.class
        at sun.security.util.SignatureFileVerifier.verifySection(Unknown Source)
        at sun.security.util.SignatureFileVerifier.processImpl(Unknown Source)
        at sun.security.util.SignatureFileVerifier.process(Unknown Source)
        at java.util.jar.JarVerifier.processEntry(Unknown Source)
        at java.util.jar.JarVerifier.update(Unknown Source)

How do you deal with this issue?