Simplify merging of typesafe config 'reference.conf' files about sbt-assembly

[eed3si9n]

I like your idea.

How about calling collisionResolution mergeStrategy instead with the default value set to MergeStrategy.discard? If you also provide MergeStrategy.append, Akka users can just say

mergeStrategy in assembly := MergeStrategy.append

[sirthias]

Very good!

What would MergeStrategy.discard do? Pick the first and discard the rest?
Then MergeStrategy.PickFirst might be a better name. And could MergeStrategy.append be even clearer as MergeStrategy.Concatenate?

[eed3si9n]

MergeStrategy.discard would kill both the old and new file.

1. I think the strategies should be a function value so it should start with lower case.
2. append in collection is adding an element to a collection, while concat used to be adding multiple collections together, so I think append is better.
3. Selecting the first one should be called MergeStrategy.head continuing on the collections metaphor. But I don't like it since it's not clear to the user which file is the "head."

[eed3si9n]

Actually, the default probably should be MergeStrategy.error, which throws exception the moment conflict is detected.
I feel like the most conservative thing to is here to let the build user know there's a problem and may present her with strategy options in the error message.

[rkuhn]

Thanks, Eugene, for looking into this; I agree with your proposal, there is only one thing missing, as we basically need a mapping from potential archive member name to MergeStrategy (it is easily imaginable that different things merge differently and the default could be

name match {
  case "reference.conf" => append
  case _ => error
}

where the name is the full member path name. This would allow multiple settings to be composed more easily.

[rkuhn]

Just to let you know: I’m beginning to work on this, so please holler if you’re also doing that.

[eed3si9n]

Yea, I think Mathias was saying pull request so I haven't done anything.

One thing I was thinking is that intent of the strategies may get clearer if the signature was TaskKey[(File, File) => Either[String, File]] since files would be unzipped one jar at a time, right?

I think keeping the strategies function values would make it composable.
I could consider adding a new strategy called MergeStrategy.appendReferenceConf, but I am against making that the default strategy.

[rkuhn]

Separating the actual strategy from the selection of when to apply it would make it more reusable, but I’ll let the code speak once I have it roughly implemented.

Quick question: I don’t see why the strategy selection should not be a SettingKey, what do I miss?

[eed3si9n]

@rkuhn I copied it from original proposal, but you're right about the SettingKey. At least so far, there's no dependance to sources.

[sirthias]

Gents,

sorry for having gone dark for a day (the weekend being holy, and all... ;-)

Couple of comments:

• Casing: I was thinking about the default strategies being implemented as objects and/or regarded as
  constants. Still, if you prefer lowercasing, no problem.
• Naming: The differences in our naming approach appear to stem from a differing perspective. E.g. 'append'
  is the action to take when you have the first file and process a second, so it's the low-level
  perspective of what to do at each step of a merge. My perspective is more high-level, from the users
  point of view onto the end product of the process.
• Composing of strategies: Theorectically we could put in a composable mini-DSL mapping strategies to file
  name matches with regexes, and so on. But I agree with Roland to KISS and go for simple pattern matching.
• TaskKey vs. SettingKey: I'm with you, should have been SettingKey all along.

So, maybe the interface could look like this:

val mergeStrategy = SettingKey[PartialFunction[String, MergeStrategy]]

type MergeStrategy = Seq[File] => Either[String, File]

Making the mergeStrategy key a PartialFunction would remove the need for a default case mapping to error and increase composability. The default strategies could then be

object MergeStrategy {
  val pickOne = ... // arbitrarily select one of the inputs, implemented as `_.head`
  val concat = ... // merge by concatenating all input files
  val error = ... // throw exception
}

Theoretically we could also drop the error strategy if throwing an exception is the default case of the
PartialFunction not being defined for the particular case. But still putting it in allows one to "short-circuit" potential long 'orElse' chains of strategies.

I'd second Roland in voting for the default mergeStrategy to be

case "reference.conf" => MergeStrategy.concat

so the plugin would work "out of the box". I think that we'll see the use of typesafe.config based libs increase even more, so that the large majority of users would have to all apply the same additional setting in order to make things work.

[rkuhn]

Please have a look at #37

Apart from append vs. concat we are in agreement. Eugene, how do you run tests to verify that I didn't break service providers? I just manually looked at what my uniqueLines strategy does, and it seems to me that it should correspond to the previous special handling for META-INF/services apart from the pruning of empty lines and the trimming.

[eed3si9n]

I think that we'll see the use of typesafe.config based libs

If this is the case, could we merge any *.config?

[eed3si9n]

@rkuhn I've written a simple scripted tests, which you can invoke by changing the version numbers of src/sbt-test/sbt-assembly/*/project/plugins.sbt files and running scripted from sbt.
If you want to create a test case for this, go right ahead. See testing sbt plugins.

[rkuhn]

My latest push includes test cases for the merge strategies. Since there was not test case for the META-INF/services special handling, would you please verify that I didn’t break anything in that regard? I have not used that feature before. What do you think of my new default strategy: if the conflicting files have the very same contents, just pick the first one (else error).

Wrt. merging any *.config: reference.conf is the only hard default case. There is a softer convention of putting the project specific things into application.conf, but that is not as special and should not routinely require merging, so I’d vote for just letting the user specify that if they want, it’s easy enough:

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
    case "application.conf" => MergeStrategy.concat
    case x => old(x)
  }
}

The sad aspect is that orElse would require a bulky type ascription.

[eed3si9n]

META-INF/services is something I inherited from assembly-sbt, and I've never used it myself. I think the current implementation looks ok.

Could the strategy take Logger as another param and log.info as it merges conflict files? For example, "Merging reference.conf by concatenating." or something?

I think mustEqual is fine, but not sure about the name yet. allowDupe, ignoreIdentical, sameOrError?

I was hoping to rid of any Akka-ness out of sbt-assembly, but if reference.conf is the only known case, I guess it's ok.

Could you document overview of this feature, individual strategies, and how to add custom strategies in README; and the release note in notes/0.7.5.markdown?

[rkuhn]

META-INF/services is something I inherited from assembly-sbt, and I've never used it myself. I think the current implementation looks ok.

Okay, thanks for the cross-check.

Could the strategy take Logger as another param and log.info as it merges conflict files? For example, "Merging reference.conf by concatenating." or something?

Sounds like a reasonable idea, need to think about it a bit, though, because then I’ll have to pass in the name, too (making it four arguments instead of two).

I think mustEqual is fine, but not sure about the name yet. allowDupe, ignoreIdentical, sameOrError?

allowDuplicates?

I was hoping to rid of any Akka-ness out of sbt-assembly, but if reference.conf is the only known case, I guess it's ok.

Could you document overview of this feature, individual strategies, and how to add custom strategies in README; and the release note in notes/0.7.5.markdown?

Yes, of course. I wanted to wait with that until the details are clear ;-)

[sirthias]

With regard to the MergeStrategy naming:
Looking at the pattern shaping up then pickFirst, pickLast, concat and error read like "commands" indicating what to do with colliding files. uniqueLines and mustEqual don't really fit that pattern. selectUniqueLines and errorIfDifferent could be one way of establishing naming pattern consistency here.

Otherwise I think Rolands pull request is right on the spot! Thanks guys for being so incredibly responsive. Can't wait for 0.7.5 to be released... :-)

[eed3si9n]

Since we can talk about the specifics in #37, I am going to close this one.