sbt / util Goto Github PK

View Code? Open in Web Editor NEW

15.0 11.0 24.0 1.46 MB

util modules for sbt

License: Other

Scala 98.86% Java 1.14%

util's Introduction

sbt/util moved back to sbt/sbt

sbt/util moved back to sbt/sbt for 1.4.x and later.

util's People

Contributors

Stargazers

Watchers

util's Issues

util-logging broken in devel on 2.10

A recent commit added "silencer-plugin" and "silencer-lib" to sbt/util, but the libraries have never been released on 2.10: https://mvnrepository.com/artifact/com.github.ghik/silencer-lib
The subproject "util-logging" includes silencer, and is cross compiled against 2.10:

util/build.sbt

Lines 101 to 105 in cf04676

 crossScalaVersions := Seq(scala210, scala211, scala212), 

 name := "Util Logging", 

 libraryDependencies ++= 

 Seq(jline, log4jApi, log4jCore, disruptor, sjsonnewScalaJson.value, scalaReflect.value, 

 compilerPlugin(silencerPlugin), silencerLib),

sbt:Util Logging> +compile
[...]
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.github.ghik#silencer-lib_2.10;1.2: not found
[warn] 	:: com.github.ghik#silencer-plugin_2.10;1.2: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::

FilesInfo's JsonFormat stackoverflows

util/internal/util-cache/src/main/scala/sbt/internal/util/FileInfo.scala

Line 172 in 613bf8b

implicit def format[F <: FileInfo]: JsonFormat[FilesInfo[F]] = implicitly

I'm pretty sure

implicit def format[F <: FileInfo]: JsonFormat[FilesInfo[F]] = implicitly

is never going to work.

Don't publish repositories information (at least for releases)

Remove all additional resolvers from build.sbt, at least for releases, because artifacts published to Maven central repository should not contain additional repositories, and in fact don't need them, because all dependencies are already in Maven central repo.

Util needs an io bump

It's still on 1.2.1.

non-blocking readline

We need a way of driving JLine in a non-blocking way.
(Perhaps we don't use JLine at first).

We need to be a able to listen to network requests in parallel as we listen to humans for the command in console.

Kill this repo

and move it's pieces in whatever downstream sbt module requires them.

scripted tests gets stuck

steps

Use util 0.1.0-M9 in sbt/sbt
Have some error in scripted test such that the build doesn't load

problem

The scripted gets stuck forever.

expectation

scripted comes back with error.

notes

When the build doesn't load sbt waits for the user's input using SimpleReader.readLine:

val result = (SimpleReader.readLine("Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? ") getOrElse Quit).toLowerCase(Locale.ENGLISH)

Thread sleep injection seem to interfere with this. Work around is to not to inject thread sleep by default.

Output Caching caches value from before effect function is called, not after

In 0.13, the output value () ⇒ A1 function is called twice, including after the call to f: (Boolean, A1) => A2 (the function signature was (Boolean, I) ⇒ O in 0.13).

This changed in 1.x, so now the output value function p is only called once and assigned to a value.

util/util-tracking/src/main/scala/sbt/util/Tracked.scala

Lines 98 to 103 in 3811d2a

 val initial = p() 

 val help = new CacheHelp(cache) 

 val changed = help.changed(store, initial) 

 val result = f(changed, initial) 

 if (changed) { 

 help.save(store, initial)

This changes the behavior of the caching function. For example, files that did not exist prior to the call to f but were created by f are now cached as missing, where there would have been cached as created in 0.13. (I believe this is the root cause of sbt/sbt-assembly#289 and Dwolla/sbt-s3-publisher#11.)

I think I have a fix for the issue, but I'm having trouble using the locally built version of sbt to run scripted tests for sbt-assembly. (It seems to be using my local 1.2.0-SNAPSHOT version of sbt but with the 1.2.0-M2 version of the util-tracking jar, not the local one.) I'll probably push up a PR anyway, to try to move things forward.

LogExchangeSpec failed

I just observed the following message in a nightly compilation of sbt-util:

[17:48:27] [sbt-util] [info] LogExchangeSpec:
[17:48:27] [sbt-util] [info] LogExchange.stringTypeTagThrowable
[17:48:27] [sbt-util] [info] - should match real StringTypeTag[Throwable] *** FAILED ***
[17:48:27] [sbt-util] [info]   scala.Throwable did not equal Throwable (LogExchangeSpec.scala:15)

This was in a compilation of sbt-util 1.x (commit b643c98), using the nightly sbt version 1.1.5-bin-20180612T215009. Full log: https://jenkins.scala-sbt.org:8592/job/sbt-community/340/consoleFull

util-interface pulls in scala-library-2.11.7.jar

Should be Java only.

NullPointerException in StringTypeTag$.typeToString(StringTypeTag.scala:27)

nondeterministic failure 🤔

git clone https://github.com/unfiltered/unfiltered.g8.git
cd unfiltered.g8
git checkout 56d8b7cac678e869fa5da89323bf29f988f0296f
sbt ';set g8Properties in g8 in Test ~= { _ ++ Map(("scala_version" -> "2.10.6"), ("sbt_version" -> "1.0.0")) }; g8Test'

[info] [error] java.lang.NullPointerException
[info] [error] 	at sbt.internal.util.StringTypeTag$.typeToString(StringTypeTag.scala:27)
[info] [error] 	at sbt.internal.util.StringTypeTag$.apply(StringTypeTag.scala:15)
[info] [error] 	at sbt.internal.util.ManagedLogger.registerStringCodec(ManagedLogger.scala:39)
[info] [error] 	at sbt.internal.util.ManagedLogger.<init>(ManagedLogger.scala:45)
[info] [error] 	at sbt.util.LogExchange.logger(LogExchange.scala:32)
[info] [error] 	at sbt.internal.LogManager$.defaultLogger(LogManager.scala:122)
[info] [error] 	at sbt.internal.LogManager$DefaultLogManager.apply(LogManager.scala:99)
[info] [error] 	at sbt.internal.LogManager$.$anonfun$construct$1(LogManager.scala:47)
[info] [error] 	at sbt.std.Streams$$anon$3$$anon$2.log(Streams.scala:173)
[info] [error] 	at sbt.std.TaskStreams.log(Streams.scala:88)
[info] [error] 	at sbt.std.TaskStreams.log$(Streams.scala:88)
[info] [error] 	at sbt.std.Streams$$anon$3$$anon$2.log$lzycompute(Streams.scala:137)
[info] [error] 	at sbt.std.Streams$$anon$3$$anon$2.log(Streams.scala:137)
[info] [error] 	at sbt.Defaults$.$anonfun$compileInputsSettings$2(Defaults.scala:1434)
[info] [error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[info] [error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:42)
[info] [error] 	at sbt.std.Transform$$anon$4.work(System.scala:64)
[info] [error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
[info] [error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[info] [error] 	at sbt.Execute.work(Execute.scala:266)
[info] [error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
[info] [error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
[info] [error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:32)
[info] [error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] [error] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[info] [error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info] [error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info] [error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info] [error] 	at java.lang.Thread.run(Thread.java:748)
[info] [error] (test:compile::compilerReporter) java.lang.NullPointerException
[info] [error] Total time: 27 s, completed Aug 14, 2017 11:12:35 AM

Forward port JLine bump

sbt/sbt#2173

ConsoleLogger.removeEscapeSequences throws StringIndexOutOfBoundsException

From https://travis-ci.org/sbt/util/jobs/189181474#L849:

[info] ! Escapes.No escape characters remain after removeEscapeSequences: Exception raised on property evaluation.
[info] > ARG_0: "\u001b"
[info] > ARG_0_ORIGINAL: "㫅ᜨ㭾\u001b"
[info] > Exception: java.lang.StringIndexOutOfBoundsException: String index out of range: 1

Reproduced in the REPL:

> utilLogging/console
[info] Starting scala interpreter...
[info]
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala> sbt.internal.util.ConsoleLogger.removeEscapeSequences("㫅ᜨ㭾\u001b")
java.lang.StringIndexOutOfBoundsException: String index out of range: 4
  at java.lang.String.charAt(String.java:658)
  at sbt.internal.util.ConsoleLogger$.nextESC(ConsoleLogger.scala:75)
  at sbt.internal.util.ConsoleLogger$.removeEscapeSequences(ConsoleLogger.scala:63)
  ... 42 elided

jdk 7 compatibility

Starting 2573c0f , util-interface is compiled/published using jdk8, which means in order to consume zinc >= 1.0.0-X7 we would have to start telling pants users stop using jdk7, which i guess is not totally unreasonable given jdk7 reaches its eol for already 2 years? but honestly i don't know.

Would really appreciate we start with a plan and communicate first whichever decision we choose as opposed to drop immediately.

Move sbt's `testingProj` and `testAgentProj` to this repository

There are two projects in sbt that do not depend on other non-extracted sbt component:

// Runner for uniform test interface
lazy val testingProj = (project in file("testing")).
  dependsOn(testAgentProj).
  settings(
    baseSettings,
    name := "Testing",
    libraryDependencies ++= Seq(testInterface, launcherInterface % Compile, ioProj, classpathProj, logProj)
  )

// Testing agent for running tests in a separate process.
lazy val testAgentProj = (project in file("testing") / "agent").
  settings(
    minimalSettings,
    name := "Test Agent",
    libraryDependencies += testInterface
  )

Should we extract them to this repository? There's already a subproject name util-testing in this repository, so it may make sense.

Only project depending on this:

// Implementation and support code for defining actions.
lazy val actionsProj = (project in mainPath / "actions").
  dependsOn (runProj, stdTaskProj, taskProj, testingProj).
  settings(
    testedBaseSettings,
    name := "Actions",
    libraryDependencies ++= Seq(classpathProj, completeProj, apiProj, compilerIntegrationProj, compilerIvyProj,
      interfaceProj, ioProj, ivyProj, logProj, relationProj, trackingProj)
  )

Tracked.inputChanged is broken

Input stores JSON of the input value, but compared using the hash.

Reformat *.sbt files

Revert #123 ensuring it doesn't interact badly with +compile.

utilCollection needs 2.10 artifacts

[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: org.scala-sbt#util-collection_2.10;1.0.0-M19: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn] 	Note: Unresolved dependencies path:
[warn] 		org.scala-sbt:util-collection_2.10:1.0.0-M19
[warn] 		  +- org.scala-sbt:util-logging_2.10:1.0.0-M19 (/home/travis/build/sbt/zinc/project/Dependencies.scala#L46)
[warn] 		  +- org.scala-sbt:zinc-classfile_2.10:1.0.0-X8-SNAPSHOT
[info] Wrote /home/travis/build/sbt/zinc/internal/zinc-apiinfo/target/scala-2.10/zinc-apiinfo_2.10-1.0.0-X8-SNAPSHOT.pom
sbt.ResolveException: unresolved dependency: org.scala-sbt#util-collection_2.10;1.0.0-M19: not found

Regression in `Changed#uptodate`?

See the [discussion(https://github.com/sbt/sbt-scalariform/pull/57#issuecomment-319374696) in sbt/sbt-scalariform#57 for context.

This works on SBT 0.13.16 (sbt.version=0.13.16 in build.properties; run sbt bleh to reproduce):

sbtPlugin := true

val bleh = taskKey[Unit]("bleh")

bleh := {
  import sbt.Cache.uriFormat
  val changed = new Changed[URI](streams.value.cacheDirectory / "bleh")
  val res = changed(_ => false, _ => true)(new URI("http://bogus"))
  println(s"changed=$res")
}

But converted and run on 1.0.0-RC3 it produces an error since the file does not exist/is empty:

sbtPlugin := true

val bleh = taskKey[Unit]("bleh")

bleh := {
  import sbt.util.CacheStore
  import sjsonnew.BasicJsonProtocol._
  val changed = new Changed[String](CacheStore(streams.value.cacheDirectory / "bleh"))
  val res = changed(_ => false, _ => true)("hello")
  println(s"changed=$res")
}

As @godenji pointed out, 0.13, Changed returns false on an exception but not in 1.0. Is this a regression or is Changed supposed to be used in a different manner in 1.0?

A hidden file is included as a scripted test

I've got a failure like this:

sbt.internal.io.ResourcesException: Resource directory '.DS_Store' in group 'source-dependencies' not found.
    at sbt.internal.io.Resources$.error(Resources.scala:25)
    at sbt.internal.io.Resources.readOnlyResourceDirectory(Resources.scala:41)
    at sbt.internal.io.Resources.readWriteResourceDirectory(Resources.scala:47)
    at sbt.internal.scripted.ScriptedTests$$anonfun$scriptedTest$2$$anonfun$apply$2$$anonfun$apply$3.apply(ScriptedTests.scala:59)
    at sbt.internal.scripted.ScriptedTests$$anonfun$scriptedTest$2$$anonfun$apply$2$$anonfun$apply$3.apply(ScriptedTests.scala:57)
    at sbt.internal.scripted.ScriptedRunnerImpl$$anonfun$2.apply(ScriptedTests.scala:23)
    at sbt.internal.scripted.ScriptedRunnerImpl$$anonfun$2.apply(ScriptedTests.scala:23)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at sbt.internal.scripted.ScriptedRunnerImpl$.runAll(ScriptedTests.scala:23)
    at sbt.internal.scripted.ScriptedRunnerImpl$.run(ScriptedTests.scala:20)
    at sbt.internal.inc.IncScriptedRunner.run(IncScriptedRunner.scala:11)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at Scripted$.doScripted(Scripted.scala:74)
    at $0e73639291fc1196c7f3$$anonfun$scriptedTask$2$$anonfun$apply$6$$anonfun$apply$7.apply(build.sbt:264)
    at $0e73639291fc1196c7f3$$anonfun$scriptedTask$2$$anonfun$apply$6$$anonfun$apply$7.apply(build.sbt:261)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] (zincRoot/*:scripted) sbt.internal.io.ResourcesException: Resource directory '.DS_Store' in group 'source-dependencies' not found.
[error] Total time: 2 s, completed Apr 7, 2016 12:01:20 PM

The implementation tries to filter out hidden files already:

final class ListTests(baseDirectory: File, accept: ScriptedTest => Boolean, log: Logger) {
  def filter = DirectoryFilter -- HiddenFileFilter

but it doesn't work.

Publish for Scala 2.13.0

As with sbt-io a Scala 2.13.0 supported release of sbt-util is needed

FileInfo hashes use List[Byte]

FileInfo hashes are using List[Byte] to store sha1 hashes. When using Array[Byte], a sha1 hash takes up approx 40 bytes of contiguous memory (including object headers etc). By contrast, a List[Byte] is stored as a collection of :: nodes (but not java.lang.Byte instances because these are cached) and can take up to 32 bytes of memory per byte. In total, a sha1 hash stored as a List[Byte] can take up to 640 bytes of memory, across 20 non cached objects, requiring 41 object dereferences to iterate over. I don't know how big an impact this would actually have on performance, but it seems like such a needless waste - especially considering the classes are private and hence immutability would effectively be guaranteed when using Array[Byte].

Signature `Initialize.joinAny` is wrong

It looks like the changes made to Initialize.joinAny in #4 change the meaning of its signature (apart from adding a new type parameter).
sbt's codebase was compiling before, but it's not compiling anymore because it misses one type parameter.

Original definition of Initialise.joinAny:

def joinAny[M[_]](inits: Seq[Initialize[M[T]] forSome { type T }]): Initialize[Seq[M[_]]] =
  join(inits.asInstanceOf[Seq[Initialize[M[Any]]]]).asInstanceOf[Initialize[Seq[M[T] forSome { type T }]]]

Modified version:

def joinAny[M[_], T](inits: Seq[Initialize[M[T]]]): Initialize[Seq[M[_]]] =
  join(inits.asInstanceOf[Seq[Initialize[M[Any]]]]).asInstanceOf[Initialize[Seq[M[T] forSome { type T }]]]

The compilation error in sbt's codebase happens in the definition of RichInitializeTask.nonLocal.

Original version of RichInitializeTask.nonLocal:

type AnyInitTask = Initialize[Task[T]] forSome { type T }
private[this] def nonLocal(tasks: Seq[AnyInitTask], key: AttributeKey[Seq[Task[_]]]): Initialize[Task[S]] =
  (Initialize.joinAny[Task](tasks), i) { (ts, i) => i.copy(info = i.info.set(key, ts)) }

My first guess is:

type AnyInitTask = Initialize[Task[Any]]
private[this] def nonLocal(tasks: Seq[AnyInitTask], key: AttributeKey[Seq[Task[_]]]): Initialize[Task[S]] =
  (Initialize.joinAny[Task, Any](tasks), i) { (ts, i) => i.copy(info = i.info.set(key, ts)) }

This compiles, but I'm really not sure that this is correct and has the same meaning as the original definition. Is there a difference between the two definitions of type AnyInitTask?

Maybe that there shouldn't be a second type parameter in the definition of Initialize.joinAny, because it would always be Any?

	crossScalaVersions := Seq(scala210, scala211, scala212),
	name := "Util Logging",
	libraryDependencies ++=
	Seq(jline, log4jApi, log4jCore, disruptor, sjsonnewScalaJson.value, scalaReflect.value,
	compilerPlugin(silencerPlugin), silencerLib),

	val initial = p()
	val help = new CacheHelp(cache)
	val changed = help.changed(store, initial)
	val result = f(changed, initial)
	if (changed) {
	help.save(store, initial)