bazelbuild / rules_scala

Scala rules for Bazel

Scala Rules for Bazel

Where to get help

• #scala @ Bazel Slack
• Google group
• Gitter chat

Overview

Bazel is a tool for building and testing software and can handle large, multi-language projects at scale.

This project defines core build rules for Scala that can be used to build, test, and package Scala projects.

Rules

• scala_library
• scala_macro_library
• scala_binary
• scala_test
• scala_repl
• scala_library_suite
• scala_test_suite
• thrift_library
• scala_proto_library
• scala_toolchain
• scala_import
• scala_doc

Getting started

1. Install Bazel, see the compatibility table.
2. Add the following to your WORKSPACE file and update versions with their sha256s if needed:

# WORKSPACE
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "bazel_skylib",
    sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz",
        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz",
    ],
)

# See https://github.com/bazelbuild/rules_scala/releases for up to date version information.
http_archive(
    name = "io_bazel_rules_scala",
    sha256 = "71324bef9bc5a885097e2960d5b8effed63399b55572219919d25f43f468c716",
    strip_prefix = "rules_scala-6.2.1",
    url = "https://github.com/bazelbuild/rules_scala/releases/download/v6.2.1/rules_scala-v6.2.1.tar.gz",
)

load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config")
# Stores Scala version and other configuration
# 2.12 is a default version, other versions can be use by passing them explicitly:
# scala_config(scala_version = "2.11.12")
# Scala 3 requires extras...
#   3.2 should be supported on master. Please note that Scala artifacts for version (3.2.2) are not defined in
#   Rules Scala, they need to be provided by your WORKSPACE. You can use external loader like
#   https://github.com/bazelbuild/rules_jvm_external
scala_config()

load("@io_bazel_rules_scala//scala:scala.bzl", "rules_scala_setup", "rules_scala_toolchain_deps_repositories")

# loads other rules Rules Scala depends on 
rules_scala_setup()

# Loads Maven deps like Scala compiler and standard libs. On production projects you should consider 
# defining a custom deps toolchains to use your project libs instead 
rules_scala_toolchain_deps_repositories(fetch_sources = True)

load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()

load("@io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()

# optional: setup ScalaTest toolchain and dependencies
load("@io_bazel_rules_scala//testing:scalatest.bzl", "scalatest_repositories", "scalatest_toolchain")
scalatest_repositories()
scalatest_toolchain()

This will load the rules_scala repository at the commit sha rules_scala_version into your Bazel project and register a scala_toolchain at the default Scala version (2.12.18)

Then in your BUILD file just add the following so the rules will be available:

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_binary", "scala_test")

You may wish to have these rules loaded by default using bazel's prelude. You can add the above to the file tools/build_rules/prelude_bazel in your repo (don't forget to have a, possibly empty, BUILD file there) and then it will be automatically prepended to every BUILD file in the workspace.

To run with a persistent worker (much faster), you need to add

build --strategy=Scalac=worker
build --worker_sandboxing

to your command line, or to enable by default for building/testing add it to your .bazelrc.

Coverage support

It will produce several .dat files with results for your targets.

You can also add more options to receive a combined coverage report:

bazel coverage \
  --combined_report=lcov \
  --coverage_report_generator="@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main" \
  //...

This should produce a single bazel-out/_coverage/_coverage_report.dat from all coverage files that are generated.

You can extract information from your coverage reports with lcov:

# For a summary:
lcov --summary your-coverage-report.dat
# For details:
lcov --list your-coverage-report.dat

If you prefer an HTML report, then you can use genhtml provided also by the lcov package.

Coverage support has been only tested with ScalaTest.

Please check coverage.md for more details on coverage support.

Selecting Scala version

With toolchains

Rules scala supports the last two released minor versions for each of Scala 2.11, 2.12, 2.13. Previous minor versions may work but are supported only on a best effort basis.

To configure Scala version you must call scala_config(scala_version = "2.xx.xx") and configure dependencies by declaring scala_toolchain. For a quick start you can use scala_repositories() and scala_register_toolchains(), which have dependency providers configured for 2.11.12, 2.12.18 and 2.13.12 versions.

# WORKSPACE
load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config")
scala_config(scala_version = "2.13.12")

load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()

load("@io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()

Note: Toolchains are a more flexible way to configure dependencies, so you should prefer that way. Please also note, that the overriden_artifacts parameter is likely to be removed in the future.

Bazel compatible versions

minimal bazel version	rules_scala gitsha
5.3.1	HEAD
4.1.0	a40063ef97688f056824b22b9e49fae6efd1df0f
3.5.0	0f55e9f8cff6494bbff7cd57048d732286a520f5
2.0.0	116709091e5e1aab3346184217b565f4cb7ba4eb
1.1.0	d681a952da74fc61a49fc3167b03548f42fc5dde
0.28.1	bd0c388125e12f4f173648fc4474f73160a5c628
0.23.x	ca655e5a330cbf1d66ce1d9baa63522752ec6011
0.22.x	f3113fb6e9e35cb8f441d2305542026d98afc0a2
0.16.x	f3113fb6e9e35cb8f441d2305542026d98afc0a2
0.15.x	3b9ab9be31ac217d3337c709cb6bfeb89c8dcbb1
0.14.x	3b9ab9be31ac217d3337c709cb6bfeb89c8dcbb1
0.13.x	3c987b6ae8a453886759b132f1572c0efca2eca2

See the configuration file for the exact versions verified with the continuous-integration builds.

Breaking changes

If you're upgrading to a version containing one of these commits, you may encounter a breaking change where there was previously undefined behavior.

• 929b318 on 2020-01-30: Fixed a bug in the JMH benchmark build that was allowing build failures to creep through. Previously you were able to build a benchmark suite with JMH build errors. Running the benchmark suite would only run the successfully-built benchmarks.

Usage with bazel-deps

Bazel-deps allows you to generate bazel dependencies transitively for maven artifacts. Generally we don't want bazel-deps to fetch scala artifacts from maven but instead use the ones we get from calling scala_repositories. The artifacts can be overridden in the dependencies file used by bazel-deps:

replacements:
  org.scala-lang:
    scala-library:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_library//:io_bazel_rules_scala_scala_library"
    scala-reflect:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_reflect//:io_bazel_rules_scala_scala_reflect"
    scala-compiler:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler"

  org.scala-lang.modules:
    scala-parser-combinators:
      lang: scala
      target:
        "@io_bazel_rules_scala_scala_parser_combinators//:io_bazel_rules_scala_scala_parser_combinators"
    scala-xml:
      lang: scala
      target:
        "@io_bazel_rules_scala_scala_xml//:io_bazel_rules_scala_scala_xml"

Publishing to Maven repository

Go here

Dependency Tracking

Rules Scala supports multiple dependency modes including strict and unused dependency tracking. See Dependency Tracking for more info.

Advanced configurable rules

To make the ruleset more flexible and configurable, we introduce a phase architecture. By using a phase architecture, where rule implementations are defined as a list of phases that are executed sequentially, functionality can easily be added (or modified) by adding (or swapping) phases.

Phases provide 3 major benefits:

• Consumers are able to configure the rules to their specific use cases by defining new phases within their workspace without impacting other consumers.
• Contributors are able to implement new functionalities by creating additional default phases.
• Phases give us more clear idea what steps are shared across rules.

See Customizable Phase for more info.

Phase extensions

• Scala Format

Building from source

Setup bazel: We recommend using Bazelisk as your default bazel binary

Test & Build:

bash test_all.sh

You can also use:

bazel test //test/...

Note bazel test //... will not work since we have a sub-folder on the root folder which is meant to be used in a failure scenario in the integration tests. Similarly to only build you should use bazel build //src/... due to that folder.

Updates

This section contains a list of updates that might require action from the user.

• 043ba58: Updates default scrooge version to 18.6.0
• 76d4ab9: Updates naming for scala artifact replacements in bazel-deps. See Usage with bazel-deps

Contributing

See CONTRIBUTING.md for more info.

Adopters

Here's a (non-exhaustive) list of companies that use rules_scala in production. Don't see yours? You can add it in a PR!

• Ascend
• Canva
• Domino Data Lab
• Etsy
• Gemini
• Grand Rounds
• Kitty Hawk
• Meetup
• Spotify
• Stripe
• Tally
• Twitter
• VirtusLab
• VSCO
• Wix

rules_scala's Issues

Possible confusion in the docs //scala:scala.bzl vs @io_bazel_rules_scala//scala:scala.bzl

First of all thanks a lot for all your hard work, it's really appreciated!

I'm not sure this is in the scope of this repo but the fact of the matter is that I went along the readme and since it mentions the load with respect to prelude_bazel I wasn't sure if there isn't a different syntax for that use-case.
I went to the tests since they are the executable specification and failed since they use the absolute(?) path //scala:scala.bzl
This is an on-boarding issue since if I was familiar with Bazel syntax well enough I wouldn't have made that mistake but I think we want to make on-boarding easier.
2 options I had in mind:

1. Add a note in the readme at the end of the overview explaining the difference
2. Add a scala app to the bazelbuild/examples repo
   The added value of #2 is that it serves as a bit of a test application since it has to use the syntax like "regular" users.

I'd be happy to contribute both options but didn't want to barge in with a PR before we discuss it.

ci.bazel.io build is broken

Broken by PR #29. See http://ci.bazel.io/job/rules_scala/105/

Linux (with sandboxing):

____From making thrift archive //test/src/main/scala/scala/test/twitter_scrooge/thrift:thrift:
external/local_jdk/bin/jar: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
external/local_jdk/bin/jar: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
ERROR: /home/ci/workspace/rules_scala/BAZEL_VERSION/HEAD/PLATFORM_NAME/linux-x86_64/test/src/main/scala/scala/test/twitter_scrooge/thrift/BUILD:3:1: output 'test/src/main/scala/scala/test/twitter_scrooge/thrift/libthrift.jar' was not created.
ERROR: /home/ci/workspace/rules_scala/BAZEL_VERSION/HEAD/PLATFORM_NAME/linux-x86_64/test/src/main/scala/scala/test/twitter_scrooge/thrift/BUILD:3:1: not all outputs were created.
ERROR: /home/ci/workspace/rules_scala/BAZEL_VERSION/HEAD/PLATFORM_NAME/linux-x86_64/test/src/main/scala/scala/test/twitter_scrooge/thrift/thrift2/thrift3/BUILD:3:1: output 'test/src/main/scala/scala/test/twitter_scrooge/thrift/thrift2/thrift3/libthrift3.jar' was not created.
ERROR: /home/ci/workspace/rules_scala/BAZEL_VERSION/HEAD/PLATFORM_NAME/linux-x86_64/test/src/main/scala/scala/test/twitter_scrooge/thrift/thrift2/BUILD:3:1: output 'test/src/main/scala/scala/test/twitter_scrooge/thrift/thrift2/libthrift2_a.jar' was not created.

OS X:

ERROR: /Users/ci/workspace/rules_scala/BAZEL_VERSION/HEAD/PLATFORM_NAME/darwin-x86_64/test/src/main/scala/scala/test/twitter_scrooge/BUILD:4:1: creating scrooge files //test/src/main/scala/scala/test/twitter_scrooge:scrooge1_srcjar failed: generator failed: error executing command 
  (cd /private/var/tmp/_bazel_ci/ac44d3454b4099e3ebd2d39821cf0e13/darwin-x86_64 && \
  exec env - \
  bazel-out/local-fastbuild/bin/src/scala/scripts/generator bazel-out/local-fastbuild/bin/test/src/main/scala/scala/test/twitter_scrooge/test/src/main/scala/scala/test/twitter_scrooge/libscrooge1_srcjar.srcjar_only_transitive_thrift_srcs bazel-out/local-fastbuild/bin/test/src/main/scala/scala/test/twitter_scrooge/test/src/main/scala/scala/test/twitter_scrooge/libscrooge1_srcjar.srcjar_immediate_thrift_srcs bazel-out/local-fastbuild/bin/test/src/main/scala/scala/test/twitter_scrooge/libscrooge1_srcjar.srcjar bazel-out/local-fastbuild/bin/test/src/main/scala/scala/test/twitter_scrooge/test/src/main/scala/scala/test/twitter_scrooge/libscrooge1_srcjar.srcjar_remote_jars): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 127.
bazel-out/local-fastbuild/bin/src/scala/scripts/generator: line 3: bazel-out/local-fastbuild/bin/src/scala/scripts/generator.runfiles/external/local_jdk/bin/java: No such file or directory

Add documentation on how to use thrift/scrooge

@jcoveney

see #29

improve handling of java exports

They should be exported to skylark in the next release after 0.2.0:

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/bazel-discuss/CRd1BbkHVUw/X6KgxPgjIAAJ

at this location:
https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L177

The issue now is that we always pull the transitive compilation deps for java libraries, which hides dependencies from bazel.

General scrooge performance

Right now, the code generator is not as performant as it could be... of course, this is mainly with antipatterns (a ton of thrifts which depend on a ton of thrifts) that hopefully bazel will prevent, but which exist at many companies which heavily use thrift. There is a lot of work that could be done to make this maximally performant... some ideas

• Make scrooge imports jars #44
• Use the full graph of srcjars/thrift jars to minimize the number of times we extract jars
• ??? (there are scrooge changes that could also probably better support this... such as keeping around some intermediate data or something, but at that point maybe we just want smaller targets...)

Command line option for more verbose debug logging?

This is a bigger issue for bazel in general, but it would be nice to be able to optionally have verbose logging in custom rules.

TwitterScroogeGenerator would be a good place to have some, and the spirit of this comment https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L102

add support for src jars

for code gen, such as scrooge, we probably want to create src jars, then reuse the existing scala rules. Right now we require .scala files as source.

I guess this would use find after we unjar to make sure all the files can be handled.

Issue with compiling multiple Java Source files

Hey Guys,

I've been trying to use scala_library to compile a mixture of both scala and java files. It seems the java file compilation fails when there are multiple java files. Up on looking at the error, it seems that the javac command that we invoke is taking the list of files as a single string block instead of taking it as a separate files. Below is the error that im getting while invoking javac

javac: file not found: infra/BetterOutput.java infra/Changeset.java infra/CompilationException.java
Usage: javac <options> <source files>
use -help for a list of possible options
Exception in thread "main" java.lang.RuntimeException: nope
	at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:345)
Caused by: java.lang.RuntimeException: javac process failed!
	at io.bazel.rulesscala.scalac.ScalaCInvoker.compileJavaSources(ScalaCInvoker.java:303)
	at io.bazel.rulesscala.scalac.ScalaCInvoker.processRequest(ScalaCInvoker.java:238)
	at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:341)
Target //infrastructure:infra-scala failed to build

Simplest fix was to append the file names directly to the command and that seems to be working fine

diff --git a/src/java/io/bazel/rulesscala/scalac/ScalaCInvoker.java b/src/java/io/bazel/rulesscala/scalac/ScalaCInvoker.java
index 4c64c2c..4208123 100644
--- a/src/java/io/bazel/rulesscala/scalac/ScalaCInvoker.java
+++ b/src/java/io/bazel/rulesscala/scalac/ScalaCInvoker.java
@@ -281,19 +281,13 @@ public class ScalaCInvoker {
       commandParts.add(ops.javacOpts);
     }

-    StringBuilder files = new StringBuilder();
-    int cnt = 0;
-    for(String javaFile : ops.javaFiles) {
-      if (cnt > 0) files.append(" ");
-      files.append(javaFile);
-      cnt += 1;
-    }
-
     commandParts.add("-classpath");
     commandParts.add(ops.classpath + ":" + tmpPath.toString());
     commandParts.add("-d");
     commandParts.add(tmpPath.toString());
-    commandParts.add(files.toString());
+    for(String javaFile : ops.javaFiles) {
+      commandParts.add(javaFile.toString());
+    }

     Process iostat = new ProcessBuilder(commandParts)
       .inheritIO()

Not sure if anyone has seen this behaviour when using multiple java source files. My BUILD file is pretty simple,

package(default_visibility = ["//visibility:public"])

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_binary", "scala_export_to_java")

scala_library(
    name = "infra-scala",
    srcs = glob([
        "*.scala",
        "*.java",
    ], exclude = ["TestError.java"]),
    deps = [
        "//third_party/jar:mvnrepo",
    ],
)

Remove the need for remote_jars in twitter_scrooge

https://github.com/bazelbuild/rules_scala/blob/master/twitter_scrooge/twitter_scrooge.bzl#L192

This feels unnecessary, we just need a POC/documentation on using new_git_repository or something like that to get rid of the need for it

Add support for compiler plugins

https://github.com/HairyFotr/linter

is probably a good test example.

We probably need the ability to have compiler_plugins=[":plugin"] and then find the full paths of the jars that are point to those.

support plugins on scala_library

The scala_library rule doesn't support the plugins attribute. Needed for things like the auto-service plugin which facilitates ServiceLoaders, e.g., to configure logback.

we should create a single xml output

We should create a new reporter:
https://github.com/scalatest/scalatest/blob/release-2.2.6-for-scala-2.11-and-2.10/src/main/scala/org/scalatest/Reporter.scala

which is basically a clone of this:
https://github.com/scalatest/scalatest/blob/master/scalatest/src/main/scala/org/scalatest/tools/JUnitXmlReporter.scala

except that it creates a single file given by the environmental variable $XML_OUTPUT_FILE

See this tool here:

https://gist.githubusercontent.com/cgoldberg/4320815/raw/75ba2b7cc8b477bc784631e28ffd586f79efa3f7/merge_junit_results.py

how to configure compiler plugins?

I am thinking that commonly, you might want a compiler plugin for the whole repo.

We might want to have something like scala_library_internal so then repos can make a local scala_library macro that sets up default compiler options.

That seems like it would work okay. Any better approach?

I don't know now namespacing works in skylark, but I assume I can't import a rule with a name foo and also define a rule with the same name (thus the work around of adding variants of _internal to all the rules so we can import just a few of those).

REPL support

I don't think that bazel has any notion a repl, but I can imagine adding macro like

scala_repl(name = "foo", deps = [":a", ":b"])

which will create a repl with the classpath given by the runtime dependencies of :a, :b.

Actually, remembering terminal pains, the easiest thing might be to do a real skylark rule with scala in the scala-sdk, or perhaps use ammonite: http://www.lihaoyi.com/Ammonite/#Features

remove scala_library _deploy.jar target

The scala_library _deploy.jar target has no equivalent meaning on the java side and is (probably?) unnecessary.

rule_scala is broken on ci.bazel.io

http://ci.bazel.io/job/rules_scala/158/BAZEL_VERSION=HEAD,PLATFORM_NAME=linux-x86_64/console

Caused by ebaeff6 which is merged in b471dfe

//cc @johnynek @colinmarc Can you help fix this?

add support for scrooge thrift generator

See: http://twitter.github.io/scrooge/CommandLine.html

something basic can be done probably with a genrule + a macro once we solve #2.

An problem (to me) is dealing with dependencies between thrift targets. One approach is to pull the transitive thrift files, compile all of them using --gen-file-map, then only put the output classes from the current target into the current jar. This may be too slow for deep dependencies however.

Support mixed scala/java targets

Scala rules version of bug bazelbuild/bazel#500. Not sure if there is still work to do or not.

scalac gives noisey warnings with small compile classpaths

See: https://issues.scala-lang.org/browse/SI-9673

For instance:

07:09:29 warning: Class org.apache.commons.logging.Log not found - continuing with a stub.
07:09:29 one warning found
which will be added to the runtime classpath, but is indeed not on the compile time classpath.

I don't see anything we can do here but hope scalac improves.

Build is not identical when scala rules compile java sources

See example failures in #101 #106

This seems to only happen on linux. The most common reason is that somehow a timestamp is making it into an output file. We should be able to check by looking at the binary diff to find the actual difference.

We need to fix this because we can't merge a test that makes the build go red, but at the same time, this means currently when using mixed scala/java targets on linux, you will not have optimal caching (each upstream rebuild will invalidate everything downstream).

```bazel run``` broken with 0.2.3

Haven't had time to look at this so if anyone has:
With bazel 0.2.3:

~/hello_world.scala git:master ❯❯❯ bazel build app
INFO: Found 1 target...
Target //:app up-to-date:
  bazel-bin/app
INFO: Elapsed time: 26.085s, Critical Path: 1.79s
~/hello_world.scala git:master ❯❯❯ bazel run app
INFO: Found 1 target...
Target //:app up-to-date:
  bazel-bin/app
INFO: Elapsed time: 0.467s, Critical Path: 0.00s

INFO: Running command line: bazel-bin/app
/private/var/tmp/_bazel_smparkes/08fe4a003be085753c237916361d8505/hello_world.scala/bazel-out/local-fastbuild/bin/app: line 3: /private/var/tmp/_bazel_smparkes/08fe4a003be085753c237916361d8505/hello_world.scala/bazel-out/local-fastbuild/bin/app.runfiles/external/local_jdk/bin/java: No such file or directory
ERROR: Non-zero return code '127' from command: Process exited with status 127.

./7e7209a62ba3a685a6bd98b8dbd655ea/scala/bazel-out/local-fastbuild/bin/app.runfiles/__main__/external/local_jdk/bin/java

and

./7e7209a62ba3a685a6bd98b8dbd655ea/scala/bazel-out/local-fastbuild/bin/app.runfiles/local_jdk/bin/java

exist but

/private/var/tmp/_bazel_smparkes/08fe4a003be085753c237916361d8505/hello_world.scala/bazel-out/local-fastbuild/bin/app.runfiles/external/local_jdk/bin/java

does not

test_run script doesn't clean hash outputs and doesn't really work on osx vanilla

Hi,
AFAIU OS X has md5 and not md5sum which is why I get ./test_run.sh: line 25: md5sum: command not found when running the script. Don't know if you want this to run on local contributors machines but if it's not important to you maybe it should be noted.
Second problem is that the hash1/hash2 files aren't deleted and aren't ignored.
Sounds to me like they should be deleted if the diff says they're the same. In any case ignoring them could also solve the problem although being a bit more hacky.
One last option is to maybe write them to bazel-out which is currently ignored.

What would you like to do with this issue?

where do we need scala reflect?

see #30. I think macros need it, but somehow we were not adding it in the right place and tests passed.

/cc @jcoveney

Fatal compilation error when using scala-pickles/boopickle libraries (probably macros-related)

Environment:

Ubuntu 16.04

bazel version 
Build label: 0.4.3rc6
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Dec 21 15:53:43 2016 (1482335623)
Build timestamp: 1482335623
Build timestamp as int: 1482335623

In WORKSPACE:

git_repository(
    name = "io_bazel",
    remote = "git://github.com/bazelbuild/bazel.git",
    tag = "0.4.3",
)
git_repository(
    name = "io_bazel_rules_scala",
    remote = "https://github.com/bazelbuild/rules_scala.git",
    commit = "4f3fc159d64711f2b28d18b507cfe25c3348e4d5",
)
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()

maven_jar(
    name = "me_chrons_boopickle_2_11",
    artifact = "me.chrons:boopickle_2.11:1.2.4",
)

In third_party/BUILD:

java_library(
    name = "me_chrons_boopickle_2_11",
    visibility = ["//visibility:public"],
    exports = [
        "@me_chrons_boopickle_2_11//jar",
    ],
)

SimpleCaseClass.scala:

package com.komanov.bazeltest

import boopickle.Default._

case class SimpleCaseClass(name: String)

object SimpleCaseClass {
  override def toByteArray(s: SimpleCaseClass) = {
    Pickle.intoBytes(s)
  }
}

Its BUILD file:

scala_library(
    name = "bazeltest",
    srcs = ["SimpleCaseClass.scala"],
    deps = ["//third_party:me_chrons_boopickle_2_11"],
)

When I try to build it:

bazel build --verbose_failures --worker_verbose //src/com/komanov/bazeltest
INFO: Found 1 target...
WARNING: Scalac worker (id 39) can no longer be used, because its process terminated itself or got killed.
INFO: Destroying Scalac worker (id 39)
INFO: Created new non-sandboxed Scalac worker (id 40), logging to .../.cache/bazel/_bazel_user/8284de460f2a76fffe4edd44c16c874d/bazel-workers/worker-40-Scalac.log
ERROR: .../src/com/komanov/bazeltest/BUILD:1:1: scala //src/com/komanov/bazeltest:bazeltest failed: Worker process did not return a correct WorkResponse. This is probably caused by a bug in the worker, writing unexpected other data to stdout.
Target //src/com/komanov/bazeltest:bazeltest failed to build
INFO: Elapsed time: 2.200s, Critical Path: 2.09s

worker-40-Scalac.log.txt

Looks like macros expansion related (non-macros code is working fine with this library).

create a skylark macro to ease java to scala dependencies

See:

rules_scala/test/BUILD

Line 18 in c541451

jars = [":HelloLib_deploy.jar",

it is a pain to manually add all the deps back in. A skylark macro could construct that list:

scala_export_to_java(
  name = "my_scala_lib",
  exports = [":foo", ":bar"]
)

then we construct a java_library target as done here.

related to #3 #11

Add travis_ci tests

this recipe may help:

https://github.com/korfuri/bazel-travis

add documentation on using scala code from java

see:

https://github.com/bazelbuild/rules_scala/blob/master/test/BUILD#L15

would be good to have a cleaner solution to this. I guess the java rules in bazel have to be probe skylark rules to see if they export jars/ijars.

Support different versions of scala

users should easily be able to use a particular major version of scala.

Two approaches:

1. each tag/commit only supports one and users pick.
2. have scala_2_10.bzl, scala_2_11.bzl etc... Hopefully they can share code and the external repositories work just fine.

Why does test_disappearing_class appear twice in test_run.sh

Hi,
run_test test_disappearing_class appears twice in the test_run.sh script, is this on purpose? If so why? Additionally do you think a comment might be helpful because for me it was confusing since I thought this might be a bug.

Additionally,
Is there an open issue to track converting test_run.sh to a "proper" bazel test run? I know it's not high on your priority but maybe if it's open someone will grab it.

Improve scala to java exports

See: #56 (comment)

one idea could be to improve this:

https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L490

so it is a skylark rule, not a macro, that consumes targets and looks at their runtime and compile-time jars, and just exports them (right now it is not saving much effort).

Then, that could be used to test this feature. I'll add an issue.

the fix we added for #18 is still pretty mechanical: you pass output jar names. What we really want is to just list targets and get all their exports and runtime deps for use from java.

In this way, we can test that the runtime paths are correct without conflating with scala_binary.

Make scrooge generator import jars instead of having to extract them

https://github.com/twitter/scrooge/blob/develop/scrooge-generator/src/main/scala/com/twitter/scrooge/frontend/Importer.scala#L56 makes it appear that we can import jars, but some quick tests didn't work. Still, would further simplify our code

Make scrooge parsing strictness configurable

https://github.com/bazelbuild/rules_scala/blob/master/src/scala/scripts/TwitterScroogeGenerator.scala#L163

improve remote tests

see: #83

we need to have tests that verify that the rules work when using targets in remote repos. The testing is not clear there.

I think we need to set up two test WORKSPACES:

1. depends on the rules and defines some thrift/scala targets.
2. second repo depends on the first, and consumes some thrift classes from the target in 1.

specs2 support

Hi,
Most of our tests use specs2 (a small portion uses JUnit and a handful use scalatest).
Do you think specs2 support should be part of this library?
If so do you think it should be via a flag to scala_test? via an additional rule?
Not sure I like the fact that scalatest is always on the classpath even if I don't need it but might be negligible.

I can add this support in an external repo but I think it's more natural for it to reside here.

build a rule to generate the intra-target source deps

Zinc:
https://github.com/typesafehub/zinc

could analyze sources and output the dependency graph of the files in a target. This could be helpful to output a set of targets that are smaller so the bazel's built in incrementalism is more effective.

so, you could have something like this:

scala_file_dependencies(name = "foo", target = "mylibrary")

and this could generate a new BUILD as output that has each file as a target with the correct dependencies.

This seems suboptimal for usability, but given bazel's model of compilation as a pure function from source to output artifact, this might part of a practical solution for making faster builds.

Problem building with bazel 0.3.2

From bazelbuild/bazel#1967 (comment) and continued here:

@damienmg - yesterday, @dinowernli and I tried it on his ubuntu workstation, and got

ERROR: /home/dino/Private/code/everything/ArchUtil/src/main/java/improbable/archutil/testing/BUILD:7:1: scala //ArchUtil/src/main/java/improbable/archutil/testing:testing_scala failed: linux-sandbox failed: error executing command /home/dino/.cache/bazel/_bazel_dino/8d44bf8f062af3067a03c8a57a9b5ad0/execroot/everything/_bin/linux-sandbox ... (remaining 4 argument(s) skipped).
Exception in thread "main" java.lang.RuntimeException: nope
    at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:338)
Caused by: java.io.IOException: Cannot run program "external/local_jdk/bin/javac ": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at io.bazel.rulesscala.scalac.ScalaCInvoker.compileJavaSources(ScalaCInvoker.java:293)
    at io.bazel.rulesscala.scalac.ScalaCInvoker.processRequest(ScalaCInvoker.java:237)
    at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:334)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 3 more

The BUILD file is:

package(
    default_visibility = ["//visibility:public"],
)

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_export_to_java")

scala_library(
    name = "testing_scala",
    testonly = 1,
    srcs = glob(["*.java"]),
    visibility = ["//visibility:private"],
    deps = [
        "//third_party/scalatest",
    ],
)

scala_export_to_java(
    name = "testing",
    exports = [":testing_scala"],
    runtime_deps = [],
)

There is a single java source file in the package:

package improbable.archutil.testing;

import org.scalatest.TagAnnotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Adds a scalatest "tag" to a test which he have configured at top-level to be excluded from the
 * sbt test runs.
 */
@TagAnnotation
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface ExcludeInSbt {
}

Excerpts from WORKSPACE that I think are pertinent

# Import Bazel's scala build rules.
git_repository(
    name = "io_bazel_rules_scala",
    remote = "https://github.com/petemounce/rules_scala.git",
    commit = "c2c878db5caa42c65f717b72511530cfb4f39294",
)
git_repository(
    name = "io_bazel",
    remote = "git://github.com/bazelbuild/bazel.git",
    tag = "0.3.2",
)
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories", "scala_mvn_artifact", "scala_version")
scala_repositories()

(we tried with tag=0.3.1 and bazel 0.3.1 as well)

The sha that the io_bazel_rules_scala is pointed at is to my fork; we have our own fork of rules_scala that we point at and periodically keep in sync with upstream, and that's what I'm currently trying to do at the same time as updating us to bazel 0.3.2.

don't depend on all of bazel to get the worker API

The worker API uses protobufs that are housed in the bazel repo. As a result, to build that worker process we need to pull all of bazel.

We should either copy the protobufs and build our own client, or copy the generated java code and build that.

md5sum hashes do not match when you build the same code twice on macos

Just discovered this recently since I was working on build caching, but it seems like the default scala rule here do not build jars with the same hash on two separate runs with the same inputs. This seems to break the caching code in bazel that gets a digest with md5.
This is what I ran on this repo here:

bazel build test/... 
md5 bazel-bin/test/*.jar > hash
bazel clean
bazel build test/... 
md5 bazel-bin/test/*.jar > hash2

When you diff the two files it shows that all the jars have different hashes. even the ijars.

Xins-MacBook-Pro:rules_scala xinlu$ diff hash hash2
1,3c1,3
< MD5 (bazel-bin/test/ExportOnly_deploy.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/ExportOnly_ijar.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/Exported_deploy.jar) = 4e1b1b620a002219a2edda47dc9a9cd2

---
> MD5 (bazel-bin/test/ExportOnly_deploy.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/ExportOnly_ijar.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/Exported_deploy.jar) = 31ea9c77a3bbf21d2b0ffc6def3247e4
5,6c5,6
< MD5 (bazel-bin/test/HelloLibTest_deploy.jar) = 1faf15bba18d60a4d9fee7937f3d18d6
< MD5 (bazel-bin/test/HelloLib_deploy.jar) = d72085331bdb480df424be08c0b8b5bd

---
> MD5 (bazel-bin/test/HelloLibTest_deploy.jar) = 9822c828fc3a1cbb87dbcd3800ba9e62
> MD5 (bazel-bin/test/HelloLib_deploy.jar) = 398401f2f02554b2d60aa62288cb255f
10,11c10,11
< MD5 (bazel-bin/test/MacroTest_deploy.jar) = 8c4767eead2843d79b850e56c86ea42e
< MD5 (bazel-bin/test/OtherLib_deploy.jar) = 2fc708e492140ed78ff0a489df106dcd

---
> MD5 (bazel-bin/test/MacroTest_deploy.jar) = 8024d3769157936d4777fe21aa88c25e
> MD5 (bazel-bin/test/OtherLib_deploy.jar) = be1812ca6d4eda50fdbb0bf340b088c8
13c13
< MD5 (bazel-bin/test/Runtime_deploy.jar) = fffa4662d3403c7a2daa042154e03f8f

---
> MD5 (bazel-bin/test/Runtime_deploy.jar) = efea7e4838f6bcd8582a0a5d86ca38f8
15,17c15,17
< MD5 (bazel-bin/test/ScalaBinary_deploy.jar) = 75871f3e5ef7addcbbfad04fe813b87a
< MD5 (bazel-bin/test/ScalaLibBinary_deploy.jar) = c46c0b69cdea700f37b40ceeb6b0bdd6
< MD5 (bazel-bin/test/ScalaLibResources_deploy.jar) = 4c0d57f5ad553e91f83f8b8bc866a195

---
> MD5 (bazel-bin/test/ScalaBinary_deploy.jar) = 4d9d183d54111532a7b88067d6e4572d
> MD5 (bazel-bin/test/ScalaLibBinary_deploy.jar) = 79684af04f1f3fa6b581775611b05aac
> MD5 (bazel-bin/test/ScalaLibResources_deploy.jar) = ea4157e3f8f2e59587bddbd26999bd89
19c19
< MD5 (bazel-bin/test/a_deploy.jar) = 08fe25daf5678fe3e1d43bc52c88daf5

---
> MD5 (bazel-bin/test/a_deploy.jar) = 5f071b2165bd91ff1fee97b5409cf282
21,25c21,25
< MD5 (bazel-bin/test/b_deploy.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/b_ijar.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/c_deploy.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/c_ijar.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/d_deploy.jar) = ecb50527b59a959320496d20809276bc

---
> MD5 (bazel-bin/test/b_deploy.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/b_ijar.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/c_deploy.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/c_ijar.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/d_deploy.jar) = 9bbe26c787071564c34af4cd418f7b9d
27,28c27,28
< MD5 (bazel-bin/test/jar_export_deploy.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e
< MD5 (bazel-bin/test/jar_export_ijar.jar) = e513f71cb2024bf2a5c6ac0a0208fb5e

---
> MD5 (bazel-bin/test/jar_export_deploy.jar) = 764d68cad512ea85baf75aec623c465e
> MD5 (bazel-bin/test/jar_export_ijar.jar) = 764d68cad512ea85baf75aec623c465e

---

I think this is because the jar command used by the scala.bzl rule changes the timestamp of the Manifest file even if you touch it. Example:

Xins-MacBook-Pro:rules_scala xinlu$ unzip -l bazel-bin/test/ExportOnly_deploy.jar
Archive:  bazel-bin/test/ExportOnly_deploy.jar
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  05-18-16 13:58   META-INF/
      118  05-18-16 13:58   META-INF/MANIFEST.MF
 --------                   -------
      118                   2 files

Basically this means that the touch to set the timestamp to 1980 didn't matter.
In our own internal version I changed the jarring to zip -X -q -FS since -FS keeps timestamps and now the hash is the same between two different builds that have the same inputs because the timestamps match.

Also this seems to be on mac, and not linux. The hashes seem to be okay on linux, but a lot of our devs use macs and it actually breaks their local cache.

scala_binary _deploy.jar target is not flattened

The goal of the _deploy.jar targets is that they are self-contained, that they can be executed with
java -jar X_deploy.jar
The current _deploy.jar target looks to be identical to the non-_deploy.jar target.

only one resources file gets included in the jar

We should include a test that has multiple resource files, and fix the issue (which is that https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L68 needs to have a +=).

Add support for tut documentation

Tut allows us to embed scala code in markdown files, and then run the compiler on that code so our docs don't get stale.

See: https://github.com/tpolecat/tut

An example of running at the command line can be seen here:

https://github.com/tpolecat/cbt/blob/80b6409c664d48dd5aa604b3d2ee100dad9bf4cf/build/build.scala#L26-L31

it should be pretty easy to add a tut_doc rule that has a single .md source and produces a single _tut.md output, or a rule that creates a _tut.zip of several sources (since bazel rule output has to be statically known from the build file).

/cc @tpolecat

Add apache headers to .bzl files.

This is for compliance with the Google Open-Source program.

No support for building deploy jars in scala

See bazelbuild/bazel#468

Support for Scala.js

Scala.js http://www.scala-js.org/ has been very popular in Scala community. At work we are having a very large code base in both Scala (backend) and Scala.js (frontend). I am wondering if anyone has tried to compile scala.js using bazel?

Scrooge --finagle

Is it possible to generate finagle services with current scrooge support? (like passing --finagle in sbt plugin)

copy the worker protobuf to the local repo

see: bazelbuild/bazel#1995 (comment)

it seems the protocol is stable, so rather than pulling the giant repo, we should just copy the protobuf an rebuild that part locally.

Thoughts @ianoc ?

test_run should be a sh_test

This would allow to test it with bazel test

To do so, shipping the source of the repository might be needed. See how it is done in bazel in src/test/shell/bazel/...

Cross building

So this isn't anything urgent but popped into my head and thought it would be best to start the conversation now before 2.12 is released and we might have more pressure.
2 questions:

1. How to parameterize the rule with respect to scala version? (don't think it's complicated but haven't looked into it).
2. How to cross build a target to both 2.11 and 2.12 like sbt supports. This is useful for both rolling out the upgrade in an organization and for OSS libraries.

Ensime support

I was wondering if there is any simple way of adding support for ensime? As reference, the following link references support for other build tools for it: [http://ensime.github.io/build_tools/](Ensime build tools).

Wrong URL on bazel.io

On http://bazel.io/docs/be/scala.html, it says that the page was moved to https://github.com/bazelbuild/rules_scala/blob/master/README.html, although the correct URL is https://github.com/bazelbuild/rules_scala/blob/master/README.md (replace .html with .md)