isabella232 / api-scala-sifive

Package for building Scala projects with wake

API Scala SiFive

Wake build rules for building Scala projects.

Table of Contents

• Dependency Fetching and Resolution
  • ivydependencies.json
  • Coursier
  • Fetching and offline use
• Wake API
  • Usage
  • ScalaModule
• Developer Documentation
  • Running tests locally

Dependency Fetching and Resolution

ivydependencies.json

Dependencies are specified with the file called ivydependencies.json. For example, consider the ivydependencies.json file from this repository:

ivydependencies.json

{
  "bloop": {
    "scalaVersion": "2.12.8",
    "dependencies": [
        "ch.epfl.scala::bloop-frontend:1.2.5",
        "ch.epfl.scala::bsp4s:2.0.0-M3"
      ]
  }
}

The format is a JSON object where the outer key is the name of a given Scala project. You can specify multiple projects that may have their own dependencies. A project is composed of several fields, all of which are optional (although having no fields would not be particularly useful).

scalaVersion

Specifies the Scala Version. This is used for fetching Scala itself as well as filling in the Scala version in maven dependencies (more on that later).

dependencies

Dependencies follow the same standard as Coursier.

Standard Java dependencies are of the form <groupID>:<artifactID>:<revision>. Because Scala does not maintain binary compatibility between major versions, Scala dependencies are specified with two colons after groupID: <groupID>::<artifactID>:<revision>. Such dependencies are automatically expanded to <groupID>:<artifactID>_<scala major version>:<revision>.

For example, since scalaVersion is "2.12.8" in this project, "ch.epfl.scala::bloop-frontend:1.2.5" is expanded to "ch.epfl.scala:bloop-frontend_2.12:1.2.5". This allows us to bump the Scala version without having to update every single Scala dependency.

For Scala dependencies that break compatibility between Scala minor versions, one can use three colons. For example, "my.org:::cool-project:1.0.0" would expand to "my.org:cool-project_2.12.8:1.0.0".

Coursier

API Scala SiFive uses Coursier to resolve and fetch dependencies. Of particular interest are Couriser environment variables that allow the user to control fetching and resolution. For example, by default, coursier only fetches from the local cache and maven central, but to enable additional repositories, one can set the COURSIER_REPOSITORIES environment variable. Please see linked Coursier documentation above for more information.

Note that Wake isolates jobs from the user’s environment; environment variables like PATH or the Coursier variables are not propagated to Wake jobs. To set Coursier-specific environment variables, include a .wake file in your workspace that publishes to the environment topic. For example:

publish environment =
  "COURSIER_REPOSITORIES=sonatype:releases", Nil

This will have the effect of setting the COURSIER_REPOSITORIES variable in the Wake job environment causing Coursier to use the Sonatype Releases repository instead of Maven Central.

Fetching and offline use

Dependency fetching is decoupled from dependency resolution for compilation. This means that internet access is only required when new dependencies need to be fetched. Compiling and running compiled code does not require internet access.

ivydependencies.json files found in the root of packages and the root of the workspace will be used for dependency fetching. This fetching is based soley on the contents of the ivydependencies.json files. Fetching is a top-level Wake job so any job-running invocation of wake will cause dependencies to be fetched. For example:

# Creates workspace but does not run jobs
$ wake --init .
# Runs all top-level jobs including api-scala-sifive dependency fetching
$ wake

Wake API

The Wake API is a set of Wake data structures and functions that allow Wake to build Scala projects. Similarly to how dependencies are fetched, the build rules use Coursier to resolve dependencies.

The primary data structure of Scala builds in Wake are ScalaModules (discussed in more detail later). Note that ScalaModule is a Wake tuple and using ScalaModule requires understanding tuples and their generated accessor functions.

Usage

Declaration

To enable a DRY (Don’t Repeate Yourself) build specification, Wake Scala build descriptions can be populated by a corresponding ivydependencies.json.

The simplest ScalaModule is created by simply reading the JSON file:

global def exampleScalaModule = makeScalaModuleFromJSON here "example"

makeScalaModuleFromJSON takes two String arguments. The first is the directory where ivydependencies.json is expected to be found. here is a built-in function in Wake that refers to the directory where the Wake file exists. This directory will also be used as the root directory of the created ScalaModule. The second argument is the name of the ScalaModule and must correspond to an outer key in the ivydependencies.json file.

By default, ScalaModules look for sources and resources in the same default directories as SBT: <root dir>/src/main/scala and <root dir>/src/main/java for sources, <root dir>/src/main/resources for resources.

You can always override these defaults by providing your own relative directories. For example, to use Mill’s defaults of src/ and resources/:

global def exampleScalaModule =
  makeScalaModuleFromJSON here "example"
  | setScalaModuleSourceDirs ("src", Nil)
  | setScalaModuleResourceDirs ("resources", Nil)

Or to keep the SBT defaults but add extra/sources as an additional source directory:

global def exampleScalaModule =
  makeScalaModuleFromJSON here "example"
  | editScalaModuleSourceDirs ("extra/sources", _)

Dependencies

Published dependencies are specified as previously discussed in ivydependencies.json. The JSON will be used to populate the IvyDeps of the ScalaModule. You can set these dependencies directly, but unless they are included in an ivydependencies.json file, they will not be fetched and compilation will fail.

Source dependencies are specified as a List of ScalaModules in the Deps field of the ScalaModule. For example, we can add modify our earlier example to have a dependency on someScalaModule:

global def exampleScalaModule =
  makeScalaModuleFromJSON here "example"
  | setScalaModuleDeps (someScalaModule, Nil)

Compilation

You can compile a ScalaModule using the function compileScalaModule. It will recursively compile ScalaModule dependencies.

Running

You can get the full classpath needed to run a ScalaModule with the function scalaModuleClasspath. Note that scalaModuleClasspath will compile the passed module and its dependencies.

Once you have the classpath, running a compiled ScalaModule is fairly typical Wake code. For example, assume exampleScalaModule has a main function example.Main and accepts one command-line argument:

def runExample arg =
  def classpath = scalaModuleClasspath exampleScalaModule | map getPathName | catWith ":"
  def cmd = which "java", "-cp", classpath, "example.Main", arg.getPathName, Nil
  def visible = arg, Nil
  makePlan cmd visible | runJob

You can learn more about Wake and job invocations in the Wake tutorial.

ScalaModule

ScalaModule is a Wake Tuple which comes with generated accessor functions. These accessor functions are the primary mechanism for creating ScalaModules. Please see the tuple documentation for more information.

tuple ScalaModule =
  global Name:               String
  global RootDir:            String
  global ScalaVersion:       ScalaVersion
  global IvyDeps:            List UnexpandedIvyDep
  global Deps:               List ScalaModule
  # These are relative to RootDir
  global SourceDirs:         List String
  global ResourceDirs:       List String
  # These operations could be expensive
  global FnGeneratedSources: Unit => List Path
  global ScalacOptions:      List String
  global CompilerPlugins:    List UnexpandedIvyDep
  # We propogate errors
  global Error:              Option Error

Developer Documentation

Running tests locally

You must first have wake (v0.17.2).

# from the root of the repository
./tests/run-tests.sh

Note that run-tests.sh emulates the preinstall behavior which is based on Wit <= v0.12.0 Scala plugin fetching behavior. It will fetch all ivydependencies.json files that are one directory below tests/. Putting an ivydependencies.json file in a directory deeper than that prevents it from being fetched by run-tests.sh.