This project is an implementation of the Dhall configuration language for the Java Virtual Machine.
Our goal for this project is to make it as easy as possible to integrate Dhall into JVM build systems (see the dhall-kubernetes demonstration below for a concrete example of why you might want to do this).
The core modules have no external dependencies, are Java 7-compatible, and are fairly minimal:
$ du -h modules/core/target/dhall-core-0.10.0-M1.jar
168K modules/core/target/dhall-core-0.10.0-M1.jar
$ du -h modules/parser/target/dhall-parser-0.10.0-M1.jar
108K modules/parser/target/dhall-parser-0.10.0-M1.jarThere are also several Scala modules that are published for Scala 2.12, 2.13, and 3.0. While most of the examples in this README are focused on Scala, you shouldn't need to know or care about Scala to use the core DhallJ modules.
The initial development of this project was supported in part by Permutive.
- Status
- Getting started
- Converting to other formats
- Import resolution
- Command-line interface
- Other stuff
- Developing
- Community
- Copyright and license
The current release of this project supports Dhall 21.0.0. We're running the Dhall acceptance test suites for parsing, normalization, CBOR encoding and decoding, hashing, and type inference, and currently all tests are passing (with three exceptions; see the 0.10.0-M1 release notes for details).
There are several known issues:
- The parser cannot parse deeply nested structures (records, etc., although note that indefinitely long lists are fine).
- The type checker is also not stack-safe (this should be fixed soon).
- Import resolution is not provided in the core modules, and is a work in progress.
While we think the project is reasonably well-tested, it's very new, is sure to be full of bugs, and nothing about the API should be considered stable at the moment. Please use responsibly.
The easiest way to try things out is to add the Scala wrapper module to your build. If you're using sbt that would look like this:
libraryDependencies += "org.dhallj" %% "dhall-scala" % "0.10.0-M1"This dependency includes two packages: org.dhallj.syntax and org.dhallj.ast.
The syntax package provides some extension methods, including a parseExpr
method for strings (note that this method returns an
Either[ParsingFailure, Expr], which we unwrap here with Right):
scala> import org.dhallj.syntax._
import org.dhallj.syntax._
scala> val Right(expr) = "\\(n: Natural) -> [n + 0, n + 1, 1 + 1]".parseExpr
expr: org.dhallj.core.Expr = λ(n : Natural) → [n + 0, n + 1, 1 + 1]Now that we have a Dhall expression, we can type-check it:
scala> val Right(exprType) = expr.typeCheck
exprType: org.dhallj.core.Expr = ∀(n : Natural) → List NaturalWe can "reduce" (or β-normalize) it:
scala> val normalized = expr.normalize
normalized: org.dhallj.core.Expr = λ(n : Natural) → [n, n + 1, 2]We can also α-normalize it, which replaces all named variables with indexed underscores:
scala> val alphaNormalized = normalized.alphaNormalize
alphaNormalized: org.dhallj.core.Expr = λ(_ : Natural) → [_, _ + 1, 2]We can encode it as a CBOR byte array:
scala> alphaNormalized.getEncodedBytes
res0: Array[Byte] = Array(-125, 1, 103, 78, 97, 116, 117, 114, 97, 108, -123, 4, -10, 0, -124, 3, 4, 0, -126, 15, 1, -126, 15, 2)And we can compute its semantic hash:
scala> alphaNormalized.hash
res1: String = c57cdcdae92638503f954e63c0b3ae8de00a59bc5e05b4dd24e49f42aca90054If we have the official dhall CLI installed, we can confirm that this hash is
correct:
$ dhall hash <<< '\(n: Natural) -> [n + 0, n + 1, 1 + 1]'
sha256:c57cdcdae92638503f954e63c0b3ae8de00a59bc5e05b4dd24e49f42aca90054We can also compare expressions:
scala> val Right(other) = "\\(n: Natural) -> [n, n + 1, 3]".parseExpr
other: org.dhallj.core.Expr = λ(n : Natural) → [n, n + 1, 3]
scala> normalized == other
res2: Boolean = false
scala> val Some(diff) = normalized.diff(other)
diff: (Option[org.dhallj.core.Expr], Option[org.dhallj.core.Expr]) = (Some(2),Some(3))And apply them to other expressions:
scala> val Right(arg) = "10".parseExpr
arg: org.dhallj.core.Expr = 10
scala> expr(arg)
res3: org.dhallj.core.Expr = (λ(n : Natural) → [n + 0, n + 1, 1 + 1]) 10
scala> expr(arg).normalize
res4: org.dhallj.core.Expr = [10, 11, 2]We can also resolve expressions containing imports (although at the moment dhall-scala doesn't support remote imports or caching; please see the section on import resolution below for details about how to set up remote import resolution if you need it):
val Right(enumerate) =
| "./dhall-lang/Prelude/Natural/enumerate".parseExpr.flatMap(_.resolve)
enumerate: org.dhallj.core.Expr = let enumerate : Natural → List Natural = ...
scala> enumerate(arg).normalize
res5: org.dhallj.core.Expr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]Note that we're working with values of type Expr, which comes from dhall-core,
which is a Java module. The Expr class includes static methods for creating
Expr values:
scala> import org.dhallj.core.Expr
import org.dhallj.core.Expr
scala> Expr.makeTextLiteral("foo")
res6: org.dhallj.core.Expr = "foo"
scala> Expr.makeEmptyListLiteral(Expr.Constants.BOOL)
res7: org.dhallj.core.Expr = [] : BoolIf you're working from Scala, though, you're generally better off using the
constructors included in the org.dhallj.ast package, which provide more
type-safety:
scala> TextLiteral("foo")
res8: org.dhallj.core.Expr = "foo"
scala> NonEmptyListLiteral(BoolLiteral(true), Vector())
res9: org.dhallj.core.Expr = [True]The ast package also includes extractors that let you pattern match on
Expr values:
scala> expr match {
| case Lambda(name, _, NonEmptyListLiteral(first +: _)) => (name, first)
| }
res10: (String, org.dhallj.core.Expr) = (n,n + 0)Note that we don't have exhaustivity checking for these extractors, although we might be able to add that in an eventual Dotty version.
In addition to dhall-scala, there's a (more experimental) dhall-scala-codec module, which supports encoding and decoding Scala types to and from Dhall expressions. If you add it to your build, you can write the following:
scala> import org.dhallj.codec.syntax._
import org.dhallj.codec.syntax._
scala> List(List(1, 2), Nil, List(3, -4)).asExpr
res0: org.dhallj.core.Expr = [[+1, +2], [] : List Integer, [+3, -4]]You can even decode Dhall functions into Scala functions (assuming you have the appropriate codecs for the input and output types):
val Right(f) = """
let enumerate = ./dhall-lang/Prelude/Natural/enumerate
let map = ./dhall-lang/Prelude/List/map
in \(n: Natural) ->
map Natural Integer Natural/toInteger (enumerate n)
""".parseExpr.flatMap(_.resolve)And then:
scala> val Right(scalaEnumerate) = f.as[BigInt => List[BigInt]]
scalaEnumerate: BigInt => List[BigInt] = org.dhallj.codec.Decoder$$anon$11$$Lambda$15614/0000000050B06E20@94b036
scala> scalaEnumerate(BigInt(3))
res1: List[BigInt] = List(0, 1, 2)Eventually we'll probably support generic derivation for encoding Dhall expressions to and from algebraic data types in Scala, but we haven't implemented this yet.
DhallJ currently includes several ways to export Dhall expressions to other formats. The core module includes very basic support for printing Dhall expressions as JSON:
scala> import org.dhallj.core.converters.JsonConverter
import org.dhallj.core.converters.JsonConverter
scala> import org.dhallj.parser.DhallParser.parse
import org.dhallj.parser.DhallParser.parse
scala> val expr = parse("(λ(n: Natural) → [n, n + 1, n + 2]) 100")
expr: org.dhallj.core.Expr.Parsed = (λ(n : Natural) → [n, n + 1, n + 2]) 100
scala> JsonConverter.toCompactString(expr.normalize)
res0: String = [100,101,102]This conversion supports the same subset of Dhall expressions as dhall-to-json (e.g.
it can't produce JSON representation of functions, which means the normalization in the example
above is necessary—if we hadn't normalized the conversion would fail).
There's also a module that provides integration with Circe, allowing you to convert Dhall
expressions directly to (and from) io.circe.Json values without intermediate serialization to
strings:
scala> import org.dhallj.circe.Converter
import org.dhallj.circe.Converter
scala> import io.circe.syntax._
import io.circe.syntax._
scala> Converter(expr.normalize)
res0: Option[io.circe.Json] =
Some([
100,
101,
102
])
scala> Converter(List(true, false).asJson)
res1: org.dhallj.core.Expr = [True, False]Another module supports converting to any JSON representation for which you have a Jawn facade. For example, the following build configuration would allow you to export spray-json values:
libraryDependencies ++= Seq(
"org.dhallj" %% "dhall-jawn" % "0.4.0",
"org.typelevel" %% "jawn-spray" % "1.0.0"
)And then:
scala> import org.dhallj.jawn.JawnConverter
import org.dhallj.jawn.JawnConverter
scala> import org.typelevel.jawn.support.spray.Parser
import org.typelevel.jawn.support.spray.Parser
scala> val toSpray = new JawnConverter(Parser.facade)
toSpray: org.dhallj.jawn.JawnConverter[spray.json.JsValue] = org.dhallj.jawn.JawnConverter@be3ffe1d
scala> toSpray(expr.normalize)
res0: Option[spray.json.JsValue] = Some([100,101,102])Note that unlike the dhall-circe module, the integration provided by dhall-jawn is only one way (you can convert Dhall expressions to JSON values, but not the other way around).
We also support YAML export via SnakeYAML (which doesn't require a Scala dependency):
scala> import org.dhallj.parser.DhallParser.parse
import org.dhallj.parser.DhallParser.parse
scala> import org.dhallj.yaml.YamlConverter
import org.dhallj.yaml.YamlConverter
scala> val expr = parse("{foo = [1, 2, 3], bar = [4, 5]}")
expr: org.dhallj.core.Expr.Parsed = {foo = [1, 2, 3], bar = [4, 5]}
scala> println(YamlConverter.toYamlString(expr))
foo:
- 1
- 2
- 3
bar:
- 4
- 5You can use the YAML exporter with dhall-kubernetes, for example. Instead of maintaining a lot of verbose and repetitive and error-prone YAML files, you can keep your configuration in well-typed Dhall files (like this example) and have your build system export them to YAML:
import org.dhallj.syntax._, org.dhallj.yaml.YamlConverter
val kubernetesExamplePath = "../dhall-kubernetes/1.17/examples/deploymentSimple.dhall"
val Right(kubernetesExample) = kubernetesExamplePath.parseExpr.flatMap(_.resolve)And then:
scala> println(YamlConverter.toYamlString(kubernetesExample.normalize))
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
name: nginx
template:
metadata:
name: nginx
spec:
containers:
- image: nginx:1.15.3
name: nginx
ports:
- containerPort: 80It's not currently possible to convert to YAML without the SnakeYAML dependency, although we may support a simplified version of this in the future (something similar to what we have for JSON in the core module).
There are currently two modules that implement import resolution (to different degrees).
The first is dhall-imports, which is a Scala library built on cats-effect that uses http4s for its HTTP client. This module is intended to be a complete implementation of the import resolution and caching specification.
It requires a bit of ceremony to set up:
import cats.effect.{IO, Resource}
import org.dhallj.core.Expr
import org.dhallj.imports.syntax._
import org.dhallj.parser.DhallParser
import org.http4s.blaze.client.BlazeClientBuilder
import org.http4s.client.Client
import scala.concurrent.ExecutionContext
val client: Resource[IO, Client[IO]] = BlazeClientBuilder[IO](ExecutionContext.global).resourceAnd then if we have some definitions like this:
val concatSepImport = DhallParser.parse("https://prelude.dhall-lang.org/Text/concatSep")
val parts = DhallParser.parse("""["foo", "bar", "baz"]""")
val delimiter = Expr.makeTextLiteral("-")We can use them with a function from the Dhall Prelude like this:
scala> val resolved = client.use { implicit c =>
| concatSepImport.resolveImports[IO]
| }
resolved: cats.effect.IO[org.dhallj.core.Expr] = IO(...)
scala> import cats.effect.unsafe.implicits.global
import cats.effect.unsafe.implicits.global
scala> val result = resolved.map { concatSep =>
| Expr.makeApplication(concatSep, Array(delimiter, parts)).normalize
| }
result: cats.effect.IO[org.dhallj.core.Expr] = IO(...)
scala> result.unsafeRunSync()
res0: org.dhallj.core.Expr = "foo-bar-baz"(Note that we could use dhall-scala to avoid the use of Array above.)
We support an extension of the spec which allows you to also import expressions
from the classpath using the syntax let e = classpath:/absolute/path/to/file in e.
The semantics are subject to change as we get more experience with it but
currently it should generally have the same behaviour as an absolute
path import of a local file (but files on the classpath can import each other
using relative paths). This includes it being protected by the referential
sanity check so that remote imports cannot exfiltrate information
from the classpath.
Also note that classpath imports as location are currently not supported as the spec
requires that an import as Location must return an expression of type
<Local Text | Remote Text | Environment Text | Missing>.
The other implementation is dhall-imports-mini, which is a Java library that depends only on the core and parser modules, but that doesn't support remote imports or caching.
The previous example could be rewritten as follows using dhall-imports-mini and a local copy of the Prelude:
import org.dhallj.core.Expr
import org.dhallj.imports.mini.Resolver
import org.dhallj.parser.DhallParser
val concatSep = Resolver.resolve(DhallParser.parse("./dhall-lang/Prelude/Text/concatSep"), false)
val parts = DhallParser.parse("""["foo", "bar", "baz"]""")
val delimiter = Expr.makeTextLiteral("-")And then:
scala> Expr.makeApplication(concatSep, Array(delimiter, parts)).normalize
res0: org.dhallj.core.Expr = "foo-bar-baz"It's likely that eventually we'll provide a complete pure-Java implementation of import resolution, but this isn't currently a high priority for us.
We include a command-line interface that supports some common operations. It's currently similar to
the official dhall and dhall-to-json binaries, but with many fewer options.
If GraalVM Native Image is available on your system, you can build the CLI as a native binary (thanks to sbt-native-packager).
$ sbt cli/graalvm-native-image:packageBin
$ cd cli/target/graalvm-native-image/
$ du -h dhall-cli
8.2M dhall-cli
$ time ./dhall-cli hash --normalize --alpha <<< "λ(n: Natural) → [n, n + 1]"
sha256:a8d9326812aaabeed29412e7b780dc733b1e633c5556c9ea588e8212d9dc48f3
real 0m0.009s
user 0m0.000s
sys 0m0.009s
$ time ./dhall-cli type <<< "{foo = [1, 2, 3]}"
{foo : List Natural}
real 0m0.003s
user 0m0.000s
sys 0m0.003s
$ time ./dhall-cli json <<< "{foo = [1, 2, 3]}"
{"foo":[1,2,3]}
real 0m0.005s
user 0m0.004s
sys 0m0.001sEven on the JVM it's close to usable, although you can definitely feel the slow startup:
$ cd ..
$ time java -jar ./cli-assembly-0.4.0-SNAPSHOT.jar hash --normalize --alpha <<< "λ(n: Natural) → [n, n + 1]"
sha256:a8d9326812aaabeed29412e7b780dc733b1e633c5556c9ea588e8212d9dc48f3
real 0m0.104s
user 0m0.106s
sys 0m0.018sThere's probably not really any reason you'd want to use dhall-cli right now, but I think it's a
pretty neat demonstration of how Graal can make Java (or Scala) a viable language for building
native CLI applications.
The dhall-testing module provides support for property-based testing with ScalaCheck
in the form of Arbitrary (and Shrink) instances:
scala> import org.dhallj.core.Expr
import org.dhallj.core.Expr
scala> import org.dhallj.testing.instances._
import org.dhallj.testing.instances._
scala> import org.scalacheck.Arbitrary
import org.scalacheck.Arbitrary
scala> Arbitrary.arbitrary[Expr].sample
res0: Option[org.dhallj.core.Expr] = Some(Optional (Optional (List Double)))
scala> Arbitrary.arbitrary[Expr].sample
res1: Option[org.dhallj.core.Expr] = Some(Optional (List <neftfEahtuSq : Double | kg...It includes (fairly basic) support for producing both well-typed and probably-not-well-typed expressions, and for generating arbitrary values of specified Dhall types:
scala> import org.dhallj.testing.WellTypedExpr
import org.dhallj.testing.WellTypedExpr
scala> Arbitrary.arbitrary[WellTypedExpr].sample
res2: Option[org.dhallj.testing.WellTypedExpr] = Some(WellTypedExpr(8436008296256993755))
scala> genForType(Expr.Constants.BOOL).flatMap(_.sample)
res3: Option[org.dhallj.core.Expr] = Some(True)
scala> genForType(Expr.Constants.BOOL).flatMap(_.sample)
res4: Option[org.dhallj.core.Expr] = Some(False)
scala> genForType(Expr.makeApplication(Expr.Constants.LIST, Expr.Constants.INTEGER)).flatMap(_.sample)
res5: Option[org.dhallj.core.Expr] = Some([+1522471910085416508, -9223372036854775809, ...This module is currently fairly minimal, and is likely to change substantially in future releases.
The dhall-javagen module lets you take a DhallJ representation of a Dhall expression and use it to generate Java code that will build the DhallJ representation of that expression.
This is mostly a toy, but it allows us for example to distribute a "pre-compiled" jar containing the Dhall Prelude:
scala> import java.math.BigInteger
import java.math.BigInteger
scala> import org.dhallj.core.Expr
import org.dhallj.core.Expr
scala> val ten = Expr.makeNaturalLiteral(new BigInteger("10"))
ten: org.dhallj.core.Expr = 10
scala> val Prelude = org.dhallj.prelude.Prelude.instance
Prelude: org.dhallj.core.Expr = ...
scala> val Natural = Expr.makeFieldAccess(Prelude, "Natural")
Natural: org.dhallj.core.Expr = ...
scala> val enumerate = Expr.makeFieldAccess(Natural, "enumerate")
enumerate: org.dhallj.core.Expr = ...
scala> Expr.makeApplication(enumerate, ten).normalize
res0: org.dhallj.core.Expr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]Note that the resulting jar (which is available from Maven Central as dhall-prelude) is many times smaller than either the Prelude source or the Prelude serialized as CBOR.
The project includes the currently-supported version of the Dhall language repository as a submodule, so if you want to run the acceptance test suites, you'll need to clone recursively:
git clone --recurse-submodules [email protected]:travisbrown/dhallj.gitOr if you're like me and always forget to do this, you can initialize the submodule after cloning:
git submodule update --initThis project is built with sbt, and you'll need to have sbt installed on your machine.
We're using the JavaCC parser generator for the parsing module, and we have our own sbt plugin for integrating JavaCC into our build. This plugin is open source and published to Maven Central, so you don't need to do anything to get it, but you will need to run it manually the first time you build the project (or any time you update the JavaCC grammar):
sbt:root> javacc
Java Compiler Compiler Version 7.0.5 (Parser Generator)
File "Provider.java" does not exist. Will create one.
File "StringProvider.java" does not exist. Will create one.
File "StreamProvider.java" does not exist. Will create one.
File "TokenMgrException.java" does not exist. Will create one.
File "ParseException.java" does not exist. Will create one.
File "Token.java" does not exist. Will create one.
File "SimpleCharStream.java" does not exist. Will create one.
Parser generated with 0 errors and 1 warnings.
[success] Total time: 0 s, completed 12-Apr-2020 08:48:53
After this is done, you can run the tests:
sbt:root> test
...
[info] Passed: Total 1319, Failed 0, Errors 0, Passed 1314, Skipped 5
[success] Total time: 36 s, completed 12-Apr-2020 08:51:07
Note that a few tests require the dhall-haskell dhall CLI. If you don't have it installed on
your machine, these tests will be skipped.
There are also a few additional slow tests that must be run manually:
sbt:root> slow:test
...
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[success] Total time: 79 s (01:19), completed 12-Apr-2020 08:52:41
This project supports the Scala code of conduct and wants all of its channels (Gitter, GitHub, etc.) to be inclusive environments.
All code in this repository is available under the 3-Clause BSD License.
Copyright Travis Brown and Tim Spence, 2020.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent