s-zeng / dhall-python

Up-to-date and maintained python bindings for dhall, a functional configuration language

Dhall is a programmable configuration language optimized for maintainability.

You can think of Dhall as: JSON + functions + types + imports

Note that while Dhall is programmable, Dhall is not Turing-complete. Many of Dhall's features take advantage of this restriction to provide stronger safety guarantees and more powerful tooling.

You can try the language live in your browser by visiting the official website:

• https://dhall-lang.org

dhall-python

dhall-python contains Dhall bindings for Python using the rust implementation. It is meant to be used to integrate Dhall into your python applications.

If you only want to convert Dhall to/from JSON or YAML, you should use the official tooling instead; instructions can be found here.

Usage

Install using pip:

pip install dhall

Supports the following:

• Operating Systems
  • Windows
  • Mac OS
  • Linux (manylinux_2_24_x86_64)
• Python versions
  • 3.7
  • 3.8
  • 3.9
  • 3.10
  • 3.11

Python 3.5 and 3.6 support is available in older versions of dhall-python.

dhall-python implements a similar API to Python's json module:

>>> import dhall
>>> dhall.dumps({"keyA": 81, "keyB": True, "keyC": "value"})
'{ keyA = 81, keyB = True, keyC = "value" }'
>>> dhall.loads("""{ keyA = 81, keyB = True, keyC = "value" }""")
{'keyA': 81, 'keyB': True, 'keyC': 'value'}

License

dhall-python is licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in python-dhall by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

All contributions are welcome! If you spot any bugs, or have any requests, issues and PRs are always welcome.

Developer guide

This project uses poetry for managing the development environment. If you don't have it installed, run

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
export PATH="$HOME/.poetry/bin:$PATH"

The project requires the latest stable version of Rust.

Install it via rustup:

rustup install stable

If you have already installed the stable version, make sure it is up-to-date:

rustup update stable

After that, you can compile the current version of dhall-python and execute all tests and benchmarks with the following commands:

make install
make test

🤫 Pssst!... run make help to learn more.

dhall-python's Issues

Publish docs on readthedocs

Get some documentation up on readthedocs

home-anchored imports don't work

Not sure if this is an upstream dhall-rust issue, but according to the dhall documentation, the dhall language supports home-anchored imports as shown in the following example. The Python version however seems to support only the relative and absolute imports, but not the home-anchored import.

❯ dhall <<< '~/test.dhall'
"hello"

❯ python
>>> import dhall
>>> dhall.loads('./test.dhall')
'hello'
>>> dhall.loads('~/test.dhall')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Error(Dhall(Error { kind: Typecheck(TypeError { message: Custom("error: error\n --> <current file>:1:1\n  |\n1 | ~/test.dhall\n  | ^^^^^^^^^^^^ No such file or directory (os error 2)\n  |") }) }))

Serializing a loop/cycle in the structures crashes the interpreter

import dhall
el = {}
el["a"] = el
dhall.dumps(el)

dhall.dump(s) is able to generate mal-typed Dhall

Description

When passing mix-typed lists to dhall.dump(s), the resulting dhall that is outputted is not well-typed and cannot be read back with dhall.loads

Expected behaviour

python-dhall should error when receiving mal-typed inputs. For all inputs x, dhall.loads(dhall.dumps(x)) should either error on dumps, or be identitcal to the identity function on x, or else have some other way to translate heterogeneous lists and other dynamic data

Windows doesn't have versions besides 3.9

This is a publishing issue. Let me know if you are affected by this and I can schedule some time to fix it. I'm considering this low priority otherwise

Timeline for next release?

Thank you for this package. I would like to integrate it into another project; do you have a timeline for the next stable release?

Add Python 3.11 as a supported version (and to Actions)

Summary:

It looks like something previously blocked supporting 3.11, but when I downloaded and compiled this package today against 3.11 it built without any errors.

The build was on x86 MacOS, cpython 3.11.3, rustc 1.69.0 in case there are any troubles reproducing.

I... don't have an in-depth understanding of how pypi works, but I'm assuming that including 3.11 support means doing something with GitHub actions to build the wheel and publish it, and that that isn't something I can submit a PR for. If it is, apologies, and please point me in the direction of doing the work!

[Installation Bug] Pip installation of the `dhall` package fails on M1 Macbooks

Attempting to run pip install dhall on an M1 macbook fails:

~ ⌚ 21:27:32
$ source dhall-bug-venv/bin/activate
(dhall-bug-venv)
~ ⌚ 21:27:35
$ pip --version
pip 19.2.3 from /Users/gizmo385/dhall-bug-venv/lib/python3.8/site-packages/pip (python 3.8)
(dhall-bug-venv)
~ ⌚ 21:27:39
$ uname
Darwin
(dhall-bug-venv)
~ ⌚ 21:27:43
$ pip install dhall
Collecting dhall
  ERROR: Could not find a version that satisfies the requirement dhall (from versions: none)
ERROR: No matching distribution found for dhall
WARNING: You are using pip version 19.2.3, however version 21.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The only way to get it installed is to point at the git coordinate and have it manually rebuild the wheel. My theory on the cause is that there isn't a compatible wheel being pushed to PyPI for M1 Macbooks?

[Installation Bug] Pip installation of the dhall package fails on M2 Macbooks

python version: 3.10.2
pip version: 23.1.2
macOS: 13.3.1
chip: Apple M2 Pro

When I run pip install dhall, I get the following error:

  Using cached dhall-0.1.9.tar.gz (25.8 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [5 lines of output]
      💥 maturin failed
        Caused by: The following metadata fields in `package.metadata.maturin` section of Cargo.toml are removed since maturin 0.14.0: classifier, please set them in pyproject.toml as PEP 621 specifies.
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/private/var/folders/tv/xyv2127n0sn6gxfdk24rczd00000gp/T/pip-modern-metadata-dkx0j1hl', '--interpreter', '/Users/dkJeBaRa/.asdf/installs/python/3.10.2/bin/python3.10']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /private/var/folders/tv/xyv2127n0sn6gxfdk24rczd00000gp/T/pip-modern-metadata-dkx0j1hl --interpreter /Users/dkJeBaRa/.asdf/installs/python/3.10.2/bin/python3.10`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.```

From what I can see it seems like you have fixed the metadata fields issue in later versions, but 0.1.9 seems to be the only package version available to me. Any plans to release a version that is compatible with my system?

Dictionaries with numbers as keys get dumped into invalid dhall objects

>>> dhall.dumps({3: 4})
'{ 3 = 4 }'

Having numbers as object keys is invalid syntax in dhall. We should either fix the dump by quoting keys in backticks, or by dumping to a List {mapKey: Natural, mapValue: Natural}, or by erroring

Sharing by reference is not implemented

import dhall
a = {"c": 2}
el = {"a": a, "b": a}
dhall.dumps(el)

emits

{ a = { c = 2 }, b = { c = 2 } }

instead of

let o1 = {c = 2}
let res = {a = o1, b = o1}
in res

dump(s) and load(s) should support appropriate kwargs flags

As of right now, the kwargs flags for dump(s) and load(s) are still the ones inherited from hyperjson, and as of right now most of those don't do anything. We should remove the ones that don't do anything, and add flags for current and future features like type erasures, or omitting nulls, etc.

Cross build for ARM on MacOS

We would love to use this on the new ARM macbooks, but it is currently unavailable in pypi for that platform. Would it be easy to add it?

Add Cargo.lock to repo

I'm trying to create a nix-build definition for this project, which in turn would let us build the ARM version for the new macbooks, but the recommended way is to use an existing lock-file. Since there is none in this repo, I had to manually create one.
According to the nix docs, it's also best practice for rust projects to keep this in version control. I'm sure there's a reason it's ignored here, but if not, I would love for it to be included in the repo :)

Type stubs

dhall-python should have type stubs for type checking w/ mypy or pyright or similar tools

loading .dhall files with relative imports from outside their directory

environment

• dhall version = 1.39.0
• windows 11
• python version = 3.9
  • dhall==0.1.12

context

I have the following config folder structure --

/configs/main.dhall
/configs/db/influx.dhall
/configs/test.py

with main.dhall as follows

{
    db = 
    {
        influx = ./db/influx.dhall
    }
}

and test.py as follows

import dhall
from pathlib import Path

with open(Path(__file__).parent / 'main.dhall', 'r', encoding='utf-8') as fn:
    print(dhall.load(fn))

I want run this script from outside the /configs directory -- e.g. python ./configs/test.py

expected

prints out the dict object to stdout

(NOTE: using dhall-to-yaml.exe --file ./configs/test.py works correctly)

actual

The following error is thrown

Traceback (most recent call last):
  File "...\influxdb\py-api\configs\test.py", line 5, in <module>
    print(dhall.load(fn))
TypeError: Error(Dhall(Error { kind: Typecheck(TypeError { message: Custom("error: error\n --> <current file>:1:18\n  |\n...\n4 |         influx = ./db/influx.dhall\n  |             
     ^^^^^^^^^^^^^^^^^ 系统找不到指定的路径。 (os error 3)\n  |") }) }))

Support dumping empty lists with type annotation

Dumping an empty list currently gives a TypeError. We should support a mechanism to allow empty lists to be tagged with a type.

Example:
Currently, the following fails with a TypeError:

>>> dhall.dumps([[3, 4], []])
TypeError: cannot serialize value without a type annotation: List([])

... but the following works just fine:

>>> dhall.dumps([[3, 4], [5]])
'[[3, 4], [5]]'

To be consistent, we could support type inference in homogeneous lists to add a type annotation to the empty list

Build failures

Between CI builds #99 and #100 (both of which are routinely scheduled weekly builds with no code changes in between), the build ceased to complete successfully. I did some light maintenance PRs to see if I could resolve them, but it appears that I will have to take more time to investigate.

I am currently a bit strapped for free time to look at this; in the meantime, I can confirm that building from source works properly, and that the previously built artifacts and PyPI binaries still work as expected. So this isn't something terribly urgent, for the meantime, at least until serde_dhall/dhall-lang gets more updates.

When that happens, or if any users have any urgent needs, this will be addressed more urgently. Until then, thank you for the patience

How to distinguish between integers and natural numbers?

Is there a way to distinguish between integers and natural numbers when dumping? E.g. if I dump 4, it shows up as '4' instead of '+4' and that does not work if a reader using the Haskell binding expects an integer ('+4') instead of a natural number ('4').

Furthermore, if I dump [4, -4], I'll get '[4, -4]', which I think is not valid Dhall because 4 is a natural number but -4 is an integer and elements of a list needs to be the same type.

$ echo '[+4, -4]' | dhall
[ +4, -4 ]
$ echo '[4, -4]' | dhall

Use "dhall --explain" for detailed errors

Error: List elements should all have the same type

- Natural
+ Integer

1│     -4

(input):1:5

This is probably related to #1.

AUR Package

It would be nice to be able to install this from the AUR. Perhaps we can have issues as well for packaging else where, but the AUR seems like the easiest place to start for me personally.

Python representation of Optional and Unions

We should have types/objects in Python that we can use to serialize into dhall unions. Optionally, we can have a flag that lets dhall deserialize into such types.

i.e. desired ux should be something like:

>>> import dhall
>>> dhall.dumps([dhall.Some(5), None])
'[Some 5, None Natural]'

Loading Dhall objects results in nondeterministic key order

Is this an upstream dhall-rust issue?

``
➜ ~ cat test.dhall
{ a = "a", b = "b" }

In [2]: dhall.load(open('test.dhall'))
Out[2]: {'b': 'b', 'a': 'a'}

In [5]: dhall.load(open('test.dhall'))
Out[5]: {'a': 'a', 'b': 'b'}

How to assert the type of loaded value?

In Haskell bindings, I can use some typeclasses to load a Dhall value of an expected type (official example). I would like to do the same in Python:

@dataclass
class Example:
    foo: int
    bar: list[double]

x = dhall.load(open('./config'), type_=Example)

The intended result:

• success if ./config defines a record with foo and bar
• failure if ./config doesn't exist or isn't valid Dhall
• failure if ./config defines any other Dhall value (number, boolean, different fields, etc.)

How do I do this with dhall-python? If it's not currently possible, any pointers on implementation?

Dhall scripts using value sharing are not properly deserialized

res = dhall.loads("""
let o1 = {c = 2}
let res = {a = o1, b = o1}
in res
""")
assert res["a"] is res["b"] // AssertionError