Build a statically linked binary about erd

[mmzx]

@Mogztter the master just got updated with the descriptions and help from @nh2 in the form of a script.
@nh2 Thanks, static-haskell-nix is quite nice!

[mmzx]

@Mogztter Yes, should be possible. Will do the binaries over the weekend, stay tuned please.

[BurntSushi]

No, I don't know, sorry.

[ggrossetie]

Alright, thanks for your quick reply, I will try to dig a bit further 😄

[ggrossetie]

Just a quick note, it's actually working using "old" cabal commands:

# build static executable binary
FROM haskell:8

RUN git clone https://github.com/BurntSushi/erd.git

WORKDIR erd

# use "old" commands on purpose to workaround an issue with static link
RUN cabal update
RUN cabal install --only-dependencies --force-reinstalls
RUN cabal configure --disable-executable-dynamic --disable-shared --ghc-option=-optl=-static \
    && cabal build

@BurntSushi If you want I can submit a pull request with this Dockerfile ? Or add a documentation page somewhere if you think that it can be useful ? 😉

[BurntSushi]

I'd be happy to add a link, but I definitely do not want to adopt a Dockerfile in a project that is already unmaintained.

[ggrossetie]

Oh I didn't know that the project is unmaintained.
The idea is not to publish this Docker image to the Docker registry: https://hub.docker.com
It's really just about building a statically linked binary then you can copy this binary and do whatever you want.
Should I add a mention in the README ?

[nh2]

Hey, I found this because I got pinged on haskell/cabal#5847.

What you want to do is really easy with https://github.com/nh2/static-haskell-nix: Just check out the repo, do what it says in the README, and run:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build --no-link survey/default.nix -A haskellPackages.erd

This will generate a 3.8 MB static erd for you.

I have attached it here: erd-0.1.3.0-static-linux-x86_64.zip

Building it took only 2 minutes due to my binary cache. I had to make a miniscule change nh2/static-haskell-nix@3729754 to get it compile, because the erd hackage package has a declared a fixed dependency base == 4.7 which only allows older GHCs; the doJailbreak I've added relaxes that constraing. If this was fixed on Hackage, you'd be able to build this static exe without any change to my repo.

I also recommend you adding erd to Stackage, because then erd will get built statically as part of my repo automatically.

[ggrossetie]

Hello @nh2
Thanks for your reply.

As mentioned, I'm not familiar with the whole Haskell ecosystem. I finally got this working using cabal "old" commands.

What are the advantages of using nixpkgs ? Also I'm building the binary from a Docker image to make the build reproducible (and avoid Haskell build tools installation on the local machine).
In your repository you talk about building against musl instead of gcc. Does it mean that I can use a tiny Alpine image to build erd ?

The dependency should be base >= 4.7.
The Haskell policy is to provide backward compatible version of base ?

Regarding Stackage, I don't really know because @BurntSushi already mentioned that the project is unmaintained. So ultimately he will decide.

[nh2]

@Mogztter

As mentioned, I'm not familiar with the whole Haskell ecosystem.

Yep, understood.

What are the advantages of using nixpkgs ?

Reproducibility to a high degree, and getting a musl based toolchain. It can be a pain to get a full musl-based compiler toolchain available on Linux distributions that default to glibc, because you have to rebuild everything yourself. With nixpkgs you can "codify" this toolchain, and easily share the work across multiple different binaries (e.g. my repo building all executables on Stackage, vs each upstream project having to maintain their own Docker setup to provide static executables).

In your repository you talk about building against musl instead of gcc.

Small correction: musl instead of glibc. gcc is still used as the compiler. This is because glibc doesn't like being linked statically, and musl supports this use case officially.

Does it mean that I can use a tiny Alpine image to build erd ?

Yes, likely. As long as your build toolchain is built on musl, using this approach is equivalent to my nixpkgs approach.

You also need to ensure that all system libraries a given Haskell package needs provide .a files for static linking. I don't know to what extent Alpine does that already; in my repo I can do it relatively conveniently for all kinds of system libraries (example).

The dependency should be base >= 4.7.

I think this is only the case in master, because 51b0807 appears to have never made it into a Hackage release. On Hackage (and thus in my repo) it still has the old base == 4.7.*.

Regarding Stackage, I don't really know because @BurntSushi already mentioned that the project is unmaintained. So ultimately he will decide.

You don't need to be the author to add something to Stackage (like you don't need to be an author to add something to Debian).

If @BurntSushi does not need to project any more, but you have an interest in erd because you use it for your work, you could be the Stackage maintainer of it to ensure it keeps compiling well with newer GHCs and libraries. I also do that for some libraries that aren't mine (example).

I hadn't heard of erd before this ticket, but it looks pretty useful and the output looks nice, so it might be worthwhile.

Hope this answers all questions!

[BurntSushi]

If someone wanted to take over this code and manage the Stack package, that'd be great!

[nh2]

If someone wanted to take over this code and manage the Stack package, that'd be great!

@BurntSushi mind making a short, pinned "looking for new maintainer" issue?

That would make it obvious for people looking at issues. I'd then link it for you on Haskell reddit and mailing lists unless you want to do that. The package seems beginner friendly so chances are high somebody would like to take it over.

[BurntSushi]

@nh2 Sure, good idea. See #42.

[ggrossetie]

@mmzx I have seen that the project can now be built using stack. What is the recommended way to create a statically linked binary? Thanks!

[mmzx]

@Mogztter, can you maybe have a look at the link below?
I would be able to give a better answer towards the weekend, at the moment I'm in the road...
commercialhaskell/stack#3420

[nh2]

What is the recommended way to create a statically linked binary?

static-haskell-nix can do it for stack projects: https://github.com/nh2/static-haskell-nix/#building-stack-projects

[ggrossetie]

@nh2 I'm using a Docker image to build the project and I'm not familiar with nix so https://github.com/nh2/static-haskell-nix/blob/master/static-stack2nix-builder-example/default.nix is really cryptic for me 🤔

Do you provide a Docker image to build a statically linked binary with static-haskell-nix ?
Ideally I want to do something like:

FROM xyz

RUN git clone https://github.com/BurntSushi/erd.git

WORKDIR erd

RUN git checkout v0.2.0.0

RUN nixpkgs nix-build

[ggrossetie]

I also recommend you adding erd to Stackage, because then erd will get built statically as part of my repo automatically.

@nh2 Is that the case now? Can I get a pre-built statically linked binary of erd somewhere?

[ggrossetie]

For reference the following is working:

FROM ubuntu:18.04

RUN apt-get update && apt-get install -y graphviz curl git

RUN curl -sSL https://get.haskellstack.org/ | sh

RUN git clone https://github.com/BurntSushi/erd.git

WORKDIR erd

RUN git checkout v0.2.0.0

RUN /usr/local/bin/stack install --ghc-options="-fPIC" \
  --ghc-options="-static" \
  --ghc-options="-optl=-static" \
  --ghc-options="-optc=-static"

[nh2]

@Mogztter

Is that the case now?

It looks not, erd is not on https://www.stackage.org/lts-14.7. Did anybody make an effort to add it?

Do you provide a Docker image to build a statically linked binary with static-haskell-nix ?

No, Docker does not really provide any benefit over using nix directly on a host system, so it is unlikely that anybody will do that.

For reference the following is working:

Be aware that your Docker image is using Ubuntu, which means you are statically linking against glibc, which does not properly support static linking (neither technically, nor do its maintainers consider static linking a use case worth supporting). If you use any networking functionality, you must expect segfaults (e.g. nh2/static-haskell-nix#17), and may experience other odd behaviour.

To get a proper static binary without surprises, you must use a libc that supports static linking, such as musl, and the entire dependency chain (including all dependent system C libraries) must be built against it. This is what static-haskell-nix does, and the reason why stack alone cannot give you a static binary (because it has no control over your system C libraries). An alternative is to do what you do (building with stack) in Alpine Linux (or a docker container thereof), which comes with musl as its system libc.

Building erd with static-haskell-nix

I'm not familiar with nix so https://github.com/nh2/static-haskell-nix/blob/master/static-stack2nix-builder-example/default.nix is really cryptic for me

Understood. Nix has a significant learning curve (though certainly easier than Haskell), but it is usually worth it because it gives full control over all dependencies (like using a different libc), and high reproducibility.

In this case, you have it easy though:

You don't need to understand the static-stack2nix-builder in detail. I've written it such that for most projects, you can just drop it into your directory and run it. This also works for erd.

First install nix.

Download default.nix, and set the relevant variables for erd:

  cabalPackageName = "erd";
  compiler = "ghc844"; # matching stack.yaml

Then run the command $(nix-build --no-link -A fullBuildScript) as explained in the top of the the file.

It will build and print /nix/store/5i3b3n7amd0yb9fx0zdrww95ancyy9iz-erd-0.2.0.0, which contains the desired static binary in the bin directory.

(Follow the steps at Binary caches for faster building (optional) if you want to use my binary cache so you don't have wait for building GHC from scratch.)

For convenience I've attached the binary built this way, so you can try out if it works before making any effort of building it yourself: erd-0.2.0.0.tar.gz

[ggrossetie]

It looks not, erd is not on https://www.stackage.org/lts-14.7. Did anybody make an effort to add it?

Sorry I misread, it's published on hackage not stackage 😅
http://hackage.haskell.org/package/erd

No, Docker does not really provide any benefit over using nix directly on a host system, so it is unlikely that anybody will do that.

Ok but I can install Nix on an Alpine image?

To get a proper static binary without surprises, you must use a libc that supports static linking, such as musl, and the entire dependency chain (including all dependent system C libraries) must be built against it. This is what static-haskell-nix does, and the reason why stack alone cannot give you a static binary (because it has no control over your system C libraries). An alternative is to do what you do (building with stack) in Alpine Linux (or a docker container thereof), which comes with musl as its system libc.

I tried to use an Alpine based image, but stack threw an error. I don't have the stack trace anymore but basically stack was unable to install GHC.

Download default.nix, and set the relevant variables for erd:

Should we add this file to the erd repository ?

Understood. Nix has a significant learning curve (though certainly easier than Haskell), but it is usually worth it because it gives full control over all dependencies (like using a different libc), and high reproducibility.

Thanks for the explanation.
I was a bit confused with NixOS... I thought it was mandatory to use NixOS.

It will build and print /nix/store/5i3b3n7amd0yb9fx0zdrww95ancyy9iz-erd-0.2.0.0, which contains the desired static binary in the bin directory.

Is it possible to get a stable output (ie /nix/store/stable-identifier-erd-0.2.0.0)? Because I need to reference the path to copy the binary.

For convenience I've attached the binary built this way, so you can try out if it works before making any effort of building it yourself: erd-0.2.0.0.tar.gz

Thanks I will give it a try 👍

[nh2]

Ok but I can install Nix on an Alpine image?

@Mogztter That's possible but the question may not make sense. You need either nix (on any OS, without Docker) or Alpine (for example in Docker) in order to get a musl-based build.

Should we add this file to the erd repository ?

Yes.

I was a bit confused with NixOS... I thought it was mandatory to use NixOS.

Ah I see. Indeed NixOS is not necessary. Only the nix build tool, and the nixpkgs repository of software, which you can run on any Linux distro.

I tried to use an Alpine based image, but stack threw an error. I don't have the stack trace anymore but basically stack was unable to install GHC.

Yes, there have been some problems with stack on Alpine with its GHC versions recently (commercialhaskell/stack#2387). This was one of the reasons I made static-haskell-nix: To make reproducible static builds possible on any distro, without having to depend on whether a specific Linux distro's GHC setup is currently in good shape or not. (That's not to say Alpine isn't good -- I think it's great, but nixpkgs is much easier to override with a bit of declarative code than Alpine is.)

Is it possible to get a stable output

Yes: The nix build prints the path to stdout, and it's the only thing it prints to stdout. So you can do e.g.

cp $($(nix-build --no-link -A fullBuildScript))/bin/erd somewhere-else/erd

Thanks I will give it a try +1

Let me know if you encounter any issues 👍

[mmzx]

I am currently working on to put erd into stackage, but having some obstacles on the way.

[akdor1154]

FWIW the following gives me a static binary (except for glibc) from current master:

FROM docker.io/haskell:8

WORKDIR /src

RUN cabal v2-update
RUN cabal v2-configure --enable-executable-static --enable-optimization=2
RUN cabal v2-build
RUN cabal v2-install --install-method=copy --overwrite-policy=always --installdir .

Run with

buildah bud \
    --volume "$(pwd)":"/src" \
    --volume "$(pwd)/.cabal":"/root/.cabal" \
    -t erd \
    .

EDIT: or better, because the container image is actually not needed here:

mkdir -p .cabal

podman run \
    --volume "$(pwd)":"/src" \
    --volume "$(pwd)/.cabal":"/root/.cabal" \
    --workdir /src \
    --interactive \
    --rm \
    docker.io/haskell:8 \
    sh <<EOF
cabal v2-update
cabal v2-configure --enable-executable-static --enable-optimization=2
cabal v2-build
cabal v2-install --install-method=copy --overwrite-policy=always --installdir .
EOF

[ggrossetie]

@mmzx I see that https://github.com/BurntSushi/erd/releases/tag/v0.2.1.0-RC1 has binaries attached. Would it be possible to do that for the current and next releases?

[mmzx]

@Mogztter I've just created a PR to release a new version, it would be due anyway regardless static binary availability.

However, I have failed making the compiling work using static-haskell-nix. How would you - if ever that happened - otherwise create a statically linked executable?

Would you be fine using a docker container as an alternative?

[ggrossetie]

For now, we are using a Docker container to produce an executable:

https://github.com/yuzutech/kroki/blob/b7b102d074ace07ba6260378691088b88e04a032/server/ops/docker/jdk11-alpine/Dockerfile#L102-L122

Since we are using a tag, it would be better if the result was attached to a release. Currently, we are building it every time 😞

[dduportal]

We have the same requirement for the official asciidoctor Docker image which provides erd.

The recent upgrade to alpine 3.17 break the cabal install so we need to find another static binary. Would be useful to get static binaries on releases.

@mmzx I can help if you need some