kaushalmodi / hello_musl

NimScript (config.nims) for building a static binary using Nim + musl + pcre + libressl/openssl

Nim + musl + pcre + libressl/openssl

This repo contains a generic ~config.nims~ that adds a Nim “sub-command” or task named musl. You can simply add that file to your Nim project, and run nim musl foo.nim with optional -d:pcre and/or -d:libressl / -d:openssl switches (assuming that the below prerequisites are met).

Prerequisites

• OS: One of Linux x86 (32/64), ARM (32/64), MIPS (32/64), PowerPC (32/64), S390X, SuperH, Microblaze, OpenRISC (ref)
• Nim: https://nim-lang.org/ (built from *devel* branch as of <2018-09-13 Thu>)
• musl library: https://www.musl-libc.org/download.html
• curl and GNU tar are needed on the system for the -d:pcre / -d:libressl / -d:openssl switches to work.

Optional

These optional command-line utilities for binary size optimization will be run automatically one by one, if present.

• strip (from binutils): https://sourceware.org/binutils/docs/binutils/strip.html
• upx: https://github.com/upx/upx

Generating static binary

For this example hello_musl project

1. git clone https://github.com/kaushalmodi/hello_musl
2. cd hello_musl
3. Run nim musl [-d:pcre] [-d:libressl|-d:openssl] <FILE>.nim

Static links with just musl lib

nim musl src/hello_musl.nim

That will generate hello_musl binary in src/ directory.

• file ./src/hello_musl will print:

  src/hello_musl: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
      

• ./src/hello_musl will print:

  Hello, World!
      

Static linking with musl + pcre libs

nim musl -d:pcre src/hello_musl_pcre.nim

That will generate hello_musl_pcre binary in src/ directory.

• file ./src/hello_musl_pcre will print:

  src/hello_musl_pcre: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
      

• ./src/hello_musl_pcre will print:

  Hello, World!
  Bye, World!
      

Static linking with musl + libressl libs

nim musl -d:libressl src/hello_musl_ssl.nim

That will generate hello_musl_ssl binary in src/ directory.

• file ./src/hello_musl_ssl will print:

  src/hello_musl_ssl: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
      

• ./src/hello_musl_ssl will print:

  {"name":"Kaushal Modi","type":"card","url":"https://scripter.co/"}
  {
    "name": "Kaushal Modi",
    "type": "card",
    "url": "https://scripter.co/"
  }
  "Kaushal Modi"
      

Note

Binary built using -d:libressl on RHEL 6.8 is known to crash with just a “Killed” message. But the same, when built on Travis, run fine on RHEL 6.8 (mystery). So a RHEL 6.8 user may look at the slightly less secure option of building using with -d:openssl switch instead.

Static linking with musl + openssl libs

nim musl -d:openssl src/hello_musl_ssl.nim

That will generate hello_musl_ssl binary in src/ directory, and the outputs would be same as those you see for the -d:libressl switch.

Note about compiling with statically linked OpenSSL (-d:openssl)

When building with -d:openssl, a statically linked version of OpenSSL library is first built with the -DOPENSSL_NO_SECURE_MEMORY Configure option, because of an issue with it getting built using MUSL.

This security laxing switch is not added if LibreSSL is used instead (~-d:libressl~)

For your Nim project

• Copy the ~config.nims~ to your Nim project.
• While being the same directory as config.nims, do:

  nim musl <path/to/your/nim/file>             # without static pcre lib linking
  nim musl -d:pcre <path/to/your/nim/file>     # *with* static pcre lib linking
  nim musl -d:libressl <path/to/your/nim/file> # *with* static libressl lib linking
  nim musl -d:openssl <path/to/your/nim/file>  # *with* static openssl lib linking (less secure)
      

References

• https://github.com/nim-lang/Nim/wiki/Using-nimscript-for-configuration
• https://nim-lang.org/docs/nims.html
  • https://nim-lang.org/docs/nimscript.html
• https://hookrace.net/blog/nim-binary-size/#using-the-c-standard-library
• https://www.reddit.com/r/programming/comments/2wk7q6/static_linking_with_nim/corwtl7/

Todo-List

• [ ] Figure out how to have nimble install install the binary generated by nim musl.
• [X] Don’t hard-code the muslGcc const in config.nims
• [X] Not require hello_musl.nimble
  • Currently that is needed just to add the -d:musl and -d:release switches, and then to auto-run strip -s.
  • So to remove dependency on this file, I need to figure out how to get the current foo.nim file name from within the config.nims.
  • Eventual goal is to reuse the same config.nims for all projects. So I cannot hardcode the pkgName as I do in hello_musl.nimble.

hello_musl's Issues

feature request: add libcurl

thanks for the project, it's a great reference. it would be nice to have a target for a static curl build.

Figure out static linking of openssl

Ref: https://github.com/emk/rust-musl-builder/blob/14ff0ce4bf84857e83a026eac335c4299ee8518b/Dockerfile#L72-L79

Figure out how to create a static binary along with musl and pcre

Ref: https://gitter.im/nim-lang/Nim?at=5bbe4163ae7be9401633eb18

/cc @xmonader: I will update here about this if I learn something.

discussion

• after nim-lang/Nim#10324 is merged, simply the travis file, eg:

If the git repo contains a config.nims, temporarily move it, else it fails the
      # building of koch.

• you need set -e .travis.yml as done here https://github.com/nim-lang/Nim/blob/fbd6743ea1db89eef3dd89ceb7c0cdcb648c8055/.travis.yml#L35 so it fails 1st error ; it'll make the script more robust, and probably simpler

• instead of a giant (and complex) https://github.com/kaushalmodi/hello_musl/blob/master/config.nims why not instead have a bootstrapping nim binary where u can enjoy full benefits of nim (instead of nims limitations), analog to koch.nim eg:

travis:
calls `sh build_all.sh` to build nim
then calls: `nim c -r musl.nim example.nim` where `musl.nim` contains the samle logic as in `config.nims` but as a nim file

nim musl foo.nim would become: nim c -r musl.nim foo.nim (with proper caching, u'd only pay (minimal IMO) cost of building musl.nim the 1st time; alternatively u can turn your repo into a proper nimble project that contains a binary musl that gets built; u can even keep the musl task but it'd forward to : exec "musl ..."

• likewise, instead of a giant travis.yml (which would be a pain to port to appveyor), why not move bulk of it to a bootstraping nim file (maybe same as above) that takes care of everythign once there is an initial nim built; ie, same spirit as what i did here: nim-lang/Nim#10183 which greatly simplified my life after that PR

• is this actually needed (ie makes a practical difference)? if so, could we benefit from it in Nim's travis/appveyor too?

     # After building nim, wipe csources to save on cache space.
      rm -rf csources

• ditto with:
  git clone --single-branch --branch devel --depth=1 "${NIMREPO}" "${NIMDIR}" instead of what we have in Nim's travis (simply git clone)

• replace a bunch of lines with: sh build_all.sh eg:

rm -rf nim
      mkdir -p nim
      git clone --single-branch --branch devel --depth=1 "${NIMREPO}" "${NIMDIR}"
      cd "${NIMDIR}" || exit
      [ -d csources ] || git clone --depth 1 https://github.com/nim-lang/csources.git
      cd csources
      sh build.sh
      cd ..
      ./bin/nim c koch
      ./koch boot -d:release
      # After building nim, wipe csources to save on cache space.
      rm -rf csources

if that doesn't work for u, maybe we should make it work for u by improving sh build_all.sh (eg making optional some of the stuff u may not need like koch tools)
=> that way, things r more DRY

• regarding ur TODO:
  Figure out how to have nimble install install the binary generated by nim musl
  happy to help if u tell me where you're blocking

to use regexes the option --enable-unicode-properties must be provided

Hi, thanks for this, it was incredibly helpful to me, I found out I was getting some errors regarding regular expressions, turns out
for them to work one must pass the --enable-unicode-properties flag to the pcre config command, this is more a pcre thing anyway, but I think talking about it would be cool, anyhow if someone has the same issue I had this issue could be a good place to look.

Thanks once again!

Look into crosstool-NG integration

Ref: https://gitter.im/nim-lang/Nim?at=5bc0c190435c2a518e961a75

https://crosstool-ng.github.io/

leorize here:

actually, if you integrate crosstool-NG, you might be able to write a .nims that can spit out windows, mac,... binaries