coord-e / magicpak

:hammer: Build minimal docker images without static linking

magicpak

magicpak enables you to build minimal docker images without any bothersome preparation such as static linking.

# You prepare /bin/your_executable here...

ADD https://github.com/coord-e/magicpak/releases/download/v1.4.0/magicpak-x86_64-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak

RUN /usr/bin/magicpak -v /bin/your_executable /bundle

FROM scratch
COPY --from=0 /bundle /.

CMD ["/bin/your_executable"]

That's it! The resulting image shall only contain what your executable requires at runtime. You can find more useful examples of magicpak under example/.

Feature

magicpak is a command-line utility that analyzes and bundles runtime dependencies of the executable. magicpak basically collects all shared object dependencies that are required by a dynamic linker at runtime. Additionally, magicpak's contributions are summarized as follows:

• Simple. You can build a minimal image just by adding a few lines to your Dockerfile.
• Full-featured. You can bundle, test, and compress your executable at once. You can focus on your business because magicpak handles all Dockerfile-specific matters to decrease image size.
• Dynamic analysis. --dynamic flag enables a dynamic analysis that can discover dependencies other than dynamically linked libraries.
• Flexible. We expose a full control of resulting bundle with a family of options like --include and --exclude. You can deal with dependencies that cannot be detected automatically.
• Stable. We don't parse undocumented and sometimes inaccurate ldd(1) outputs. Instead, we use dlopen(3) and dlinfo(3) in glibc to query shared library locations to ld.so(8).

magicpak is especially useful when you find it difficult to produce a statically linked executable. Also, magicpak is powerful when building from source is bothering or the source code is not public, because magicpak only requires the executable to build a minimal docker image.

Usage

You can start with magicpak path/to/executable path/to/output. This simply analyzes runtime dependencies of your executable statically and put everything your executable needs in runtime to the specified output directory. Once they've bundled, we can simply copy them to the scratch image in the second stage as follows.

RUN magicpak path/to/executable /bundle

FROM scratch
COPY --from=0 /bundle /.

Some executables work well in this way. However, others fail to run properly because magicpak's static analysis isn't enough to detect all files needed by them at runtime. For this case, magicpak has --include <GLOB> option to specify the missing requirements manually. Moreover, you can use --dynamic to automatically include files that are accessed by the executable during execution.

Despite our careful implementation, our analysis is unreliable in a way because we can't completely determine the runtime behavior before its execution. To ensure that magicpak collected all dependencies to perform a specific task, --test option is implemented. --test enables testing of the resulting bundle using chroot(2).

The size of the resulting image is our main concern. magicpak supports executable compression using upx. You can enable it with --compress.

Supported options

Usage: magicpak [OPTIONS] <INPUT>... <OUTPUT>

Arguments:
  <INPUT>...  Input executable
  <OUTPUT>    Output destination

Options:
  -i, --include <GLOB>                Additionally include files/directories with glob patterns
  -e, --exclude <GLOB>                Exclude files/directories from the resulting bundle with glob patterns
      --mkdir <PATH>                  Make directories in the resulting bundle
  -r, --install-to <PATH>             Specify the installation path of the executable in the bundle
      --log-level <LEVEL>             Specify the log level [default: Warn] [possible values: Off, Error, Warn, Info, Debug]
  -v, --verbose                       Verbose mode, same as --log-level Info
  -t, --test                          Enable testing
      --test-command <COMMAND>        Specify the test command to use in --test
      --test-stdin <CONTENT>          Specify stdin content supplied to the test command in --test
      --test-stdout <CONTENT>         Test stdout of the test command
  -d, --dynamic                       Enable dynamic analysis
      --dynamic-arg <ARG>             Specify arguments passed to the executable in --dynamic
      --dynamic-stdin <CONTENT>       Specify stdin content supplied to the executable in --dynamic
  -c, --compress                      Compress the executable with npx
      --upx-arg <ARG>                 Specify arguments passed to upx in --compress
      --busybox <PATH or NAME>        Specify the path or name of busybox that would be used in testing [default: busybox]
      --upx <PATH or NAME>            Specify the path or name of upx that would be used in compression [default: upx]
      --cc <PATH or NAME>             Specify the path or name of c compiler that would be used in the name resolution of shared library dependencies [env: CC=] [default: cc]
      --experimental-noload-resolver  [EXPERIMENTAL] Resolve dynamic library paths without loading in dlopen(3)
  -h, --help                          Print help information

Docker images

We provide some base images that contain magicpak and its optional dependencies to get started.

name	description
magicpak/debian	library/debian with magicpak
magicpak/cc	library/debian with build-essential, clang, and magicpak
magicpak/haskell	library/haskell with magicpak
magicpak/rust	library/rust with magicpak

Example

The following is a dockerfile using magicpak for a docker image of clang-format, a formatter for C/C++/etc. (example/clang-format)

FROM magicpak/debian:buster-magicpak1.4.0

RUN apt-get -y update
RUN apt-get -y --no-install-recommends install clang-format

RUN magicpak $(which clang-format) /bundle -v  \
      --compress                               \
      --upx-arg --best                         \
      --test                                   \
      --test-stdin "int main(  ){ }"           \
      --test-stdout "int main() {}"            \
      --install-to /bin/

FROM scratch
COPY --from=0 /bundle /.

WORKDIR /workdir

CMD ["/bin/clang-format"]

Note on name resolution and glibc

If your program uses glibc for name resolution (most likely it does), the call to getaddrinfo(3) will result in an error after bundled by magicpak. This can be resolved by manually including the NSS-related shared libraries as shown below.

# example on x86_64 Debian-based image:
RUN magicpak path/to/executable /bundle --include '/lib/x86_64-linux-gnu/libnss_*'

Note on jemalloc

If your program depends on libjemalloc, magicpak may fail with the following message.

error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block

You can use --experimental-noload-resolver flag to workaround this. See #19 for details.

Disclaimer

magicpak comes with absolutely no warranty. There's no guarantee that the processed bundle works properly and identically to the original executable. Although I had no problem using magicpak for building various kinds of images, it is recommended to use this with caution and make a careful examination of the resulting bundle.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

magicpak's Issues

TLS issues on ubuntu 20 focal

don't have reproduction steps ... it works on a plain focal, but we have a ruby build that adds jmalloc and then crashes with this during packing, hoping there is some obvious "you need to set xyz" fix 🤞
(bionic works fine for some reason)

#14 0.229  INFO exe: loading location=/lib/aarch64-linux-gnu/librt.so.1
#14 0.229  WARN exe: interpreter could not be found. static or compressed executable?
#14 0.229  WARN exe: requesting dynamic libraries of the executable without the interpreter
#14 0.230 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
#14 ERROR: executor failed running [/bin/bash -xc /usr/bin/magicpak -v $(which ruby) /magicpak]: exit code: 1

v2

• hide (or separate) library interface to reduce public API surface
  • then rename Executable to something like DynamicObject
• define CLI as a public API in terms of Semantic Versioning
  • then change positional arguments into flags
  • maybe we can just mix --include and inputs
• remove --compress
  • then remove Bundle::add_pseudo_proc which has confusing effect on --test environment
• stop providing variations of base image
  • it is difficult to maintain versioned tags of upstream images
  • instead we can distribute an container image just containing magicpak binary so that users can COPY from that image

Build container images periodically

Necessary to keep images up-to-date with base images

Builds and base docker images for aarch64

Enjoying magicpak very much ❤️ and I think it would be of great value to also provide builds as well as docker base images (for haskell/rust/...) (especially for Haskell in my case :-) ) also for the arm64 architecture. When developing for Raspberry Pi this could greatly help in producing smaller images that are easier to run on this resource constrained device.

support multiple inputs

I have a project that needs to use 2 executables, ruby and mysql.
Should I be generating 2 packs or can this be supported ?

More cpu usage / throttling after switching to from scratch

for a ruby example app that we converted we saw cpu and throttling jump pretty bad (it's not a big deal since it's still super small), but some guidance on why this might happen and how to avoid it would be great for onboarding apps where performance actually matters.

Improve error messages

• gets uninformative message when cc is not found

  magicpak/src/action/bundle_shared_object_dependencies.rs

  Line 14 in e9fb346

  let cc_path = which::which(cc)?;

• empty path causes panic

  magicpak/src/domain/executable.rs

  Line 96 in de1e87f

  let file_name = path.file_name().unwrap();

Device or resource busy - getaddrinfo when doing ruby http calls on ubuntu

Device or resource busy - getaddrinfo (Errno::EBUSY)

did not find anything on the web that would help ... something about ulimit might be setup wrong :/

require 'net/http'
Net::HTTP.get_response(URI('http://www.google.com'))

ulimit seems fine:
/proc/sys/fs/file-max -> 524288

FROM ruby as ruby

ADD https://github.com/coord-e/magicpak/releases/download/v1.3.0/magicpak-x86_64-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak && /usr/bin/magicpak -v "$(which ruby)" /magicpak

FROM scratch

COPY --from=ruby /magicpak /.
COPY --from=ruby /usr/local/lib/ruby /usr/local/lib/ruby

RUN ["ruby", "-r", "net/http", "-e", "Net::HTTP.get_response(URI('http://www.google.com'))"]

The order of the -ldl option is out of place at run-time

I noticed that the command being executed has the -ldl out of place w.r.t. the source code.

[DEBUG] command: "/usr/bin/cc" "-Wl,-dynamic-linker,/lib64/ld-linux-x86-64.so.2" "-ldl" "-o" "/tmp/magicpak_resolver_14160520991579680778_3249826909874418912" "-xc" "/tmp/.tmpsNXfuc"

However, the following command succeeds:

$ "/usr/bin/cc" "-Wl,-dynamic-linker,/lib64/ld-linux-x86-64.so.2" "-o" "/tmp/magicpak_resolver_14160520991579680778_3249826909874418912" "-xc" "/tmp/.tmpsNXfuc" "-ldl"
$ echo $?
0

My current workaround is to create a wrapper script around GCC with a hardcoded -ldl as the last option and pass it to magicpack using --cc.

Test in AArch64

Currently tests in AArch46 is entirely disabled:

magicpak/.github/workflows/test.yml

Line 40 in de1e87f

if: matrix.target == 'x86_64-unknown-linux-musl' # TODO: test aarch64

We need to use native ARM builder because we can't ptrace on QEMU.
https://cirrus-ci.org/ might be an option.