mic92 / nix-ld

Run unpatched dynamic binaries on NixOS

nix-ld

Run unpatched dynamic binaries on NixOS.

Where is this useful?

While many proprietary packages in nixpkgs have already been patched with autoPatchelfHook patching, there are cases where patching is not possible:

• Use binary executable downloaded with third-party package managers (e.g. vscode, npm or pip) without having to patch them on every update.
• Run games or proprietary software that attempts to verify its integrity.
• Run programs that are too large for the nix store (e.g. FPGA IDEs).

While there are other solutions such as buildFHSUserEnv that restore a Linux file hierarchy as found on common Linux systems (ld-linux-x86-64.so.2), these sandboxes have their own weaknesses:

• setuid binaries cannot be executed inside a fhsuserenv
• inside a buildFHSUserEnv you can not use other sandbox tools like bwrap or 'nix build'.
• buildFHSUserEnv requires a subshell which does not work well with direnv

How does nix-ld work?

Also read this blog post to get the explaination in full detail. A summary is below:

Precompiled binaries that were not created for NixOS usually have a so-called link-loader hardcoded into them. On Linux/x86_64 this is for example /lib64/ld-linux-x86-64.so.2. for glibc. NixOS, on the other hand, usually has its dynamic linker in the glibc package in the Nix store and therefore cannot run these binaries. Nix-ld provides a shim layer for these types of binaries. It is installed in the same location where other Linux distributions install their link loader, ie. /lib64/ld-linux-x86-64.so.2 and then loads the actual link loader as specified in the environment variable NIX_LD. In addition, it also accepts a colon-separated path from library lookup paths in NIX_LD_LIBRARY_PATH. This environment variable is rewritten to LD_LIBRARY_PATH before passing execution to the actual ld. This allows you to specify additional libraries that the executable needs to run.

Installation

nix-ld is part of nixpkgs since NixOS 22.05. There one can enable it with the following nixos setting:

{
  programs.nix-ld.enable = true;
}

To install nix-ld from the repository instead, use the following method:

$ sudo nix-channel --add https://github.com/Mic92/nix-ld/archive/main.tar.gz nix-ld
$ sudo nix-channel --update

/etc/nixos/configuration.nix

{
  imports = [
    <nix-ld/modules/nix-ld.nix>
  ];
  # The module in this repository defines a new module under (programs.nix-ld.dev) instead of (programs.nix-ld) 
  # to not collide with the nixpkgs version.
  programs.nix-ld.dev.enable = true;
}

With nix flake

Add the following lines to /etc/nixos/flake.nix. Replace myhostname with the actual hostname of your system.

# flake.nix
{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/master";
  inputs.nix-ld.url = "github:Mic92/nix-ld";
  # this line assume that you also have nixpkgs as an input
  inputs.nix-ld.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { nix-ld, nixpkgs, ... }: {
    # replace `myhostname` with your actual hostname
    nixosConfigurations.myhostname = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        # ... add this line to the rest of your configuration modules
        nix-ld.nixosModules.nix-ld

        # The module in this repository defines a new module under (programs.nix-ld.dev) instead of (programs.nix-ld) 
        # to not collide with the nixpkgs version.
        { programs.nix-ld.dev.enable = true; }
      ];
    };
  };
}

Usage

After setting up the nix-ld symlink as described above, one needs to set NIX_LD and NIX_LD_LIBRARY_PATH to run executables. For example, this can be done with a shell.nix in a nix-shell like this:

with import <nixpkgs> {};
mkShell {
  NIX_LD_LIBRARY_PATH = lib.makeLibraryPath [
    stdenv.cc.cc
    openssl
    # ...
  ];
  NIX_LD = lib.fileContents "${stdenv.cc}/nix-support/dynamic-linker";
}

A full example is shown in ./examples/masterpdfeditor.nix.

In nix-autobahn there is also a script called nix-autobahn-ld that automates generating shell expressions.

In nix-alien there is another script called nix-alien-ld that uses another strategy, wrapping the program in a writeShellScriptBin with the NIX_LD/NIX_LD_LIBRARY_PATH environment variables set.

To figure out what libraries a program needs, you can use ldd on the binary or set the LD_DEBUG=libs environment variable.

Known Issues

LD_LIBRARY_PATH is inherited by child processes

nix-ld is currently rewrites NIX_LD_LIBRARY_PATH to LD_LIBRARY_PATH. This can cause problems if a program loaded with this loader executes a normal binary, which should not get these libraries. In the future, it may be possible to redirect execution back to nix-ld after the actual library loader has done its job by changing the entry point in memory to fix this.

FAQ

How to find libraries for my executables?

You can use tools like nix-autobahn, nix-alien or use nix-index

Why not set LD_LIBRARY_PATH directly instead of NIX_LD_LIBRARY_PATH?

LD_LIBRARY_PATH affects all programs, which can inject the wrong libraries in correct build nix application that have an RPATH set in their executable.

Does this work on non-NixOS system?

No. Normal Linux distributions will have their own link-loader. Replacing those with nix-ld will break the system.

My python/nodejs/ruby/$interpreter libraries do not find the libraries configured by nix-ld

Nix-ld is only used by unpatched executables that use the link loader at /lib or /lib64. If you use for example python from nixpkgs than it will not pick up NIX_LD_LIBRARY_PATH and NIX_LD since these types of binaries are configured to use a glibc from the nix store. If you encounter these cases i.e. when you are trying to use python packages installed in a virtualenv than you need to set LD_LIBRARY_PATH directly. You can also create yourself a wrapper like this:

(pkgs.writeShellScriptBin "python" ''
  export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
  exec ${pkgs.python3}/bin/python "$@"
'')

nix-ld's Issues

Docker image support

According to https://github.com/Mic92/nix-ld#does-this-work-on-non-nixos-system, nix-ld does not support non-NixOS systems.

I wonder if it is possible to add nix-ld's ld-linux-x86-64.so.2 to a Docker image that does not include neither NixOS nor other Linux distribution, i.e. built from pkgs.dockerTools.buildLayeredImage, which is not a NixOS because most container runtimes do not support systemd and tmpfiles.d.

Dynamic linking for `boot.binfmt.emulatedSystems`

nix-ld is nice, thanks for putting it together!

I was wondering if it would be possible to make it work for emulated binaries. The current failure (running an aarch64-linux dynamically linked binary on an x86_64-linux box) is:

qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory

I'd need some way of pulling in the binaries for the libraries on the alternative system and support from nix-ld somewhere. Does it seem possible?

Can nix-ld be used in docker image created with nix2container?

Hi, I'm trying to create minimal docker image, that can run binaries. I've created repo @ https://github.com/nxy7/tiny and published this image to docker hub (in case anyone wants to test it) as nxyt/tiny:latest. Anyway - it doesn't work and any binary I run results in bash: ./main: cannot execute: required file not found.
As far as I understand binaries expect linker at specific location, so even if ldd my-binary shows that all dependencies are present it still tries to call linker at /lib64/xxx and because there's none, that's why my binaries are failing.

If I got that right, that's the problem that nix-ld is trying to solve, but I cannot really get that to work with nix2container.
What I've tried was adding nix_ld to copyToRoot and adding 'lib' and 'lib64' to 'pathsToLink' and I've also added NIX_LD env variable, but that still doesn't work.

Can nix-ld be used to fix that error, or is it made only for NixOS? Any guidance toward getting that to work would be great :-)

value is a function while a set was expected

1. Enabled the module and saw I got the file in /lib64/ld-linux-x86_64.so.2.
2. Using this change in my /etc/nixos/flake.nix:

diff --git i/flake.nix w/flake.nix
index 0b86e95..306bf02 100644
--- i/flake.nix
+++ w/flake.nix
@@ -2,13 +2,21 @@
   inputs = {
     nixpkgs.url = "github:doronbehar/nixpkgs/myChannel";
     syncthing-private.url = "git+file:///etc/nixos/syncthing-private";
+    nix-ld = {
+      url = "github:Mic92/nix-ld";
+      # this line assume that you also have nixpkgs as an input
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
   };
-  outputs = { self, nixpkgs, syncthing-private }: {
+  outputs = { self, nixpkgs, nix-ld, syncthing-private }: {
     nixosConfigurations = (
       nixpkgs.lib.genAttrs [ "ZENIX" "NUX" ]
       (hostname: nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
-         modules = [ (import ./configuration.nix hostname syncthing-private) ];
+         modules = [
+          nix-ld.nixosModules.nix-ld
+           (import ./configuration.nix hostname syncthing-private)
+         ];
       })
     ) // {
         # Other machines, with dedicated, configuartion.nix

And then:

$ nix-shell zoom.nix
at: (281:44) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   280|         options = {};
   281|         config = addFreeformType (addMeta (removeAttrs m ["_file" "key" "disabledModules" "require" "imports" "freeformType"]));
      |                                            ^
   282|       };

value is a function while a set was expected
(use '--show-trace' to show detailed location information)

And:

$ nix shell nixpkgs#nixStable -c nix-shell zoom.nix
warning: unknown setting 'experimental-features'
error: value is a function while a set was expected, at /nix/store/g9ph7c86sbwyp9pyv6wfdh9s0yl6h3lg-myChannel.tar.gz/lib/modules.nix:281:44
(use '--show-trace' to show detailed location information)

Full trace:

error: --- TypeError --------------------------------------------------------------------------------------------------------------- nix-shell
at: (281:44) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   280|         options = {};
   281|         config = addFreeformType (addMeta (removeAttrs m ["_file" "key" "disabledModules" "require" "imports" "freeformType"]));
      |                                            ^
   282|       };

value is a function while a set was expected
----------------------------------------------------------------- show-trace -----------------------------------------------------------------
trace: while evaluating 'addMeta'
at: (256:17) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   255|     let
   256|       addMeta = config: if m ? meta
      |                 ^
   257|         then mkMerge [ config { meta = m.meta; } ]

trace: from call site
at: (281:35) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   280|         options = {};
   281|         config = addFreeformType (addMeta (removeAttrs m ["_file" "key" "disabledModules" "require" "imports" "freeformType"]));
      |                                   ^
   282|       };

trace: while evaluating 'addFreeformType'
at: (259:25) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   258|         else config;
   259|       addFreeformType = config: if m ? freeformType
      |                         ^
   260|         then mkMerge [ config { _module.freeformType = m.freeformType; } ]

trace: from call site
at: (281:18) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   280|         options = {};
   281|         config = addFreeformType (addMeta (removeAttrs m ["_file" "key" "disabledModules" "require" "imports" "freeformType"]));
      |                  ^
   282|       };

trace: while evaluating the attribute 'config'
at: (281:9) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   280|         options = {};
   281|         config = addFreeformType (addMeta (removeAttrs m ["_file" "key" "disabledModules" "require" "imports" "freeformType"]));
      |         ^
   282|       };

trace: while evaluating 'pushDownProperties'
at: (576:24) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   575|   */
   576|   pushDownProperties = cfg:
      |                        ^
   577|     if cfg._type or "" == "merge" then

trace: from call site
at: (337:73) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   336|     mergeModules' prefix modules
   337|       (concatMap (m: map (config: { file = m._file; inherit config; }) (pushDownProperties m.config)) modules);
      |                                                                         ^
   338|

trace: while evaluating anonymous lambda
at: (337:19) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   336|     mergeModules' prefix modules
   337|       (concatMap (m: map (config: { file = m._file; inherit config; }) (pushDownProperties m.config)) modules);
      |                   ^
   338|

trace: from call site
at: (337:8) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   336|     mergeModules' prefix modules
   337|       (concatMap (m: map (config: { file = m._file; inherit config; }) (pushDownProperties m.config)) modules);
      |        ^
   338|

trace: while evaluating 'byName'
at: (362:25) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   361|       */
   362|       byName = attr: f: modules:
      |                         ^
   363|         foldl' (acc: module:

trace: from call site
at: (374:21) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   373|       # an attrset 'name' => list of submodules that define ‘name’.
   374|       defnsByName = byName "config" (module: value:
      |                     ^
   375|           map (config: { inherit (module) file; inherit config; }) (pushDownProperties value)

trace: while evaluating 'byName'
at: (362:25) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   361|       */
   362|       byName = attr: f: modules:
      |                         ^
   363|         foldl' (acc: module:

trace: from call site
at: (378:22) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   377|       # extract the definitions for each loc
   378|       defnsByName' = byName "config" (module: value:
      |                      ^
   379|           [{ inherit (module) file; inherit value; }]

trace: while evaluating the attribute 'values'
at: (637:7) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   636|     in {
   637|       values = concatMap (def: if getPrio def == highestPrio then [(strip def)] else []) defs;
      |       ^
   638|       inherit highestPrio;

trace: while evaluating the attribute 'values'
at: (538:9) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   537|       in {
   538|         values = defs''';
      |         ^
   539|         inherit (defs'') highestPrio;

trace: while evaluating the attribute 'mergedValue'
at: (544:5) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   543|     # Type-check the remaining definitions, and merge them. Or throw if no definitions.
   544|     mergedValue =
      |     ^
   545|       if isDefined then

trace: while evaluating the option `_module.freeformType':
trace: while evaluating the attribute 'value'
at: (512:9) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   511|     in warnDeprecation opt //
   512|       { value = builtins.addErrorContext "while evaluating the option `${showOption loc}':" value;
      |         ^
   513|         inherit (res.defsFinal') highestPrio;

trace: while evaluating anonymous lambda
at: (139:72) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

   138|           # For definitions that have an associated option
   139|           declaredConfig = mapAttrsRecursiveCond (v: ! isOption v) (_: v: v.value) options;
      |                                                                        ^
   140|

trace: from call site
at: (279:20) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/attrsets.nix

   278|               then recurse (path ++ [name]) value
   279|               else f (path ++ [name]) value;
      |                    ^
   280|         in mapAttrs g set;

trace: while evaluating 'g'
at: (276:19) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/attrsets.nix

   275|           g =
   276|             name: value:
      |                   ^
   277|             if isAttrs value && cond value

trace: from call site
trace: while evaluating the attribute '_module.freeformType'
trace: while evaluating 'evalModules'
at: (62:17) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/modules.nix

    61|      evalModules) and the less declarative the module set is. */
    62|   evalModules = { modules
      |                 ^
    63|                 , prefix ? []

trace: from call site
at: (58:12) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/nixos/lib/eval-config.nix

    57|   # system configuration.
    58|   inherit (lib.evalModules {
      |            ^
    59|     inherit prefix check;

trace: while evaluating the attribute 'config.nixpkgs.overlays'
at: (64:5) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/nixos/lib/eval-config.nix

    63|       { modulesPath = builtins.toString ../modules; } // specialArgs;
    64|   }) config options _module;
      |     ^
    65|

trace: while evaluating anonymous lambda
at: (1:7) in file: /etc/nixos/overlays-compat/overlays.nix

     1| self: super:
      |       ^
     2| with super.lib;

trace: from call site
at: (69:67) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/fixed-points.nix

    68|   #
    69|   extends = f: rattrs: self: let super = rattrs self; in super // f self super;
      |                                                                   ^
    70|

trace: while evaluating 'extends'
at: (69:24) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/fixed-points.nix

    68|   #
    69|   extends = f: rattrs: self: let super = rattrs self; in super // f self super;
      |                        ^
    70|

trace: from call site
at: (69:42) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/fixed-points.nix

    68|   #
    69|   extends = f: rattrs: self: let super = rattrs self; in super // f self super;
      |                                          ^
    70|

trace: while evaluating 'extends'
at: (69:24) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/fixed-points.nix

    68|   #
    69|   extends = f: rattrs: self: let super = rattrs self; in super // f self super;
      |                        ^
    70|

trace: from call site
at: (19:20) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/fixed-points.nix

    18|   # details.
    19|   fix = f: let x = f x; in x;
      |                    ^
    20|

trace: while evaluating 'fix'
at: (19:9) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/lib/fixed-points.nix

    18|   # details.
    19|   fix = f: let x = f x; in x;
      |         ^
    20|

trace: from call site
at: (252:3) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/stage.nix

   251|   # Return the complete set of packages.
   252|   lib.fix toFix
      |   ^

trace: while evaluating anonymous lambda
at: (12:1) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/stage.nix

    11|
    12| { ## Misc parameters kept the same for all stages
      | ^
    13|   ##

trace: from call site
at: (116:26) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/default.nix

   115|   # sets. Only apply arguments which no stdenv would want to override.
   116|   allPackages = newArgs: import ./stage.nix ({
      |                          ^
   117|     inherit lib nixpkgsFun;

trace: while evaluating 'allPackages'
at: (116:17) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/default.nix

   115|   # sets. Only apply arguments which no stdenv would want to override.
   116|   allPackages = newArgs: import ./stage.nix ({
      |                 ^
   117|     inherit lib nixpkgsFun;

trace: from call site
at: (101:12) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

   100|       then args'
   101|       else allPackages ((builtins.removeAttrs args' ["selfBuild"]) // {
      |            ^
   102|         adjacentPackages = if args.selfBuild or true then null else rec {

trace: while evaluating 'folder'
at: (89:33) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

    88|   # debugging purposes.
    89|   folder = nextStage: stageFun: prevStage: let
      |                                 ^
    90|     args = stageFun prevStage;

trace: from call site
at: (68:18) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

    67|           # Note the cycle -- call-by-need ensures finite fold.
    68|           cur  = op pred (builtins.elemAt list n) succ;
      |                  ^
    69|           succ = go cur (n + 1);

trace: while evaluating 'go'
at: (63:18) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

    62|       len = builtins.length list;
    63|       go = pred: n:
      |                  ^
    64|         if n == len

trace: from call site
at: (72:13) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

    71|       lapp = lnul cur;
    72|       cur = go lapp 0;
      |             ^
    73|     in cur;

trace: while evaluating 'dfold'
at: (60:27) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

    59|   */
    60|   dfold = op: lnul: rnul: list:
      |                           ^
    61|     let

trace: from call site
at: (136:4) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

   135|
   136| in dfold folder postStage (_: {}) withAllowCustomOverrides
      |    ^

trace: while evaluating anonymous lambda
at: (42:1) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/stdenv/booter.nix

    41| # other words, this does a foldr not foldl.
    42| stageFuns: let
      | ^
    43|

trace: from call site
at: (126:10) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/default.nix

   125|
   126|   pkgs = boot stages;
      |          ^
   127|

trace: while evaluating anonymous lambda
at: (20:1) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/default.nix

    19|
    20| { # The system packages will be built on. See the manual for the
      | ^
    21|   # subtle division of labor between these two `*System`s and the three

trace: from call site
at: (84:1) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/impure.nix

    83|
    84| import ./. (builtins.removeAttrs args [ "system" "platform" ] // {
      | ^
    85|   inherit config overlays crossSystem crossOverlays;

trace: while evaluating anonymous lambda
at: (15:1) in file: /nix/store/bqkrixcaa2h9awdsb4b5rrwc288fl264-source/pkgs/top-level/impure.nix

    14|
    15| { # We combine legacy `system` and `platform` into `localSystem`, if
      | ^
    16|   # `localSystem` was not passed. Strictly speaking, this is pure desugar, but

trace: from call site
at: (1:6) in file: /home/doron/repos/nix-ld/examples/zoom.nix

     1| with import <nixpkgs> {};
      |      ^
     2|

Don't allow NIX_LD on setuid binaries

In the unlikely event if someone has NIX_LD set and an setuid binary using /lib64/ld-linux-x86-64.so.2, an attacker might be able to diverge execution by pointing to a untrusted patched libc. This should never happen in normal usage of nixos.

Discussion: An Idea for fixing the child inheritance problem

This isn't a feature request; I'm considering working on the problem myself in a different tool. I just want to make sure I'm not trying something that definitely won't work.

I know mutation is a bad idea, but let's ignore just that one aspect for now.

_

Let's say a binary runs, and calls nix-ld.

If there's no way to know what binary is calling nix-ld then this idea is dead in the water. But I'm hoping there's some way, maybe via strace or some similar tool if there's no direct way to know.

If that's the case, I imagine nix-ld could predict the absolute path of each of the .so files the target binary was looking for (using nix ld env var).

In that case, could it mutate the RPATH (not RUNPATH) of the binary at runtime? And then simply not change any env vars for the binary.

If it wouldn't work at runtime, do you think it could it work on the second run? (fall on the first call, then patch, then call again)

_

Back to the mutation problem; if there is a test suite for a tool, and we are running the test suite over and over again inside of a nix build, even if the binary is mutating if it's stable by the end of the test suit calls then the final build of a nix tool with a lot of outside binaries could still lead to a consistent output.

Decouple configuration from activation

In order to easily run applications on the system that are not patched (steam-run is not practical at all and integrates poorly with the host system) I set NIX_LD and NIX_LD_LIBRARY_PATH on my whole system. It worked great so far but I don't really like the idea that my system may be less reproducible as nix-ld is now always enabled. So I would love instead if nix-ld could read an environment variable like NIX_ENABLE_FHS that would enable it if it is set. Of course I could manually set NIX_LD_LIBRARY_PATH when I want a shell but it is not as practical to manually specify a huge list of items. Moreover I am also thinking to use this same environment variable to enable other systems like a /bin/bash or other programs (for modularity one could also define another environment variable when one wants to enable only nix_ld but not /bin/bash).

This also have the advantage that if the NIX_ENABLE_FHS is not set, then the user can get a meaningful error explaining that the loader /lib are disabled by default but could easily be enabled (in a quick and dirty way) by setting the above environment variable or by writting a proper derivation. We could also provide more meaningful default values that can run most programs. This way, nix-ld could even be set by default on nixos to provide more helpful help messages.

Do you think it would make sense to integrate this into nix-ld or should I create my own project to test these ideas?

thoughts on reading NIX_LD and NIX_LD_LIBRARY_PATH from configuration file?

I find building fuchsia difficult because fuchsia's build script runs env --ignore-environment. My manually-set environment variableNIX_LD is thus unset. If nix-ld is to read the interpreter from some configuration file, then I can run fuchsia's pre-built binaries. What is your opinion of adding configuration file support?

Support 32 bits applications on 64 bits systems

As far as I understand, in most cases 32 bits applications should work on 64 bits systems. However the nixos module creates a single folder /lib64 making it unable to run 32 bits applications.

The discussion was started here #27

Whitelist/Blacklist paths that can use nix-ld

I really like nix-ld tho it can come with some security issues, so I was thinking.
Is it possible to restrict the binaries that can be run by nix-ld or is impossible by design?

sigsegv trying to start remote IDE session via JetBrains Gateway

When trying to start a remote IDE session via JetBrains Gateway on a nix-ld enabled host, java crash-loops with SEGV:

# coredumpctl info
           PID: 205075 (ld-linux-x86-64)
           UID: 1000 (…)
           GID: 100 (users)
        Signal: 11 (SEGV)
     Timestamp: Tue 2023-01-17 12:16:55 CET (3min 3s ago)
  Command Line: /lib64/ld-linux-x86-64.so.2 --library-path /home/…/.cache/JetBrains/RemoteDev/dist/1e90e5f5fc61f_CLion-2022.3.1/plugins/remote-dev-server/selfcontained/lib /home/…/.cache/JetBrains/RemoteDev-CL/…/pid.204940.temp.jbr/bin/java.bin …
    Executable: /nix/store/pm0ihc6sl28ishcz6w8gxfxg4f6rzj8r-nix-ld-1.0.3/libexec/nix-ld
 Control Group: /user.slice/user-1000.slice/session-3.scope
          Unit: session-3.scope
         Slice: user-1000.slice
       Session: 3
     Owner UID: 1000 (…)
       Boot ID: …
    Machine ID: …
      Hostname: …
       Storage: /var/lib/systemd/coredump/core.ld-linux-x86-64.….zst (present)
  Size on Disk: 12.9K
       Message: Process 205075 (ld-linux-x86-64) of user 1000 dumped core.

                Module /nix/store/pm0ihc6sl28ishcz6w8gxfxg4f6rzj8r-nix-ld-1.0.3/libexec/nix-ld without build-id.
                Stack trace of thread 205075:
                #0  0x00007f1a213d3b60 _dl_process_pt_gnu_property (/nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 + 0x7b60)
                #1  0x00007f1a213eb457 dl_main (/nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 + 0x1f457)
                #2  0x00007f1a213e8a06 _dl_sysdep_start (/nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 + 0x1ca06)
                #3  0x00007f1a213ea5ad _dl_start (/nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 + 0x1e5ad)
                #4  0x00007f1a213e93a8 _start (/nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 + 0x1d3a8)
                ELF object binary architecture: AMD x86-64

I was unable to debug with gdb due to missing debuginfo. Do you happen to know how to override /lib64/ld-linux-x86-64.so.2 with debuginfo, without having to override glibc globally, which would trigger a full system rebuild?

nix-ld breaks if executable contains a TLS section

$ readelf -l zoom/opt/zoom/zoom

Elf file type is DYN (Shared object file)
Entry point 0x479b2f
There are 10 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x0000000000000230 0x0000000000000230  R E    0x8
  INTERP         0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000043b1db2 0x00000000043b1db2  R E    0x200000
  LOAD           0x00000000043b27b0 0x00000000045b27b0 0x00000000045b27b0
                 0x000000000018ca88 0x000000000028c818  RW     0x200000
  DYNAMIC        0x00000000045182c8 0x00000000047182c8 0x00000000047182c8
                 0x00000000000003f0 0x00000000000003f0  RW     0x8
  NOTE           0x000000000000028c 0x000000000000028c 0x000000000000028c
                 0x0000000000000044 0x0000000000000044  R      0x4
  TLS            0x00000000043b27b0 0x00000000045b27b0 0x00000000045b27b0
                 0x0000000000000010 0x0000000000000010  R      0x8
  GNU_EH_FRAME   0x0000000003a899d0 0x0000000003a899d0 0x0000000003a899d0
                 0x000000000012d084 0x000000000012d084  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x00000000043b27b0 0x00000000045b27b0 0x00000000045b27b0
                 0x000000000016f850 0x000000000016f850  R      0x1
$ gdb ./zoom/opt/zoom/zoom
#0  0x00007ffff7ff8941 in memcpy () from /lib64/ld-linux-x86-64.so.2
#1  0x00007ffff7ff8c6d in __copy_tls () from /lib64/ld-linux-x86-64.so.2
#2  0x00007ffff7ff8e09 in static_init_tls () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7ff4e5f in __init_libc () from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff7ff4f7d in __libc_start_main () from /lib64/ld-linux-x86-64.so.2
#5  0x00007ffff7ff470a in _start () from /lib64/ld-linux-x86-64.so.2
#6  0x0000000000000001 in ?? ()
#7  0x00007fffffffa87e in ?? ()
#8  0x0000000000000000 in ?? ()

Why does it fail and can we disable force musl to ignore this section?

Enable nix-ld system-wide and include all system packages

Is there a recommended way to enable nix-ld system-wide on NixOS and include all specifyied systemPackages (and also ones that are added through other options, like libGL etc.)?

This would make running unpatched binaries much easier I think.

arm64 non-NixOS support on amd64?

Hi!

I have binfmt working on my NixOS installation and it works for cross-compiled NixOS binaries, but not for non-NixOS aarch64 binaries. I think nix-ld should work for this purpose, but I am not sure how?

I get
qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory

My try:
programs.nix-ld.libraries = with pkgs; [ pkgsCross.aarch64-multiplatform.glibc ];

Executing binary with link loader SIGSEVs

 ▶ strace /lib64/ld-linux-x86-64.so.2 $(which bash)
execve("/lib64/ld-linux-x86-64.so.2", ["/lib64/ld-linux-x86-64.so.2", "/run/current-system/sw/bin/bash"], 0x7ffca101e9c8 /* 151 vars */) = 0
openat(AT_FDCWD, "/nix/store/axkalfklvygxjlmc9vivkyb4xy18v30f-ld.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\323\1\0\0\0\0\0@\0\0\0\0\0\0\0\360\315\3\0\0\0\0\0\0\0\0\0@\08\0\v\0@\0\32\0\31\0", 64) = 64
mmap(NULL, 680, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5dc6ce9000
mmap(NULL, 233472, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5dc6cb0000
mmap(0x7f5dc6cb2000, 159744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x7f5dc6cb2000
mmap(0x7f5dc6cd9000, 49152, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x29000) = 0x7f5dc6cd9000
mmap(0x7f5dc6ce5000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x34000) = 0x7f5dc6ce5000
munmap(0x7f5dc6ce9000, 680) = 0
close(3)              = 0
brk(NULL)             = 0x5555559ed000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd3d959300) = -1 EINVAL (Invalid argument)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x238} ---
+++ killed by SIGSEGV +++
Segmentation fault

This works with the normal ld

$ /nix/store/76l4v99sk83ylfwkz8wmwrm4s8h73rhd-glibc-2.35-224/lib/ld-linux-x86-64.so.2 $(which bash)
 ▶

error: access to canonical path is forbidden in restricted mode

Following the Nix flake instructions, I get

$ nixos-rebuild build --flake path:.
error: access to canonical path '/nix/store/55ayck4pllh4qallb1dbhbkrzwrrl02r-binutils-wrapper-2.35.2/nix-support/dynamic-linker' is forbidden in restricted mode

my diffs are quite simple

 {
+  inputs.nixpkgs.url = "github:nixos/nixpkgs";
   inputs.home-manager = {
     url = "github:rycee/home-manager/master";
     inputs.nixpkgs.follows = "nixpkgs";
   };
+  inputs.nix-ld = {
+    url = "github:Mic92/nix-ld";
+    inputs.nixpkgs.follows = "nixpkgs";
+  };

-  outputs = { self, nixpkgs, home-manager, ... }@inputs:
+  outputs = { self, nixpkgs, nix-ld, home-manager, ... }@inputs:
     let
       system = "x86_64-linux";
       pkgs = import nixpkgs {
@@ -22,6 +25,8 @@
         modules = [
           ./machines/poseideep/configuration.nix

+          nix-ld.nixosModules.nix-ld
+
           home-manager.nixosModules.home-manager

in my home manager config

+    home.sessionVariables =
+      let
+        cc = pkgs.stdenv.cc;
+      in
+      {
+        NIX_LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [
+          cc.cc
+        ];
+        NIX_LD = pkgs.lib.fileContents "${cc}/nix-support/dynamic-linker";
+      };

The existence of this project is depressing

The problem it solves is described in 127-word paragraph. It solves an edge case of a leaky abstraction in software distribution. There's no place for neither this project nor the problem it solves had the entire modern computing stack got rewritten from first principles today. The industry is sorely lacking, and we are cavemen swinging sticks.

Remove Need For Environment Variables?

Hi! What if nix-ld.c inspected the PT_LOAD segments put into RAM by the ELF handler in order to get a list of shared objects required by the binary being run, used nix-index to find suitable packages that provide those libraries, and then used lib.makeLibraryPath on those packages to compute the correct value for NIX_LD_LIBRARY_PATH automatically?

Would this work??? I want to attempt this myself but I don't know how I would go about debugging additional code that I write for this project. What does your dev environment for working on nix-ld look like?

I have so many packages that run in various types of env-dropping sandboxes (which seems to be an issue for other people including @contrun from issue #17) and something like this would make my life single-booting NixOS so much more convenient. Also, it would be really nice to be able to run foreign executables without having to use tools like nix-alien and nix-autobahn.

Thanks so much

Build is broken for 32 bit targets

Disclaimer: I'm not actually relying on this (and it seems there aren't many people that do, given that the error went unnoticed for 2 years), but simply opening a ticket so I have some documentation I can link to in a ZHF PR (marking nix-ld as broken for 32 bit targets for now).

nix-ld doesn't build for 32 bit targets since the move to nolibc, as UINTPTR_MAX is #defined as nothing:

nix-ld/src/nix-ld.c

Lines 20 to 21 in 6ac25c2

	#if PTR_SIZE == 4
	#define UINTPTR_MAX

This leads to compilation errors, as the assignment on line 127 later on results in a missing expression after macro expansion, e.g.:
Hydra build for i686: https://hydra.nixos.org/build/241835411#tabs-summary
Log: https://hydra.nixos.org/build/241835411/nixlog/2

[...]
Found ninja-1.11.1 at /nix/store/qzrhb4pkv61dfmax136wjx2g2mfq64ll-ninja-1.11.1/bin/ninja
mesonConfigurePhase: enabled\ parallel\ building
building
build flags: -j2
[1/8] Compiling C object test/libexample-lib.so.p/example-lib.c.o
[2/8] Linking target test/libexample-lib.so
[3/8] Compiling C object nix-ld.p/src_nix-ld.c.o
FAILED: nix-ld.p/src_nix-ld.c.o 
gcc -Inix-ld.p -I. -I.. -I../vendor/nolibc -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -ffast-math -fno-common -fdiagnostics-show-option -fno-strict-aliasing -fvisibility=hidden -fcf-protection --param=ssp-buffer-size=4 -fPIE -fno-asynchronous-unwind-tables -fno-ident -ffreestanding -static-pie -DPTR_SIZE=4 -MD -MQ nix-ld.p/src_nix-ld.c.o -MF nix-ld.p/src_nix-ld.c.o.d -o nix-ld.p/src_nix-ld.c.o -c ../src/nix-ld.c
../src/nix-ld.c: In function 'total_mapping_size':
../src/nix-ld.c:127:32: error: expected expression before ';' token
  127 |   size_t addr_min = UINTPTR_MAX;
      |                                ^
[4/8] Generating symbol file test/libexample-lib.so.p/libexample-lib.so.symbols
ninja: build stopped: subcommand failed.

fail to start VSCode Remote (Node) when using nushell

nushellis a modern terminal written in Rust.

When the user enables nushell, the Node for VSCode Remote cannot start. However, VSCode Remote's Node can start when the user enables zsh.

users.users."root" = {
  shell = pkgs.nushell;
};

cannot execute /root/.vscode-server/bin/6c3e3dba23e8fadc360aed75ce363ba185c497944/node
You are trying to run an unpatched binary on nixos, but you have not configured NIX_LD or NIX_LD_x86_64-linux. See https://github.com/Mic92/nix-ld for more details

I think it may be due to environment variables.

LD_LIBRARY_PATH is set within VSCode integrated terminal

I am using nix-ld to run vscode-server on a nixos machine. Which works quite well, thanks!

I realized that within VSCode's integrated terminal LD_LIBRARY_PATH is set. This wasn't an issue, since NIX_LD was set to the system's LD anyways so most binaries continued to work.

However, it all broke down when using an old nixpkgs pin within a direnv environment. All binaries fail to launch with errors like:

/nix/store/q29bwjibv9gi9n86203s38n0577w09sx-glibc-2.33-117/lib/libc.so.6: version `GLIBC_2.34' not found (required by /nix/store/sqhyhvf3qpnnj6xnb55kv46ckfjx2na8-gcc-11.3.0-lib/lib/libstdc++.so.6)

Unsetting LD_LIBRARY_PATH within the terminal fixes the error.

I asume that somehow vscode-server stores the LD_LIBRARY_PATH variable and sets it on all child shells. I have yet to find a non-hacky way to get rid of the behaviour. Is there a way from nix-ld's side, to prevebt LD_LIBRARY_PATH to be propagated to vscode-servers child processes?

Recent update breaks flake's module

After updating from 7d251c0c5adf6b9b003499243be257d0f130b3d6 to 3210f71f0171b491b88bbb099effc5d3c7ef99e9, I get this error:

error: The option `programs.nix-ld.package' does not exist. Definition values:
       - In `/nix/store/fcr15jsj9nrbpzsm0c4hpvd9gd9w8ay0-source/flake.nix': <derivation /nix/store/3x7p9ds58969q2cgvnfmpkmcz2wvcvad-nix-ld.drv>

I'm importing the flake module inputs.nix-ld.nixosModules.nix-ld and then setting programs.nix-ld.enable = true;.

NixOs on Mac M1 got confused between /lib and /lib64

As I was trying to help someone here to use nix-ld, it seems that on the Mac M1 hardware nix-ld creates a /lib folder while softwares try to launch softwares in /lib64. So I guess that the Mac architecture should be added to the list in https://github.com/Mic92/nix-ld/blob/c25cc4b5ec65f03c2bb96f927ae724a59c51d7ee/nix-ld.nix But as I'm not an expert I may misunderstood something.

Possible to enable nix-ld on a per-project basis?

I'm considering replacing my current usage of buildFHSEnv with nix-ld, but this would come with a UX tradeoff. Users of nix-ld must install it system-wide (please correct me if that's wrong) before it can be used in a shell.nix or flake.nix of a specific project. This makes it harder to isolate the dependency on nix-ld, as all collaborators (on NixOS) on a project would need to install nix-ld before they can nix-shell or nix develop.

Is it feasible to install nix-ld on a per-project basis like buildFHSEnv can?

Library in system-wide base libraries not found by executable from python venv

I was hoping to try out nix-ld to fix my issue of the numexpr 2.8.4 python package not finding libstdc++.so.6 (imported by tables 3.7.0, which is imported by a flask 1.1.4 app), but it seems nix-ld can't fix that issue either at the moment. NIX_LD_LIBRARY_PATH is set correctly to /run/current-system/sw/share/nix-ld/lib, and that also contains libstdc++.so.6, but I guess the relevant program is not using the correct linker or something else is going wrong.

LD_DEBUG=libs gets me:

     -- more ld_debug=libs find/search/init stuff --
     16974: find library=libstdc++.so.6 [0]; searching
     16974:  search cache=/nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/etc/ld.so.cache
     16974:  search path=/nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib     (system search path)
     16974:   trying file=/nix/store/9xfad3b5z4y00mzmk2wnn4900q0qmxns-glibc-2.35-224/lib/libstdc++.so.6
     -- python backtrace to the import of numexpr.interpreter --
     ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
     -- more ld_debug=libs fini stuff --

Since /run/current-system/sw/share/nix-ld/lib is nowhere to be found in those logs, this makes me believe that the paths in NIX_LD_LIBRARY_PATH are just being ignored, possibly due to the wrong linker being used.

I'm running flask run in a nix shell nixpkgs#python39 with a venv setup with that python, with dependencies installed via pip.

I'll try to fumble together a small example to reproduce this in the coming days, but maybe if this is an obvious mistake on my part, someone could already point it out to me.

Edit: I almost forgot to mention that the program runs just fine if I set LD_LIBRARY_PATH=/run/current-system/sw/share/nix-ld/lib, except that this breaks other stuff like fish and what not, so I can only set it as I start flask run