mic92 / envfs

Fuse filesystem that returns symlinks to executables based on the PATH of the requesting process.

Envfs

A fuse filesystem that dynamically populates contents of /bin and /usr/bin/ so that it contains all executables from the PATH of the requesting process. This allows executing FHS based programs on a non-FHS system. For example, this is useful to execute shebangs on NixOS that assume hard coded locations like /bin or /usr/bin etc.

Demo

Mount envfs on /usr/bin sudo envfs /usr/bin

Programs are magically available, based on the PATH of the calling process. $ /usr/bin/env --version

env (GNU coreutils) 9.1
Packaged by https://nixos.org
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Richard Mlynarik, David MacKenzie, and Assaf Gordon.

$ echo $PATH

If the caller doesn't have the binary in the PATH, it will fail. $ PATH= /usr/bin/env --version 2>&1 || true

As you can see, /usr/bin is empty. $ ls -la /usr/bin

total 1
drwxr-xr-x 3345300086 root root 0 Jan  1  1970 .
drwxr-xr-x          3 root root 3 Sep  5 13:04 ..

By default, binaries are only available whenever the calling process executes or open a program, not when using stat or listing the directory: $ ls -la /usr/bin/env 2>&1 || true

ls: cannot access '/usr/bin/env': No such file or directory

This behaviour can be overridden by setting ENVFS_RESOLVE_ALWAYS=1. $ ENVFS_RESOLVE_ALWAYS=1 ls -la /usr/bin/env

lr----x--t 1 root root 0 Jan  1  1970 /usr/bin/env -> /nix/store/4vjigg3pr8bns6id4af51mza5p73l9lx-coreutils-9.1/bin/env

In conclusion, combined with the usual Nix wrappers or nix-shells, it makes things magically work!

Installation in NixOS

Choose one of the following methods:

In NixOS starting with 23.05 (Current recommendation)

Since NixOS 23.05 you can enable envfs with a single line:

{
  services.envfs.enable = true;
}

Flakes

If you use nix flakes support:

{
  inputs.envfs.url = "github:Mic92/envfs";
  inputs.envfs.inputs.nixpkgs.follows = "nixpkgs";
  
  outputs = { self, nixpkgs, envfs }: {
    # change `yourhostname` to your actual hostname
    nixosConfigurations.yourhostname = nixpkgs.lib.nixosSystem {
      # change to your system:
      system = "x86_64-linux";
      modules = [
        ./configuration.nix
        envfs.nixosModules.envfs
      ];
    };
  };
}

niv

First add it to niv:

$ niv add Mic92/envfs

Then add the following to your configuration.nix in the imports list:

{
  imports = [ "${(import ./nix/sources.nix).envfs}/modules/envfs.nix" ];
}

nix-channel

As root run:

$ nix-channel --add https://github.com/Mic92/envfs/archive/main.tar.gz envfs
$ nix-channel --update

Then add the following to your configuration.nix in the imports list:

{
  imports = [ <envfs/modules/envfs.nix> ];
}

fetchTarball

Add the following to your configuration.nix:

{
  imports = [ "${builtins.fetchTarball "https://github.com/Mic92/envfs/archive/main.tar.gz"}/modules/envfs.nix" ];
}

or with pinning:

{
  imports = let
    # replace this with an actual commit id or tag
    commit = "f2783a8ef91624b375a3cf665c3af4ac60b7c278";
  in [ 
    "${builtins.fetchTarball {
      url = "https://github.com/Mic92/envfs/archive/${commit}.tar.gz";
      # replace this with an actual hash
      sha256 = "0000000000000000000000000000000000000000000000000000";
    }}/modules/envfs.nix"
  ];
}

Build and run from source

$ nix-build
$ sudo ./result/bin/envfs -o bind-mount=/bin /usr/bin

envfs's Issues

Deadlock on stat when PATH=/bin

When I run envfs as follows:

sudo strace -e file envfs -o debug -f /bin

and in another terminal:

PATH=/bin sh -c uname

The result is a deadlock, which can be seen in the strace output produce by the first command above:

mount("envfs", "/bin", "fuse", MS_RDONLY|MS_NOSUID|MS_NODEV, "fd=3,rootmode=40755,user_id=0,gr"...) = 0
DEBUG - FUSE(  2) ino 0x0000000000000000 INIT kernel ABI 7.39, capabilities 0x73fffffb, max readahead 131072
DEBUG - INIT response: ABI 7.8, flags 0x1, max readahead 131072, max write 16777216
DEBUG - FUSE(  4) ino 0x0000000000000001 GETATTR
DEBUG - FUSE(  6) ino 0x0000000000000001 GETATTR
DEBUG - FUSE(  8) ino 0x0000000000000001 GETATTR
DEBUG - FUSE( 10) ino 0x0000000000000001 LOOKUP name "sh"
openat(AT_FDCWD, "/proc/92512/environ", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/proc/92512/syscall", O_RDONLY|O_CLOEXEC) = 4
statx(4, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0400, stx_size=0, ...}) = 0
openat(AT_FDCWD, "/proc/92512/mem", O_RDONLY|O_CLOEXEC) = 4
statx(AT_FDCWD, "/bin", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL,  <unfinished ...>) = ?

The reason is this stat call, which tries to prevent recursion. I guess that previously, envfs was multi-threaded and usually there was another thread which could handle that stat request, but now envfs is single-threaded and cannot handle the request recursively.

I know that somewhere it's written not to set PATH to /bin, but unfortunately that's what some software does on its own and envfs should survive that. The problem is that after the deadlock happens, everything using /bin/sh or /usr/bin/env blocks as well.

readme: recommend enabling envfs service

Since envfs is in nixpkgs now (congrats!), it would probably be good to change the recommend installation method in the readme to be installing from nixpkgs by enabling the service. It's only available in nixos unstable, so it's still worth mentioning other methods, but I'm sure many people would appreciate knowing there's a service too.

Loading module via flake duplicates fallback-path and doesn't work as expected

I added the module as described here to my flake and the fallback-path mount option is now duplicated.

none /usr/bin envfs fallback-path=/nix/store/hiyvx1mfzah7lqr8bphyys7n08j36z8x-fallback-path,nofail,bind-mount=/bin,fallback-path=/nix/store/jzdimbli3q7fba1izbxnrpn3qq39s05x-fallback-path,nofail 0 0

Disabling it via disabledModules = [ "tasks/filesystems/envfs.nix" ]; works but then the options defined in nixos (https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/tasks/filesystems/envfs.nix#L25-L50) are missing and I can no longer use envfs in wsl.

shebang interpreter cannot open it's file

Recently I tried to open "home-manager" in "envfs". However, looks like we correctly invokes "bash" interpreter, but it cannot find "/usr/bin/home-manager" source file in /usr/bin/.

$ /usr/bin/home-manager
/nix/store/j80wcbm6bb1nsqsgmniq43rg980p27q9-bash-5.2-p15/bin/bash: /usr/bin/home-manager: No such file or directory

Hmm, recently I made duplicated work on my own repo before finding out this one :( based on a different approach: libfuse3 which does not seem to have this issue.

https://github.com/inclyc/pathfs

Edit: related to "stat" operation?

fuse doesn't work

Hello! Thank you for your project, however I wasn't able to get it to work properly. I installed envfs on stable 23.11 NixOS using services.envfs.enable = true;, haven't tried it on unstable, because the module appears to be the same.

$ /bin/sh
fish: Unknown command. '/bin/sh' exists but is not an executable file.
$ la /bin
"/bin": Transport endpoint is not connected (os error 107)

Even though it is mounted mount|grep /bin:

none on /usr/bin type fuse.envfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)
none on /bin type fuse.envfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Rebooting doesn't help.
After doing sudo umount /bin:

# RUST_LOG=trace envfs /bin -o debug
DEBUG - spawn worker
DEBUG - spawn worker
DEBUG - spawn worker
DEBUG - spawn worker
DEBUG - FUSE(  2) ino 0x0000000000000000: INIT kernel ABI 7.39, flags 0x73fffffb, max readahead 131072
DEBUG - INIT response: ABI 7.19, flags 0x150049, max readahead 131072, max write 131072
// trying to cd /bin
DEBUG - FUSE(  4) ino 0x0000000000000001: GETATTR
DEBUG - FUSE(  6) ino 0x0000000000000001: GETATTR
DEBUG - FUSE(  8) ino 0x0000000000000001: GETXATTR name "system.posix_acl_access", size 4096
DEBUG - FUSE( 10) ino 0x0000000000000001: OPENDIR flags 0x8000
ERROR - Unknown FUSE opcode (52)
INFO - Unmounted
// trying to ls /bin
DEBUG - FUSE( 14) ino 0x0000000000000001: GETATTR
ERROR - Failed to send FUSE reply: Bad file descriptor (os error 9)
INFO - Unmounted
ERROR - Unknown FUSE opcode (52)
INFO - Unmounted
DEBUG - FUSE( 13) ino 0x0000000000000000: INTERRUPT unique 12
ERROR - Failed to send FUSE reply: Bad file descriptor (os error 9)
INFO - Unmounted

journalctl:

systemd-coredump[5972]: [🡕] Process 5927 (envfs) of user 0 terminated abnormally without generating a coredump.

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.10-xanmod1, NixOS, 23.11 (Tapir), 23.11.20240122.d7f206b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `""`
 - nixpkgs: `/nix/store/2xgpqy6dyicqnhwym6nnaysd8mzrwkr8-source`

Bump version automatically

to have to avoid bumping https://github.com/Mic92/envfs/blob/main/default.nix#L5 by hand

Allow remounting to update fallback-paths

When updating fallback-path the fs must be remounted because usually the nix store path has changed but currently envfs is not implementing that:

 ▶ sudo mount -o remount /usr/bin
Ignoring remount request.

Could this project make the default PATH usable?

I'm curious if this project could somehow make the default PATH work like it does on FHS systems? My exact issue is that I'm using Bazel to build projects and Bazel executes commands where the PATH is not set.

On a normal FHS system the default PATH would be used which would be something like /usr/local/bin:/usr/bin but on NixOS the default PATH is set to /no-such-path.

If this does not fit into the scope of this project then feel free to close the issue!

Demo in the README is outdated

This command from the README:

$ ls -l /usr/bin/{bash,python}
lr----x--t 0 root  1 Jan  1970  /usr/bin/bash -> /nix/store/j37555sj2w3xsql3f8qrwbaim7pv67hg-bash-interactive-4.4-p23/bin/bash
lr----x--t 0 root  1 Jan  1970  /usr/bin/python -> /home/joerg/.nix-profile/bin/python

now results in:

ls: cannot access '/usr/bin/bash': No such file or directory
ls: cannot access '/usr/bin/python': No such file or directory

because envfs only responds lookups from execve syscalls.

On a related note - what is the reason to only respond to execve lookups? I guess it's to minimize the filesystem polution. I now work with a project that not only has shebangs pointing to /usr/bin, but also checks for interpreter presence in /usr/bin before running the scripts. Having envfs responding to all lookups as before would help me a lot in this case. What about responding all lookups if some variable (e.g. ENVFS_LOOKUP_ALL) is set in the requesting process environment? I may try to send a PR if you're not against.

Breaks many, many services

I've tried the latest update and it broke ssh, Getty, phpfpm. Since I didn't have any way to access the machine in that state I couldn't collect any logs. journalctl -b -1 also didn't contain anything I could find at least closely related to this. I have no real idea how to help to fix this. I think running nixos VM tests with envfs enabled could help making sure less regressions like this happen.

ENVFS_RESOLVE_ALWAYS=1 only works for entries in the fallback-path option

 ▶ ENVFS_RESOLVE_ALWAYS=1 ls -la /usr/bin/env
lr----x--t root root 0 B Thu Jan  1 01:00:00 1970  /usr/bin/env ⇒ /nix/store/mqvnklaf6mzmj7ng13aph5ymkvzpyhn0-fallback-path/env

 ▶ ENVFS_RESOLVE_ALWAYS=1 ls -la /usr/bin/systemd-detect-virt
lsd: /usr/bin/systemd-detect-virt: No such file or directory (os error 2).

 ▶ ENVFS_RESOLVE_ALWAYS=1 ls -la /usr/bin/bash
lsd: /usr/bin/bash: No such file or directory (os error 2).


 ▶ ENVFS_RESOLVE_ALWAYS=1 ls -la /usr/bin/cat
lsd: /usr/bin/cat: No such file or directory (os error 2).


 ▶ ENVFS_RESOLVE_ALWAYS=1 ls -la /usr/bin/sh
lr----x--t root root 0 B Thu Jan  1 01:00:00 1970  /usr/bin/sh ⇒ /nix/store/mqvnklaf6mzmj7ng13aph5ymkvzpyhn0-fallback-path/sh

 ▶ ls /nix/store/mqvnklaf6mzmj7ng13aph5ymkvzpyhn0-fallback-path
 env   login   mount   sh   wslpath

The Cargo.lock file is missing for the build

Hi! :)

The import fails to build the derivation because it fails to locate the Cargo.lock. Do you know how to fix that?

ERROR: The Cargo.lock file doesn't exist

Cargo.lock is needed to make sure that cargoSha256 doesn't change
when the registry is updated.

builder for '/nix/store/l1w9wp7whiy7fjrics2b2wb0ha1bbrly-envfs-0.0.1-vendor.tar.gz.drv' failed with exit code 1
cannot build derivation '/nix/store/w4v3rf1na2ddqmkfh68y5y3zxry3kdy2-envfs-0.0.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/lrn0mflw7mjnlyyjrfgzvc5ym2lv80i8-system-path.drv': 1 dependencies couldn't be built

`services.envfs.enable = true;` breaks nixos-install

If I enable envfs (either via the nixos configuration, or via flake as per the current instructions in the README), right at the end of the installation process while installing the bootloader I get:

updating GRUB 2 menu...
Can't exec "/bin/sh": No such file or directory at /nix/store/kdlihrdxakv00jm07vqix8svk2wssdcq-install-grub.pl line 51.
Failed to execute: /nix/store/56kvc3cxxwcxy84rfbgmvafnhvibz9md-util-linux-2.38.1-bin/bin/blkid -o export /dev/nvme1n1p2

This does not appear to happen if I disable envfs, starting with a fresh machine each time.