gytis-ivaskevicius / flake-utils-plus Goto Github PK

Needed for this: nix-community/nixos-generators#110
And can serve as a temporary workaround for this: NixOS/nixpkgs#101475

I'm speaking purely on behalf of myself, but I would find it excitinh if this can be brought into a shape that it will be easy for devos to built upon.

I see the great powers that both projects combined could yield out of flakes and for its users.

I sense devos is at a moment where it would wish to consolidate its library and declare concise and reusable interfaces.

It looks like you (coming from the other end of flake-utils) already have gone down that route.

Some time ago, I tried to achive something similar myself with numtide/flake-utils#18. But the whole PR and it's shape perfectly reveals where I stand in terms of nix-skills 😉

So I would love to see a good bunch of collaboration striving for getting this and devos, both in a shape where devos could adopt / inherit this library without braking any features.

I think for devos it would be good / convenient, until reaching 1.0 to have such a well-defined (and well-guarded) library in-tree.

@gytis-ivaskevicius what do you think?

@nrdxp what do you think?

@Pacman99 what do you think?

Shall we add something along the lines of this by default to sharedOverlays so people could easily define src = inputs.someGitRepo; in their packages

final: prev: { inherit inputs; }

I got something similar setup https://github.com/gytis-ivaskevicius/nixfiles/blob/master/flake.nix#L140 - obviously, this would crash if consumers do not have something like that already defined. But then again - that's probably the best way to pass inputs down to overlays

To promote the fup user community's sharing model, exporting sane packages derived from a user overlay, I believe, is in order.

That way, other users will have a light time to reuse custom packagings from other fup users via flake-native mechanisms.

numtide/flake-utils#27 lays the foundation to filter packges generically. It would be left to fup to extract multi instantiated packages out of self.overlay.

devos might lend an inspiration on how to do that (mkPackages)

/cc @Pacman99

I have no idea why fup does not implement just a single builder for everything - but now that I look at it - it's obvious that this would be quite a bit nicer way of handling things.

Possible issues:
Merging logic, it's always has been a pain. Tho if I recall correctly existing mergeAny function should just work out of the box

Expected outcome:

utils.lib.systemFlake {

  exports = channels: { 
    packages = { 
      inherit (channels.nixpkgs) hello coreutils neovim; 
    };
    
    apps = {
      neovim = mkApp {
        drv = channels.nixpkgs.neovim;
        exePath = "/bin/nvim";
      };   
    };

    checks = { 
      inherit (channels.nixpkgs) hello coreutils neovim; 
    };
    # ... etc ...
  };

}

@blaggacao @Pacman99 Looks good?

Should, I assume, be: else if (builtins.isAttrs arg) && (!(hasFileAttr arg)) then

generateNixPathFromInputs symlinks relevant inputs into /etc/nix/inputs and synthesizes a NIX_PATH with an entry for each input. I assume it does this so that way the NIX_PATH entries will always reference the current configuration without having to spawn a new shell. The downside is any additions/removals to the input set won't be reflected in existing shells.

Instead of adding one entry per input to NIX_PATH, it should just add a single entry for /etc/nix/inputs. This way it will always reflect the current state of the system regardless of what happens to the inputs.

however, #84 introduced a regression, where
builtins.pathExists in the new pathIsDirectory requires a string coercible: functions are not.

/cc @fufexan

If I have;

mkFlake {
  hosts.foo = {
    modules = [ ./hosts/foo.nix ];
  };
}

and then in foo.nix I have;

{
  networking.hostName = "bar";
}

I get an error;

error: The option `networking.hostName` has conflicting definition values:
          - In `/nix/store/path/to/foo.nix`: "bar"
          - In `<unknown-file>`: "foo"

There is (seemingly?) an attempt to support explicit hostName specification;

The new-ish way of defining overlays is like so;

{
   overlays.default = final: prev: {...};
}

but using overlaysBuilder (seemingly) doesn't make it easy to then use overlays defined this way.

channels.nixpkgs.overlaysBuilder = channels: [
  inputs.some-overlay.overlays.default  # doesn't work; no such attribute
];

I can't seem to find any documentation from devos side of things on these exporters - the only docs there are part of the code - I think usage example should be added and double-checking formatting is kinda needed (there are some weird comment indentations)

@blaggacao @Pacman99 - Does devos have some docs on this that I could steal that I have not noticed? (Of course in nice, legal, MIT manner :D )

If not - It would be great if you guys could take care of it since you guys built em

Also, there might be a bug (or maybe a missing feature) with fromOverlays exporter. Need to do some investigation. Not related to this issue

upstream states:

{
                , # This would be remove in the future, Prefer _module.args option instead.
                  args ? {}
}

at the reasoning of

/* Evaluate a set of modules.  The result is a set of two
     attributes: ‘options’: the nested set of all option declarations,
     and ‘config’: the nested set of all option values.
     !!! Please think twice before adding to this argument list! The more
     that is specified here instead of in the modules themselves the harder
     it is to transparently move a set of modules to be a submodule of another
     config (as the proper arguments need to be replicated at each call to
     evalModules) and the less declarative the module set is. */

It travels via this nixos waypoint.

Unless, I'm misunderstanding something, that is.

overlayNames calls the overlay attribute with null null for the arguments final and prev. This works if the overlay attribute looks like:

{
  overlay = final: prev: {
    blah = prev.callPackage ./blah {};
    foo = prev.callPackage ./foo {};
  };
}

because nix is lazy, so it will not even evaluate the attribute values.

However, if the overlay looks something like:

{
  overlay = final: prev:
    prev.lib.genAttrs 
      (builtins.readDir ./packages) 
      (pkgName: _: prev.callPackage ./packages + "/${pkgName}");
}

Then the overlay needs access to prev.lib before the set of names is known, and so overlayNames errors with value is null while a set was expected

For reference, this is where overlayNames is defined: https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/lib/exportOverlays.nix#L63

As stated here #7 (comment), we need an interface to be able to extend lib.nixosSystemwhile retatining the environment (pkgs, hostname, extraArgs, etc) from the genericConfigurationsBuilder.

It looks like

Provides ability to solve #7 downstream.

/cc @Pacman99

After adding a patch to channels.nixpkgs.patches, I get the following error:

error: a 'aarch64-linux' with features {} is required to build '/nix/store/is49g8n6gwyk91aa7pq6n1bnfy7pynmv-nixpkgs-patched-83667ff.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

If I then restrict supportedSystems to only include x86_64-linux, it works just fine on this machine. However, that's not an option in my case because I still need to support an aarch64-linux host.

What is the point of an appsBuilder for system flakes? Is that remeniscent of an analogy with flake-utils or has it a use case, which I'm not aware of / have not explored yet?

I am using std to configure my repository and want to avoid using overlays where feasible. To that end, I've been using the packages output that most overlays expose (see emacs-overlay as a good example). Would it be possible for fup-repl to be made available this way as well?

https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/modules/saneFlakeDefaults.nix#L1-L18

Another little thing to simplify implementation for end users.
What needs to be done:

• Make sure that everything is easily overwritable
• Make sure that it does not conflict with non-nixos systems

Also, I notice that we have this logic duplication 🤔
https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/staging/src/systemFlake.nix#L157
https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/modules/saneFlakeDefaults.nix#L15

Related:
#60

If I understood it correctly, outputsBuilder is a map fn from system to outputs, but the most crucial information, which system it is, is hidden from it.

Use case:

I want to use my-flake.stuff.${system} in outputsBuilder.

Now, there is NixOS/nixpkgs#120141.

We need to wait and observer a little, but let's backlog this for consideration.

When doing remote builds with flakes, sometimes it's not obvious which host is being built. --flake . might build the wrong host. The fix for this is usually just to use --flake .#host, however, if you nixos-rebuild switch, it might have already switched to the wrong configuration.

In the past, before using flake-utils-plus, i just had builtins.trace "building ${name}" <actual builder>, which did the job.

Even though I think this might be a good idea, others may disagree. So it would be nice to either be able just opt-in to this, or just be able to setup a default builder, and not have to define a builder for each host.

I'm trying to get cross-compilation working but without any luck so far. I tried every incantation of buildPlatform, hostPlatform, localSystem, crossSystem in either channels config or host's config, but I just can't get it to work.
In particular:

{
  nixpkgs.buildPlatform = { config = "x86_64-unknown-linux-gnu"; };
  nixpkgs.hostPlatform = { config = "aarch64-unknown-linux-gnu"; };
}

This should cross-compile according to the nixpkgs documentation, but it doesn't.

Also this (unsure what the difference to above is supposed to be):

{
  nixpkgs.localSystem = { config = "x86_64-unknown-linux-gnu"; };
  nixpkgs.crossSystem = { config = "aarch64-unknown-linux-gnu"; };
}

Additionally, setting pkgs seems to have zero effect when using fup:

{
  nixpkgs.pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform
}

I also tried it in channels config:

{
    channels.cross = {
      input = nixpkgs; # input = nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform; doesn't work
      pkgs = nixpkgs.legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform; # ignored?
      localSystem = { config = "x86_64-unknown-linux-gnu"; }; # ignored?
      crossSystem = { config = "aarch64-unknown-linux-gnu"; }; # ignored?
  };
}

Trying to read the code it looks like on is merely able to influence nixpkgs.config from the flake level, however all these options go into options directly under nipkgs.

Do you know of anyone successfully using fup for aarch64?

My aarch64 hosts either wrongly build as x86_64, or use qemu to build. The latter is extremely slow to a point it's unusable, and stopped working alltogether recently trying to build "grafana" (just stops doing anything without load).
Any pointers?

nixosProfiles do contain system profiles, in a sense, but more accurately they contain host configs.

This clashes with devos' usage of the term profiles which is reserved for a very special meaning:

• profiles cannot declare options, but only make use of them. (while anything that declares options are modules)
• this rule makes modules a lot more reusable among the community

With fup reserving the name, we might have a problem finding an alternative naming scheme. (and also it's somewhat established, already)

/cc @Pacman99

Looks like this script does not fetch reproducible nix or sed from pkgs.

When I run nixos-rebuild I get these warnings:

trace: warning: The args argument to evalModules is deprecated. Please set config._module.args instead.
trace: warning: The check argument to evalModules is deprecated. Please set config._module.check instead.

I checked nixpkgs and found that the deprecation has already hit stable, so there's probably no reason to wait to fix it.

This is effectively an implementation of an importer and it is of undisputed handy nature.
Are other (generic!) importers potentially in scope?

/cc @Pacman99

It is not entirely clear or self evident why fup should provide this service.

Sure, it is nice to have, but it is also only a // nixosConfigurations away in the top level flake.

/cc @Pacman99

This is more of a question than an issue.
@blaggacao @Pacman99 - if we were to compare master vs staging branches - currently NIX_PATH implementation does not exist there. (https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/modules/saneFlakeDefaults.nix)

I am considering re-adding it with nixpkgs definition only. What are your guy's thoughts on it? And how should it be implemented?

Note: I'd like to avoid linking nixpkgs via environment.etc

Related: #5

Similar to #96, so feel free to close this if you feel it is a duplicate.

I have been trying to move my macOS system to nix-darwin, starting with home-manager for dotfiles and ran into a similar problem as the aforementioned issue when adding a package (which does not export an overlay);

channels.nixpkgs.overlaysBuilder = channels: [
  (final: prev: {
    emanote = inputs.emanote.defaultPackage.${prev.system};
  })
  ...
];

See here for full context.

This results in the same error as #96;

$ nix flake show     
...
error: a 'x86_64-darwin' with features {} is required to build '/nix/store/path/to/some.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

The error occurs regardless of the current platform, e.g. "x86_64-linux ... but I am a x86_64-darwin" if on macOS

Just a simplification for a very common use case. Should not be complicated to implement

https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/staging/src/systemFlake.nix#L182-L193

I originally had this issue in divnix/digga#339 and thought perhaps I could avoid it with flake-utils-plus since there are less layers of abstraction.

However, it appears that flake utils plus also has the issue or the problem has been me making some mistake all along 😅

Here was my attempt to add emacs with that external package. Note that you can reproduce by trying to build that repo at that commit.

Thoughts on adding a deployment part to the utils? Ideally this would work across multiple deployment tools, but the one I use is deploy-rs so I'll roll with that specific example, though it could also be extended to nixops, etc. My thought is a flake along the lines of:

hosts.HostnameExample = {
        ....
    deploy-to = {
    type = "deploy-rs";
    hosts = [ "101.101.101.101"; "101.101.101.102"; ];
  };
};

The idea is that FUP builds the attributes and checks used by deploy-rs if the deploy-to set is not empty. The big question in my mind is: is this out of scope for this project? And is there demand for this feature? Another issue is this might be oversimplifying deployment. However, this would let the user take the equivalent utility functions out of their flake.nix. As far as I can tell, this is the entire point of FUP. And I think it hits the sweet spot of use cases for deployment tool users: "I want to deploy my flake hosts to my server cluster."

On the implementation side, all this would need to do is generate a bunch of deploy.nodes attributes, similarly to what I've done here. I could see FUP picking sane defaults, such as fastConnection = false and user=sshUser="root" which could be overridden.

Does this fits in the vision for FUP?

Demo flake.nix https://gist.github.com/azuwis/47a67c40f61afaa02362b79a0496afeb:

{
  description = "Darwin demo";

  inputs = {
    nixpkgs.url = github:nixos/nixpkgs/nixpkgs-unstable;
    utils.url = "github:gytis-ivaskevicius/flake-utils-plus/v1.3.1";

    darwin.url = "github:lnl7/nix-darwin/master";
    darwin.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = inputs@{ self, nixpkgs, utils, darwin, ... }:
    utils.lib.mkFlake {
      inherit self inputs;

      hosts.demo.system = "aarch64-darwin";
      hosts.demo.output = "darwinConfigurations";
      hosts.demo.builder = args: darwin.lib.darwinSystem (removeAttrs args [ "system" ]);
    };
}

Error log:

$ nix build 'git+https://gist.github.com/azuwis/47a67c40f61afaa02362b79a0496afeb?ref=main#darwinConfigurations.demo.system'            
trace: warning: literalExample is deprecated, use literalExpression instead, or use literalDocBook for a non-Nix description.
trace: lib.crossLists is deprecated, use lib.cartesianProductOfSets instead
trace: warning: literalExample is deprecated, use literalExpression instead, or use literalDocBook for a non-Nix description.
trace: warning: lib.mkFixStrictness has no effect and will be removed. It returns its argument unmodified, so you can just remove any calls.
trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
trace: warning: lib.readPathsFromFile is deprecated, use a list instead
trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
trace: lib.zip is deprecated, use lib.zipAttrsWith instead
error: The option `_module.args.inputs' is defined multiple times.

       Definition values:
       - In `/nix/store/5m88sbxj815z8bzrij15bp4w4siakw1i-source/eval-config.nix'
       - In `<unknown-file>'
(use '--show-trace' to show detailed location information)

$ nix-info -m    
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.3.0, macOS 12.2.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.6.1`
 - channels(azuwis): `""`
 - channels(root): `""`
 - nixpkgs: `/nix/store/a94a18pdxp29hggy2v5aij4z11h54y8p-source`

Hi, thanks for this project!

I wanted to resort to a default package from the default channel, if some predicate on the channel is satisfied.
Using the channels argument to mkFlake, it is only possible to select the channels by name, while the function sharedOverlays doesn't have the channels argument of overlaysBuilder, so I didn't know how to access the default channel.

Wouldn't it be more consistent and more flexible to add the channels argument to sharedOverlays, and letting it appear as just another overlaysBuilder to the user?

The solution I used is to specify channels by name (https://github.com/jorsn/ancienTeX.nix/blob/707118d0f2202b016dcdcd5cff4e74877db0986b/flake.nix#L82), but I'd like to have it based on a predicate.

I've been attempting to use systemFlake as it cleans up my messy flake. However I have all my laptops on nixos-unstable while I have a server on nixos-20.09 and the server system doesn't build because the unstable uses package util-linux somewhere while only utillinux exists in 20.09.

Now I noticed the comment at https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/systemFlake.nix#L62. What is the reasoning this isn't possible?

It would fill my need perfectly if this would become possible.

Thanks for this project.

If I set nix.generateNixPathFromInputs = true it will symlink my inputs into /etc/nix/inputs even if I also set nix.linkInputs = false. This configuration should be considered an error instead (e.g. using an assertion).

Also, I would suggest that setting nix.generateNixPathFromInputs = true should actually change the default value of nix.linkInputs, so that way nixos-option nix.linkInputs shows the right value.

I'm thinking of writing something similar to hosts, but for standalone Home Manager. I believe it's not just me who would like to see this go through, as the difference between configuring nixosConfigurations and homeManagerConfigurations is a bit stark at the moment.
Would this benefit the project or is it something better suited in the user's personal lib?

The current default for experimental-features enables ca-derivations, which enables one to build content addressed derivations, but not to have them as inputs to other derivations, regardless of them beeing CA or IA.

Therefore please also enable ca-references by default.

Changes required:

• Make nixosConfigurations optional
• Turn it into a tiny overlay that is added by default. Maybe package name should be fup-repl, exec name repl? Any thoughts?
• CLI behavior:

1. $ repl - opens pretty much existing repl
2. $ repl /path/to/flake.nix - opens repl with specified flake loaded (I am really missing this feature :D )

More or less fixes divnix/digga#17

I have some infrastructure in my nixos config to allow only specific unfree packages (to avoid accidental unfree use). Therefore I need to set nixpkgs.config.allowUnfreePredicate in my config, but the value it is set to doesn't seem to work anymore.

As per NixOS/nixpkgs#139075 (comment), there is now an error in nix.conf validation, courtesy: https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/master/lib/mkFlake.nix#L175 . As a workaround one has to disable nix.conf configuration check.

Thanks!

Currently the experimental-features setting for nix is hardcoded to a specific set of items.

https://github.com/gytis-ivaskevicius/flake-utils-plus/blob/staging/lib/systemFlake.nix#L198-L200

This makes it hard to change the values, eg. to add recursive-nix or ca-references to the list, or any other not yet known experimental feature.

Therefore, it might be sensible to change the current implementation in one of these ways:

1. Use mkDefault instead, then a user can easily overwrite the extraOptions for nix and does not have to use mkForce, which again might even break other stuff, as eg. the hydra options wouldn't be able anymore to add their required settings, while at the same time , enabling hydra would disable any mkDefault enabled stuff.
2. Provide an option, nix.experimentalFeatures, which can be set instead, and merges from individual strings to "string concatenated with spaces" (types.separatedString " "). This should be more compatible with other modules that touch nix.extraOptions.

What is the rational of defaulting to inputs rather than have people make use of inputs in their top level flake?

The inputs thought model as extraArgs neglects the "receptionist" thought model of the flake.nix.
I think flake.nix as receptionist is an important feature of the distributed future of flakes, "to show new guests around": Hans visit's James' repo on github. flake.nix gives him the map and tells at least the outline of the story.

What cases cannot be covered by receiving inputs in flake.nix and dispatching eg overlays via input.otherflake.overlay or inputs.otherflake.nixosModules?

I think it was not an afterthought to make those two public static api of flakes themselves, in the first place by the designers of flakes.

One thing that comes to my mind ...

In a sharing model based on overlay it is not always trivial to pull in the following:

let
   bob = {
     kakoune = {};
     chrome = {};
   };
   alice = {
     kakoune = {};
     chrome = {};
   };
in
[
  bob.kakoune
  alice.chrome
]

flake-utils.lib.simpleFlake solves this by implicitly doing a namespacing of overlays.

Should we really go down that route and just globally pass through inputs, by default?
→ What this does not do is expose and promote encapsulation via the top level flake api.

/cc @Pacman99 @nrdxp

probably confilcts with https://github.com/divnix/quick-nix-registry/blob/e566b3d61ba91f8d797d309cf2a93b3c0ce9df5e/flake.nix#L27

I don't know if this actually manifests, but it looks like things are crossing at the wrong angle, here...

I'd have to think more about it, but wanted to kick-off a discussion while these changes are still fresh.

/cc @nrdxp

If the system is updated while a user is logged in, their NIX_PATH variable can get out of date when using the saneFlakeDefaults. This can get noticeable if one updates through a full rebuild on a graphical system, resulting in graphical applications accessed through a nix-shell refusing to start. I solved this this way in my flake: https://github.com/chvp/nixos-config/blob/f3a7f168d5bc46fb5d9c7946b59bed327cf1e609/flake.nix#L28-L33 (the .version-suffix file is also useful, nix-direnv relies on it to do its caching).

nix.generateNixPathFromInputs only adds inputs to NIX_PATH if they have a packages or legacyPackages output. This is a rather curious restriction, as the existence of such a package set does not prove that the flake source can be imported directly. It also means that flakes that can be imported, such as nix-darwin, don't end up in NIX_PATH as they don't have a package set.

Instead of testing for the package set, it should check if the flake has a default.nix file. Unfortunately I don't think Nix lets you test if a file exists, so this might require using an activation script instead of config.environment.etc. Though perhaps setting environment.etc."nix/input".source to a derivation whose outPath is a directory of symlinks might work? That way the derivation build script can test the inputs for a default.nix.

Making an .iso file (or any other format) out of currently running system can be helpful.
Especially in case where one wants to use it as on-the-go system and carry their configs and apps everywhere with them. Useful thing when you're forced to work not on your hardware.
Currently, seems like there is such way to do so with nixos-generators. It would be great to integrate such features to your project as well, instead of rewriting flake.nix just for the sake of having an .iso from current system.

currently, nixosConfigurationsGenerator are implemented, however a typical flake based nixos or — generalized — nix controlled environment (machines, phones, iot, routers, etc), further generators are in order.

I suggest the canonical way to implement additional generators is through accessing self:

{
  outputs = inputs@{ self, nixpkgs, unstable, nur, utils, home-manager, neovim }:
    utils.lib.systemFlake { "..." = "..."; }
    // { 
        hmConfigurations = hmConfigigurationsGenerator self.nixosConfigurations;
        deploy.nodes = deployConfigurationsGenerator deploy self.nixosConfigurations;
    };
}

Or do we need an optional lib of generators to support well-known upstream projects? Since it's not called flake-utils-nixos, maybe there is a good argument to make about this.

/cc @Pacman99