nixos / nixops-aws

NixOps AWS Plugin

NixOps (formerly known as Charon) is a tool for deploying NixOS machines in a network or cloud.

This repo contains the NixOps AWS Plugin.

• Manual
• Installation / Hacking
• Continuous build
• Source code
• Issue Tracker
• Mailing list / Google group
• Matrix - #nix:nixos.org

Developing

To start developing on the NixOps AWS plugin, you can run:

  $ nix-shell
  $ poetry install
  $ poetry shell

To view active plugins:

nixops list-plugins

Documentation for the nixops-aws plugin is in the nixops repo

The python code is formatted with the latest release of black and is checked in PR validation. See the black target in ci.yaml for the cmd to run.

Building from source

You can build the Nix package by simply invoking nix-build on the project root:

$ nix-build

See the main NixOps repo for more up-to-date instructions for how to built NixOps with this AWS plugin.

nixops-aws's Issues

deployment.ec2.subnetId ignored

Hi,

i am using a non-default-VPC and specified the correct subnet-id for it.
I get the error:
error: EC2ResponseError: 400 Bad Request

MissingInputNo subnets found for the default VPC 'vpc-meh'. Please specify a subnet.a662bdee-b2eb-4c39-b28d-7d8a41a98cef

my testing .nix files:
http://pastebin.com/u7qEgDkF
http://pastebin.com/NBJ6eifW

EC2 resources dependent on config.nodes.mynode.config.networking.privateIPv4

I would like to define an EC2 resource, which depends on the IP address of a machine being deployed. I.e. config.nodes.mynode.config.networking.privateIPv4. Is that impossible or if not how can that be done?

The manual only contains overly simple examples where there is no interaction between the nodes being deployed and for example the security groups that should apply to it.

The use case is that I want two nodes to be able to talk to each other, but not to anything else.

EC2 security groups on EC2 classic fail with --check enabled

Error error: 'GroupOrCIDR' object has no attribute 'groupName'
which is thrown during deploy --check : https://github.com/NixOS/nixops/blob/master/nixops/resources/ec2_security_group.py#L133

NixOps does not clean up route53 DNS entry on destroy.

Critical error: impossible to do deployments on machines with multiple ENIs

I think this happened during the update of security groups on a machine with multiple ENIs (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html) attached:

<Response><Errors><Error><Code>InvalidInstanceID</Code><Message>There are multiple interfaces attached to instance 'i-<snip>'. Please specify an interface ID for the operation instead.</Message></Error></Errors>

Disassociate EIP from an instance in VPC fails

frontend> disassociating IP address ‘54.235.75.201’...
Traceback (most recent call last):
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/bin/..nixops-wrapped-wrapped", line 951, in <module>
    args.op()
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/bin/..nixops-wrapped-wrapped", line 379, in op_deploy
    repair=args.repair, dry_activate=args.dry_activate)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 990, in deploy
    self._deploy(**kwargs)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 947, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 920, in worker
    r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/backends/ec2.py", line 971, in create
    self._assign_elastic_ip(elastic_ipv4, check)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/backends/ec2.py", line 597, in _assign_elastic_ip
    self._conn.disassociate_address(public_ip=self.elastic_ipv4)
  File "/nix/store/n5irc6vd4wlf4g7ya3mii7f8819gci49-python2.7-boto-2.45.0/lib/python2.7/site-packages/boto/ec2/connection.py", line 2046, in disassociate_address
    return self.get_status('DisassociateAddress', params, verb='POST')
  File "/nix/store/n5irc6vd4wlf4g7ya3mii7f8819gci49-python2.7-boto-2.45.0/lib/python2.7/site-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>You must specify an association id when unmapping an address from a VPC instance</Message></Error></Errors><RequestID>5cda77cc-a1d9-4404-a95b-228ad8b4f5ca</RequestID></Response

Rate limits

NixOps doesn't cope with rate limits.

boto.exception.BotoServerError: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>

NixOps could output the amount of requests being done to the various APIs and produce a report on such failure which could then be sent to AWS support immediately.

In my case running the exact same command (nixops deploy) again worked.

NixOps could also try a bit harder. E.g. run three times in a row with 5 seconds in between before raising the white flag.

NixOps should warn if deployment.ec2.associatePublicIpAddress is not set

In a VPC deployment I forgot to set deployment.ec2.associatePublicIpAddress. Then NixOps will wait forever for a private IP address to become reachable:

btc....> creating EC2 instance (AMI ‘ami-d40185ad’, type ‘m5.large’, region ‘eu-west-1’)...
btc....> waiting for IP address... [pending] [pending] [running] None / 172.30.1.104
btc....> waiting for SSH..............................

Allow creating ec2 security groups without group owner id

According to the documentation, the parameter SourceSecurityGroupOwnerId (CTRL+F for it) is not required.

However, it appears that if you pass a sourceGroup.groupName to a security group (snippet below), that nixops just throws an error error: 'NoneType' object has no attribute 'get'

  resources.ec2SecurityGroups.zk-cluster-security-group-member = { 
    inherit region;
    rules = [ 
      { fromPort = 2888; toPort = 2888; sourceGroup.groupName = "zk-member"; }
      { fromPort = 3888; toPort = 3888; sourceGroup.groupName = "zk-member"; }
    ];  
  };  

AWS: cannot change key pair of an existing instance

NixOps should destroy the previous infrastructure and build the new infrastructure when

deployment.ec2.keyPair = "your-key-name";

has changed when --allow-recreate is set.

Uploading to S3 buckets

NixOS/nixops#173 discusses creating S3 buckets.

Can nixops also upload things into the buckets?

Add tests for iamRoles provisioning.

AWS VPC instances aren't assigned multiple security groups

With the following configuration, using either the security group names or sg-... IDs, the instance is only ever launched with the default security group. Same if I remove the spot bid.

  myserver = { resources, config, ... }: {
      deployment.targetEnv = "ec2";
      deployment.ec2.accessKeyId = accessKeyId;
      deployment.ec2.region = region;
      deployment.ec2.instanceType = "m1.small";
      deployment.ec2.keyPair = resources.ec2KeyPairs.graham_ndndx;
      deployment.ec2.instanceProfile = "machine-insight";
      deployment.ec2.spotInstancePrice = 1;
      deployment.ec2.securityGroupIds = [
        "insight" # "sg-..." # insight
        "default" # "sg-..." # default
      ];

Dynamic securityGroupIds

We can define resources.ec2KeyPairs and refer them in deployment.ec2. E. g.:

{
# ...
resources.ec2KeyPairs.petcatKey = { inherit accessKeyId; region = "ap-southeast-1"; };
# ...
petcat = { resources, config, pkgs, lib, ... }: {
#...
 deployment.ec2.keyPair = resources.ec2KeyPairs.petcatKey;
# ...
};
# ...
}

We can do the same for security groups, but only when not deploying to VPC. There is a difference between securityGroupIds which should be used for VPC and securityGroups for non-VPC deployments.

If one tries to write securityGroupIds = [ resources.ec2SecurityGroups.petcatSecGroup ]; an error will be raised:

error: The option value `deployment.ec2.securityGroupIds.[definition 1-entry 1]' in `/...' is not a string

Below are some scribbles to fix that. If this looks good, we can get rid of securityGroupIds and just use securityGroups for both VPC and non-VPC deployments. But i'm not vary familiar with nixops to make it the right way :-)

diff --git a/nix/ec2.nix b/nix/ec2.nix
index b1c08d0..ef10b1f 100644
--- a/nix/ec2.nix
+++ b/nix/ec2.nix
@@ -308,7 +308,7 @@ in

     deployment.ec2.securityGroupIds = mkOption {
       default = [];
-      type = types.listOf types.str;
+      type = types.listOf (types.either types.str (resource "ec2-security-group"));
       description = ''
         Security Group IDs for the instance. Necessary if starting
         an instance inside a VPC/subnet. In the non-default VPC, security
diff --git a/nixops/backends/ec2.py b/nixops/backends/ec2.py
index cc95dbd..006ab95 100644
--- a/nixops/backends/ec2.py
+++ b/nixops/backends/ec2.py
@@ -595,13 +595,13 @@ class EC2State(MachineState, nixops.resources.ec2_common.EC2CommonState):


     def _get_network_interfaces(self, defn):
-        groups = defn.security_group_ids
+        groups = map(lambda g: g['name'] if isinstance(g, dict) else g, defn.security_group_ids)

-        sg_names = filter(lambda g: not g.startswith('sg-'), defn.security_group_ids)
+        sg_names = filter(lambda g: not g.startswith('sg-'), groups)
         if sg_names != []:
             self.connect_vpc()
             vpc_id = self._conn_vpc.get_all_subnets([defn.subnet_id])[0].vpc_id
-            groups = map(lambda g: nixops.ec2_utils.name_to_security_group(self._conn, g, vpc_id), defn.security_group_ids)
+            groups = map(lambda g: nixops.ec2_utils.name_to_security_group(self._conn, g, vpc_id), groups)

         return boto.ec2.networkinterface.NetworkInterfaceCollection(
             boto.ec2.networkinterface.NetworkInterfaceSpecification(

Spot instances: support BlockDurationMinutes

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#fixed-duration-spot-instances

You can specify a duration of 1, 2, 3, 4, 5, or 6 hours. The price that you pay depends on the specified duration. To view the current prices for a 1 hour duration or a 6 hour duration, see Spot Instance Prices.

We'll have to use boto3 to be able to pass this parameter.

Disallow using security group with an instance out side of it's region

So if you provision a security group in region A and try to use it on instance in region B you'd get:

node......................> got (possibly transient) EC2 error code 'InvalidParameterValue': Value () for parameter groupId is invalid. The value cannot be empty. retrying...

No way to create Elastic IPs in a given VPC

Can't attach elastic IP:

$ nixops deploy -d personal
webserverEip> creating elastic IP address (region ‘us-east-1’)...
webserverEip> IP address is 50.19.83.180
webserver...> waiting for machine to be in running state... [running]
webserver...> associating IP address ‘50.19.83.180’...
error: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>You must specify an allocation id when mapping an address to a VPC instance</Message></Error></Errors><RequestID>5d5b9758-d0a3-4c2b-90ba-9744d0887080</RequestID></Response>

Here's the gist of my config:

{
  network.description = "Web server";

  webserver =
    { config, pkgs, resources, ... }:
    { deployment.targetEnv = "ec2";
      deployment.ec2.keyPair = keyPair;
      deployment.ec2.region = region;
      deployment.ec2.instanceType = "t2.medium";
      deployment.ec2.accessKeyId = accessKeyId;
      deployment.ec2.subnetId = subnetId;
      deployment.ec2.ami = amis."17.03".${region}.hvm-ebs;
      deployment.ec2.elasticIPv4 = resources.elasticIPs.webserverEip;
    };

    resources.elasticIPs.webserverEip = { inherit accessKeyId region; };
}

`nixops destroy` ask for confirmation for keypairs

I have ec2 instance, container and my-key-pair resources. When running nixops destroy, program asks for confirmation to destroy ec2 instance (which I deny), confirmation to destroy container (which I say yes) and silently drops my-key-pair. Now I cannot control ec2 instance.

Would be better to add confirmation to all resources, including key pairs.

Allow disabling AWS t3 Unlimited support

See https://www.cloudsqueeze.ai/amazons-t3-who-should-use-it-when-how-and-the-why/index.html for context. We need some way to specify that we want to change the behavior of the "Unlimited" feature from the default.

New AWS images

The default kernel for AWS images is broken in various ways. We would like to run at least version 4.17 (or newer). Can you add an AMI that isn't broken?

Blocker: NixOps tries to delete a non-existing AWS resource

nixops destroy -d mydeployment outputs:

fooResource..................> deleting EC2 security group `fooResource' ID `None'...


  * fooResource: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>MissingParameter</Code><Message>The request must contain the parameter groupName or groupId</Message></Error></Errors>

I expected that my fooResource would be deleted without errors. I also don't expect a Python thing like "None" in user facing output.

The state has been created by some set of NixOps invocations; I did not delete any resource manually.

Recover encrypted EBS volumes upon spot instance kill/termination

Currently, the LUKS passphrase for EBS volumes encryption is generated during the ec2 instance creation and deleted if we loose the machine or destroy it.
It might be more appropriate to have the LUKS passphrase part of the EBS volume physical specification as well (created during the volume provisioning if an option encrypt is set to true) and that way it can be passed to the instance filesystem options to be used directly.

Example of the issue:

{ account ? "lb-dev"
, region ? "us-east-1"
, zone ? "us-east-1b"
, instanceType ? "r3.2xlarge"
, volumeSize ? 50
, rootSize ? 50
, spotInstancePrice ? 301
, description ? "Experiment with spot instances"
, ...
}:
{
  network.description = description;
  
  resources.ec2KeyPairs.keypair = { accessKeyId = account; inherit region; };
  resources.iamRoles.role = { lib, ... }: {
    accessKeyId = account;
    policy = builtins.toJSON
          {
            Statement = [
              {
                Effect = "Allow";
                Action = [ "sns:Publish" "sns:ListTopics"];
                Resource = "*";
              }
            ];
          };
      };
  resources.ebsVolumes.data-vol = { resources, ... }:
    {
      inherit region zone;
      accessKeyId = account;
      size = resources.machines.machine.fileSystems."/home".ec2.size;
      disk = resources.machines.machine.fileSystems."/home".ec2.disk;
    };

  machine = { resources, lib, pkgs, ... }:
  {
    deployment.targetEnv = "ec2";
    deployment.ec2 = {
      inherit region zone instanceType spotInstancePrice;
      accessKeyId = account;
      securityGroups = [ "admin" ];
      keyPair = resources.ec2KeyPairs.keypair.name;
      ebsInitialRootDiskSize = rootSize;
      ebsOptimized = true;
      instanceProfile = resources.iamRoles.role.name;
    };

    fileSystems."/home" = {
      fsType = "ext4";
      options = ["noatime" "nodiratime"];
      device = "/dev/mapper/xvdf";
      autoFormat = true;
      ec2.disk = resources.ebsVolumes.data-vol;
      ec2.size = volumeSize;
      ec2.encrypt = true;
      ec2.volumeType = "gp2";
    };

    environment.systemPackages = [ pkgs.awscli ];
  };

}

After deploying the above network, to reproduce the issue, we can run aws ec2 terminate-instances --instance-ids <machine_id>
Then redeploying will trigger the following failure

machine.> A dependency job for local-fs.target failed. See 'journalctl -xe' for details.
machine.> the following new units were started: keys.target, nixops-keys.service, sshd-keygen.service
machine.> warning: the following units failed: cryptsetup-xvdf.service
machine.> 
machine.> ● cryptsetup-xvdf.service - Cryptographic Setup of Device /dev/mapper/xvdf
machine.>    Loaded: loaded (/nix/store/hm9lznk67dm7ky1x8zfx3vpbkfazw7jw-unit-cryptsetup-xvdf.service/cryptsetup-xvdf.service; bad; vendor preset: enabled)
machine.>    Active: failed (Result: exit-code) since Sun 2016-12-04 22:47:33 UTC; 1min 27s ago
machine.>   Process: 3581 ExecStart=/nix/store/22q816nwgzjq6dfasvkrs8hgfsfsfl3f-unit-script/bin/cryptsetup-xvdf-start (code=exited, status=2)
machine.>  Main PID: 3581 (code=exited, status=2)
machine.> 
machine.> Dec 04 22:47:31 machine systemd[1]: Starting Cryptographic Setup of Device /dev/mapper/xvdf...
machine.> Dec 04 22:47:33 machine cryptsetup-xvdf-start[3581]: No key available with this passphrase.
machine.> Dec 04 22:47:33 machine systemd[1]: cryptsetup-xvdf.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
machine.> Dec 04 22:47:33 machine systemd[1]: Failed to start Cryptographic Setup of Device /dev/mapper/xvdf.
machine.> Dec 04 22:47:33 machine systemd[1]: cryptsetup-xvdf.service: Unit entered failed state.
machine.> Dec 04 22:47:33 machine systemd[1]: cryptsetup-xvdf.service: Failed with result 'exit-code'.
machine.> error: unable to activate new configuration
error: activation of 1 of 1 machines failed (namely on ‘machine’)

ebsOptimized seems to have no effect

I'm seeing the same issue as NixOS/nixops#488

If I deploy the same instance in the same region and availability zone via the AWS console, it's EBS Optimized. If I deploy via nixops, EBS Optimized is false. Instance is m4.large.

Keep backup data after network is destroyed

Currently, after a deployment is destroyed, backup data (result of nixops backup ) is removed from the state file.

It makes sense to keep that data because a reasonable use case for a backup is re-deploying a network or a copy of it after it's destroyed.

I think deleting backup data should only happen when deleting a network.

Consolidate the deployment.ec2 security group options

Also, make sure to allow changing sec. groups for instances that were started without explicit VPC, but deployed in default VPC.

resources.ec2SecurityGroups is not being processed correctly.

Changing the description from "a" to "b" and doing a deployment does not result in a change on AWS.

    resources.ec2SecurityGroups.a = {
      inherit accessKeyId region;
      description = "a";
       rules  = [];
};

Correct behavior is that it should.

aws securityGroup configuration seems to be broken

Using nixops master trying to use security groups in the following way:

        deployment.ec2.securityGroups = [];
        deployment.ec2.securityGroupIds = ["ssh-access" "default"];

I get:

error: Multiple exceptions (5):
  * aws_a: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>No attributes specified.</Message></Error></Errors><RequestID><snip></RequestID></Response>

(repeated another 4 times for different machines)

in vpc.nix I can see:

 deployment.ec2.securityGroups = [];
 deployment.ec2.securityGroupIds = [ resources.ec2SecurityGroups.sg.name ];

All I want right now is just to be able to get ssh access to work.

Cannot specify egress rules for ec2 security groups

With other deployment tools, I can specify security group rules that limit egress from machines in EC2. Currently in nixops, I can only specify ingress rules.

The relevant boto API is authorize_security_group_egress.

EC2: use public IP by default when elasticIPv4 is set

Back to NixOS/nixops#221:

Private IP should be used if ec2.associatePublicIpAddress is not true (done) or elasticIPv4 is not set (should be done).

renaming an aws keypair silently breaks ssh access to machines

{
  host1 = { resources, ... }: {
    deployment.targetEnv = "ec2";
    deployment.ec2.region = "eu-central-1";
    deployment.ec2.keyPair = resources.ec2KeyPairs.keypairone;
    deployment.ec2.accessKeyId = "keyname";
    deployment.ec2.ami = "ami-2869bb47";
    deployment.ec2.instanceType = "t2.large";
  };
  resources.ec2KeyPairs.keypairone = {
    accessKeyId = "keyname";
    region = "eu-central-1";
  };
}

to recreate the problem, deploy the above deployment, then rename keypairone to keypairtwo, and you will loose all access to the machine via nixops

the problem can be repaired, if you rename it back to keypairone

--check re-deployment of an AWS security group with a port wildcard (1-65535) grant fails due to None being fed to int()

When --check mode is used to _re_deploy a EC2 Security Group with has a port wildcard grant (1-65535) fails, because boto returns (None, None) as the port range.

Sadly, I no longer have the logs to prove this, but the relevant piece of code is this:

• https://github.com/NixOS/nixops/blob/master/nixops/resources/ec2_security_group.py#L129

Deployment to EC2 fails with: Neither the root account nor any wheel user has a password or SSH authorized key.

Problem

Deploying an EC2 instance using the command nixops deploy --debug -d idea-board-aws produces the following error:

Failed assertions:
- Neither the root account nor any wheel user has a password or SSH authorized key.
You must set one to prevent being locked out of your system.
(use ‘--show-trace’ to show detailed location information)
Traceback (most recent call last):
  File "/nix/store/s8zczaqr24pvb5fsymqbz6jymn3q0q4v-nixops-1.5.1/bin/..nixops-wrapped-wrapped", line 953, in <module>
    args.op()
  File "/nix/store/s8zczaqr24pvb5fsymqbz6jymn3q0q4v-nixops-1.5.1/bin/..nixops-wrapped-wrapped", line 380, in op_deploy
    repair=args.repair, dry_activate=args.dry_activate)
  File "/nix/store/s8zczaqr24pvb5fsymqbz6jymn3q0q4v-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 990, in deploy
    self._deploy(**kwargs)
  File "/nix/store/s8zczaqr24pvb5fsymqbz6jymn3q0q4v-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 958, in _deploy
    self.configs_path = self.build_configs(repair=repair, include=include, exclude=exclude)
  File "/nix/store/s8zczaqr24pvb5fsymqbz6jymn3q0q4v-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 633, in build_configs
    raise Exception("unable to build all machine configurations")
Exception: unable to build all machine configurations

The assertion is being generated by nixos/modules/config/users-groups.nix, yet strangely I can nixops ssh successfully, confirming the root user does indeed have SSH authorized keys. I even checked the authorized keys file manually. The trouble is that the assertion prevents NixOps from pushing the closure. I had already successfully deployed the same machine to libvirt. Here's the configuration:

deploy.nix - The logical configuration.

{
    network.description = "Idea Board SaaS";
    
    app =
        { config, pkgs, ... }:
        let
            ideaboard = import ./idea-board.nix { inherit pkgs; };
        in 
            {
                services.nginx = {
                  enable = true;

                  config = ''
                    events {
                        worker_connections 2000;
                    }

                    http {
                        server {
                            listen 80;
                            location / {
                                proxy_pass http://localhost:5050;
                            }
                        }
                    }
                  '';
                };

                networking.firewall.allowedTCPPorts = [ 80 ];
                environment.systemPackages = [ ideaboard ];

                systemd.services.idea-board = {
                     after = [ "idea-board-key.service" ];
                     wants = [ "idea-board-key.service" ];
                     wantedBy = [ "multi-user.target" ];

                     script = ''
                        ${ideaboard}/bin/idea-board
                     '';

                     serviceConfig = {
                         User = "ideaboard";
                         Group = "ideaboard";
                     };
                 };

                 users = {
                     mutableUsers = false;
                     extraGroups.ideaboard = {};

                     extraUsers.ideaboard = {
                         group = "ideaboard";
                         description = "Idea Board service";
                     };
                 };
            };
}

deploy-aws.nix - The physical EC2-specific configuration.

let
    accessKeyId = "nixops-dev";
    region = "us-east-1";
in
    {
        app =
        { config, pkgs, resources, ... }:
        { deployment = {
            targetEnv = "ec2";

            ec2 = {
                inherit accessKeyId region;
                instanceType = "t2.micro";
                keyPair = resources.ec2KeyPairs.ideaBoardKeyPair;
                securityGroups = [ resources.ec2SecurityGroups.ideaBoardSG ];

                tags = {
                    system = "idea-board";
                };
            };
          };
        };

        resources.ec2KeyPairs.ideaBoardKeyPair = { 
            inherit region accessKeyId; 
        };

        resources.ec2SecurityGroups.ideaBoardSG = {
            inherit accessKeyId region;

            rules = [
                { codeNumber = -1;
                  typeNumber = -1;
                  fromPort = 22;
                  toPort = 22;
                  protocol = "tcp";
                  sourceIp = "0.0.0.0/0";
                }

                { codeNumber = -1;
                  typeNumber = -1;
                  fromPort = 80;
                  toPort = 80;
                  protocol = "tcp";
                  sourceIp = "0.0.0.0/0";
                }
            ];
        };
    }

Work-around

I got around the problem and achieved a successful deployment by adding a dummy user with an SSH key, to please users-groups, despite the warning of the authorizedKeys option (which I suppose implies the root user):

Warning: If you are using NixOps then don't use this option since it will replace the key required for deployment via ssh.

let
    accessKeyId = "nixops-dev";
    region = "us-east-1";
in
...
users.extraUsers.emmanuel = {
              isNormalUser = true;
              uid = 1000;
              group = "users";
              extraGroups = [ "wheel" ];
              openssh.authorizedKeys.keys = [
                  "ssh-rsa ..."
              ];
};

Deploy without provisioning

After ec2 infrastructure is created, I want just to deploy stuff. Unfortunately nixops bails out if not provided with aws credentials. By the way, Hetzner backend supports this workflow.

Nixops has some options for gradual deploy: create, create and copy, create, copy and switch. I'd like to have an option to skip the create step.

Security Groups need either port range or icmp type/code number set

Currently setting a rule with just sourceIp = "0.0.0.0/0"; fails into 'NoneType' object has no attribute 'get'.

Nix needs to require that either fromPort/toPort or typeNumber/codeNumber need to be set.

nixops destroy frequently fails

I occasionally get an error when running "nixops destroy" - probably 10-20% of the times I run "nixops destroy". When the error happens, I end up with a volume that is no longer attached to any EC2 instance.

Here's the command I use:

nixops destroy -d some-deploy --confirm

And here is the console output I see:

warning: are you sure you want to destroy EC2 machine ‘machine’? (y/N) y
machine> destroying EC2 machine... [shutting-down] [shutting-down] [shutting-down] [shutting-down] Traceback (most recent call last):
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/bin/..nixops-wrapped-wrapped", line 951, in <module>
    args.op()
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/bin/..nixops-wrapped-wrapped", line 400, in op_destroy
    wipe=args.wipe)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 1073, in destroy_resources
    self._destroy_resources(include, exclude, wipe)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 1067, in _destroy_resources
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.resources.values(), worker_fun=worker)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/deployment.py", line 1060, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/backends/ec2.py", line 1257, in destroy
    instance = self._get_instance(update=True)
  File "/nix/store/0h2c0k9mr8y5pvjd3ml30ms5rdf4kia1-nixops-1.5.1/lib/python2.7/site-packages/nixops/backends/ec2.py", line 285, in _get_instance
    assert instance_id
AssertionError

question: autogenerated route 53 private entries

@rbvermaa

  machine_foo = <elided>;

   resources.route53RecordSets = {

        a-record = { resources, ... }: {
          recordValues = [ <expression>];
          zoneId = "Z...";
          domainName = "www.example.com.";
          recordType = "A";
    };
    };

What do I need if I want to refer to the private IPv4 address of machine_foo? The other Route53 feature in NixOps only allows using the public IP.

The problem is that I cannot just put in a number, because the private IP is assigned by AWS.

Adding AWS CloudFormation Support

I plan to add AWS CloudFormation support to NixOps.

Reasons:

• CloudFormation is a well-tested tool for declaratively creating AWS resources
• Affords us full access to the range of AWS resources managed by CloudFormation
• CloudFormation documentation is well done

Once completed, this could also be done for CloudFormation's equivalents on GCP and Azure.

Unable to attach VPC security groups to an RDS instance

In the current state of ec2_rds_instance_resource it is not possible to assign VPC security group to a provisioned RDS instance.

EC2 volume attachement doesn't work with 5th generation instances (c5/m5) having nvme devices

The deploy process will hang in https://github.com/NixOS/nixops/blob/master/nixops/backends/ec2.py#L543 i.e waiting for the device to be visible in the device filesystem due to the difference in nvme device names.

user-data has too wide permission on AWS backend

I created a bunch of machines (nixops 1.4, but I think 1.3 had the same thing).
There is /root/user-data file with content:

SSH_HOST_ED25519_KEY_PUB:ssh-ed25519 AAAAC3NzaC1lZDaaaaaaaaaaaaaaaaaaaaaaaS+cdkk NixOps auto-generated key
SSH_HOST_ED25519_KEY:-----BEGIN OPENSSH PRIVATE KEY-----|b3BlbnxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxAgME|-----END OPENSSH PRIVATE KEY-----|

this file has permissions like:

-rw-r--r-- 1 root root 573 Sep  2 14:14 user-data

Looks like permissions are too wide and someone could copy the private key.
Also this private key is copied to /root/.ssh/id_charon_vpn (and this file has OK permissions, rw for root).

Is it some legacy stuff and should be removed or is it something used and should fixed?

error: EC2 machine 'm' does not have a private IPv4 address (yet)

I am looking in the AWS Console, and I see after a refresh that machine m has a private IPv4 address. However, nixops ssh m reports:

error: EC2 machine 'm' does not have a private IPv4 address (yet)

This is a bug or a design bug. It could use the AWS EC2 API to locate the IPv4 address for example.

EC2: add support encrypted root volume

It would be pretty neat if the root EBS volume could be encrypted. The use case would be to protect it if the instance is stopped and somebody gains access to its. Since we still want the instance to (re)boot automatically, it could get the LUKS key from the EC2 instance user data. The user data could be cleared when the instance is stopped.

The main problem is that all of this has to be done in the initrd (including network setup to fetch the user data).

Allow mounting /nix/store on separate ebs volume

Currently it seems not possible to mount /nix/store to a separate ebs volume. I would expect that given the following configuration the store should be located on /dev/xvdj. Instead it is mounted on the root device. So the store does get its own mount point but on the device of the root partition. Is this intentional? Is it possible to have the store on a separate volume at all?

let
  region = "us-west-1";
  zone = "us-west-1b";
in
  {
    resources.ec2KeyPairs.xxxxx = { inherit region; };

    resources.ebsVolumes.nixStore = {
      tags.Name = "nix-store";
      inherit region zone;
      size = 250;
      volumeType = "gp2";
    };

    jenkinsMaster =
      { config, resources, ... }:
      {
        deployment.targetEnv = "ec2";
        deployment.ec2.ebsInitialRootDiskSize = 250;
        deployment.ec2.region = region;
        deployment.ec2.instanceType = "c4.xlarge";
        deployment.ec2.keyPair = resources.ec2KeyPairs.xxxx;
        deployment.ec2.subnetId = "xxxx";

        fileSystems."/nix/store" = {
          autoFormat = true;
          fsType = "ext4";
          device = "/dev/xvdj";
          ec2.disk = resources.ebsVolumes.nixStore;
        };
      };
  }

Add documentation on the permissions needed for EC2 deployments.

E.g. provide a sample policy people can use to create an user and policy in the AWS console.

Cannot destroy spot instance request

I'm getting this error when trying to destroy a spot instance request:

hydra-builder-40......> cancelling spot instance request ‘sir-ex1g53fg’... error: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSpotInstanceRequestID.NotFound</Code><Message>The spot instance request ID 'sir-ex1g53fg' does not exist</Message></Error></Errors><RequestID>14d327fb-390a-45c2-b35f-7e813f457d6c</RequestID></Response>

Note that before this, I got this error: https://github.com/NixOS/nixops/blob/121bdc218c050884305375758f5749b311d1f0b8/nixops/backends/ec2.py#L725

Deleting non-existing elastic ip fails

$ nixops deploy -d csl-testnet-0.2 --kill-obsolete
resource ‘nodeip13’ is obsolete
error: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>Invalid value 'None' for PublicIp. Not a valid IPv4 address.</Message></Error></Errors><RequestID>f060863e-68b1-4be3-ba5c-9054cf22bb8c</RequestID></Response>

Support IAM Role based authentication when running from an EC2 instance

Currently, unless I'm missing something, it's impossible. NixOps insists on either accessKeyId being defined in the config (which in turn relies on secrets being present in ~/.aws/credentials) or the environment variables AWS_ACCESS_KEY_ID/EC2_ACCESS_KEY being present.

Boto actually gives you that pretty much for free. You just need to call it without any credentials. Unfortunately the code is littered with those:

self.access_key_id = defn.access_key_id or nixops.ec2_utils.get_access_key_id()
if not self.access_key_id:
    raise Exception("please set ‘deployment.ec2.accessKeyId’, $EC2_ACCESS_KEY or $AWS_ACCESS_KEY_ID")

it would be super nice to have one less secret to worry about 😅

`deployment.ec2.spotInstancePrice` does not support fractions of a penny

Desired behavior: The current bid for a r3.large in eu-central-1 is $0.0238, and I would like to bid $0.024 per hour for an instance.

Actual behavior: Because the option is specified as an integer, and the division is by 100 (https://github.com/NixOS/nixops/blob/af75e5108088b2ebaff741d77b52e5263e24faf4/nixops/backends/ec2.py#L649) I am restricted to bidding $0.03 or $0.02 per hour for an instance.

Associate ENIs to a machine using nixops

We have a use case where we access instance privately through directconnect.
For this usecase, we use DNS names that resolve to private IP of servers within our VPC.
It's of a good use for us to associate to the instances elastic network interfaces in order to keep static private IP addresses in case of instance stop/crash.

I did not find a reference to this in the nixops manual : http://nixos.org/nixops/manual/.

Can we have this feature : attaching an ENI to a VPC instance.

If the feature already exists and I missed it, can someone help on how to do it?
Thanks.

Can't create EC2 IPv6 security group entries

I tried to make an IPv6 security group entry with ::/0:

      resources.ec2SecurityGroups.mygroup = {
        accessKeyId = "myaccesskeyid";
        region = "eu-central-1";
        name = "mygroup";
        description = "mygroup";
        rules = [
          { fromPort = 22; toPort = 22; sourceIp = "0.0.0.0/0"; }
          { fromPort = 22; toPort = 22; sourceIp = "::/0"; }
        ];
      };

But it doesn't work; it gets me (with boto logging enabled):

Action=AuthorizeSecurityGroupIngress&GroupId=sg-abc123&IpPermissions.1.FromPort=22&IpPermissions.1.IpProtocol=tcp&IpPermissions.1.IpRanges.1.CidrIp=%3A%3A%2F0&IpPermissions.1.ToPort=22&Version=2014-10-01'

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidParameterValue</Code><Message>CIDR block ::/0 is malformed</Message></Error></Errors><RequestID>....</RequestID></Response>

The commet at ansible/ansible#23507 (comment) suggests that boto doesn't actually support IPv6 CIDR ranges, and that only boto3 does.