centurylinkcloud / clc-knife Goto Github PK

The priority has been set to implement Linux nodes bootstrapping before January 5th. Feature will be invoked like this:

knife clc server create <some-server-args> --bootstrap

So, what we need to do is the following:

1. Check if source server/template is actually a Linux based one. UPDATE: Might be required for windows bootstrapping. Extracted as a separate issue.
2. Check if we have Chef connectivity before sending any launch requests.
3. After server is launched, we need to login to the server and install Chef on it. Note, that most likely, we won't need to implement any of it by ourselves. We can try to reuse Knife's own facilities for that.
4. It doesn't seem like we can implement bootstrapping in async way. In theory, it could be possible with cloud-init package and custom metadata support from CLC side. For simplicity, let's stick for synchronous variant for now. Async use-case can be covered later as an improvement.
5. Note, that the primary use-case - is bootstrapping of the node using it's public IP address. In cases, where instance isn't supposed to have public IP, we might need to login into it through 'bastion' instance residing in the same network. That case can prove troublesome by itself and should be implemented separate task later.
6. If user forgot to request public IP with SSH access, we should add it by ourselves. UPDATE: That will render bootstrapping useless if user actually wants to have private workloads. Let's skip that for now. That should allow user to bootstrap a server if he is running command on the machine in the same network.

Most of the list commands can't make it into default CMD / PowerShell window size. While Linux/Mac OS distributions allow to resize terminal window without additional configuration, Windows users need to tweak default settings to change terminal window size.

Here's a list of affected commands and possible solutions:

• Group list (table view). Omit Status, Parent ID and Description fields.
• Server list. Omit ID, DC, OS Type. Note that we need to save space for --chef-nodes variant.
• Template list. Omit OS Type, API Only, Description.

Note that once we implement scrolling/resizing we might need to display these values again.

One of our customers is utilizing the following command:

knife clc server create --name 'bst01' --group b64f524c313b4a858e96f5f3cb2f3d62 --source-server IL1EMBPTPL01 --cpu 1 --memory 1 --bootstrap --bootstrap-private --run-list role[bastion] --environment battleplansprod --type standard --allow icmp --allow ssh --wait

The issued command is creating a server with an internal ip, public ip, and another secondary private ip. Is there an alternative that can be used when using this command to only create a server with just one private ip? @jruckle @marvelousNinja

This is the most important command in the plugin. The main goal is to provide a way to launch a server in CLC and support all parameters available in API. Note, that operation is asynchronous. User should be able to wait for completion. As of now, plugin supports limited set of resources which can be used to gather required parameters (not true for bare metal - they will be covered after initial release). Command will be invoked with:

knife clc server create <OPTIONS>

What we need to do is:

1. Support whole range of CLC launch parameters and pass them to server launch code.
2. Plugin should not include complex validations. However, simplest cases (required parameters) should be covered.
3. Async support. Command should provide options to wait until operation finishes. We also need some kind of visual indication that operation is in progress.
4. Public IP. In most cases, bootstrapping will require network access from external address. We should provide user with option to assign public IP during server creation. Note, that protocol permissions should be supplied too. RDP and SSH should be supported.
5. Credentials. Bootstrapping process requires root/Administrator credentials in order to access the server.

Bootstrapping itself and bare metal support were moved out of the scope of initial release. However, we must supply user with enough information to bootstrap server manually through knife bootstrap or knife bootstrap windows commands.

This is going to be one of the most often used commands. User will invoke it with:

knife clc server list <OPTIONS>

We need to:

1. Allow user to specify a datacenter to list servers from (default behaviour).
2. Allow user to list servers from _all_ datacenters (a separate option).
3. Render servers into terminal-formatted table (default behaviour).
4. Provide display stubs for values missing in cloud response (for example, descriptions and etc.)
5. Print meaningful messages if provided datacenter parameters are invalid.

Basically we need to save already established knife clc server create interface, but allow to bootstrap nodes based on Windows. Chef doesn't include windows bootstrapping code out of the box, so we would need to add knife-windows as a dependency.

Command will be invoked with:

knife clc server reboot ID <OPTIONS>

Just like delete server, power on/off commands, we should also support sync behaviour.

Run list is ignored if it is specified among command-line args. We should pass bootstrap related arguments from server launch command to the bootstrapper.

In order to provide a better insight into data, we need to add more value to knife clc server list. Here are some features we might use:

1. Public IP. Almost every other plugin lists IPs for machines, so user can quickly connect to them.
2. Chef Enabled. Inside of Knife command, we have an access to Chef Server API. If user specifies his Chef credentials in config file, he should be able to see if machine is connected to Chef Server.

The problem is - we don't have much space on a table. Some less useful fields might have to go. So, what we need to do is:

1. Display at least one public IP if it's available for the server.
2. Add a new column. It might be just a boolean field Chef Enabled or some kind of Chef node identificatior to be used in API calls. Note, that usage of Chef API requires a separate set of creds. In cases where creds aren't available, we should skip Chef code completely.
   UPDATE: Chef querying uses a separate boolean flag. Error will be thrown if flag is specified, but creds are invalid.

Command will be invoked with:

knife clc server power_off ID <OPTIONS>

Same as other async operations, we should provide optional wait flag.

During server launch, user must specify parameter sourceServerId. While it surely can be another server created by the user, CLC provides a way to launch servers from a template. Template is a rough equivalent of images in other clouds (AWS, GCE).

We have prototype version of list operation in place. We need to validate it and cover it with specs.

CLC API provides a number of power related operations on a target machine. We have several of them prototyped, but they're not checked and not covered with tests. What we need to do is to provide a way to launch them both synchronously and asynchronously and improve code coverage.

Since we've added significant amount of behaviour for Windows hosts, we need to update some of our code examples. That includes bootstrapping listings in knife clc server create section.

These two cloud operations are the last ones we're going to add before initial release. They are both async and need to be covered with tests. API client user should be able to specify protocol restrictions, sources and internal IP during public IP assignment.

Note, that prototype is being used already in knife clc server create and might need an update after changes in client code (currently, only protocol restrictions are supported).

As always, prototype code needs to be reworked and covered with specs.

We seem to complicate things for the user during bootstrap. It is quite easy to forget that user needs to specify public IP for the bootstrap. Instead, we propose to alter default behaviour and provide additional configuration option - --bootstrap-private (basically the same as https://github.com/softlayer/knife-softlayer/blob/master/lib/chef/knife/softlayer_server_create.rb#L88).

Let's cover several cases concerning bootstrapping and public/private connectivity:
User specifies --bootstrap option, but does not provide any public IP options. We need to throw error here, since connectivity is possible from private network only. We expect user to explicitly specify that with --bootstrap-private.
User specifies --bootstrap option and provides, for example --allow icmp option. While public IP is created, SSH access is not available. Bootstrapping will fail. Yet again, we need to fail early if no SSH access available through public IP and --bootstrap-private is missing.
User specifies --bootstrap option and provides --allow ssh, or --allow tcp:22, or a combination of the --allow tcp:PORT and --ssh-port PORT. In this case, SSH should be available through public IP and we should not raise any validation errors.
User specifies --bootstrap and --bootstrap-private. In this case, we do not validate connectivity options and use private IP as FQDN during bootstrap.

Note, that --bootstrap-private makes sense for sync bootstrapping only, and should be ignored in async case.

CLC provides a Group resource and uses it to structure created servers. Group can contain child groups and servers at the same level. Every group is attached to one datacenter, and groupId is a required parameter to launch a server.
We need to find a way to query CLC API for list of groups and all information related to them (namely group relationships). Note, that we may require the following operations to be performed:

• Get all groups for specific datacenter
• Get detailed information about group by its ID

Basically clone for #73. However, transport layer is different, and so are incoming parameters. Support for SSH bootstrapping rounds up sync bootstrapping support for Windows platform.

Both are async. As for its name or way for them to be invoked, there are several options. First - we can identify IP addresses as a separate resource. Possible usage examples:

knife clc ip create --server blah --allow tcp:23
knife clc ip delete 66.12.234.12 --server blah

Another way is to add additional commands for the server. This way, commands might look like this:

knife clc server assign_ip blah  --allow tcp:23
knfie clc server remove_ip blah 66.12.234.12

This is somewhat debatable, but simplest variant is probably the second one. IP address can't exist without a server. However, in case if we will decide to provide additional functionality to the plugin (like Public IPs listing and etc.), it will be quite obvious that we won't be able to place all these commands on the server level.

So, what we need to do is:

1. Choose one the specified command design directions (or provide a better option).
2. Support wait option like we do for all other async commands.
3. For ip assignment operation, provide support for protocols, source restrictions, and internal address mapping.
4. Since we kind of allow to assign IP during server creation command, it needs to be updated to support new parameters.

We need to add additional checks before trying to parse config[:ssh_port].

Since server listing requires more than one request, it should take into account possible state changes in the cloud. Current implementation fails to list servers in cases where group objects provided links to servers, but the servers themselves were terminated a second later.

Such behaviour should not affect list calls. Instead, we should handle the error and return existing servers.

With cloud support in place, we can implement command for template listing. Functionality should be similar to the one, provided by command for servers. Some information (like reserved drive letters) might be too hard to normalise into readable table. Such cases will be covered by a separate command for detailed view. To wrap up, we need to:

1. Allow user to specify a datacenter to list templates from (default behaviour).
2. Allow user to list templates from all datacenters (a separate option).
3. Render templates into terminal-formatted table (default behaviour).
4. Provide display stubs for values missing in cloud response (for example, descriptions and etc.)
5. Print meaningful messages if provided datacenter parameters are invalid.

Update: Second option seems to be useless for now. Mostly the same templates provided among all datacenters (with a few changes in capabilities field).

On QA machine with Chef DK (Knife version 12.5.1) we get the following error during sync bootstrap:

ERROR: You must pass a node name with -N when bootstrapping with user credentials

It seems that 12.5.1 and 12.6 might differ in terms of :chef_node_name option. We might need to generate default value by ourselves.

We might need to check async version too.

That is going to be second command with switching sync behaviour. Basically, we need to accept server id and optional wait parameter (just like knife clc server create does).

It seems that CLC cloud reports IP assignment success a little too early. It's quite possible for machine not to be ready for significant amount of time. We need to implement some kind of ping&wait solution for that matter.

Basically, the same way of async bootstrapping for Linux should be implemented for Windows hosts. I.e. using custom package to run bootstrapping scripts. Same checks for incoming parameters should be applied (also, to avoid compatibility issues with old versions of Knife :chef_node_name should be generated).

While we map codes correctly to a separate error types, sometimes we can't see detailed error messages. Here's an example:

ERROR: Clc::CloudExceptions::BadRequest: POST https://api.ctl.io/v2/servers/ALTD/ca1altdtest50/publicIPAddresses:          400 The request is invalid.

Clearly, it would be much better to see error itself. In that particular case, cloud returned this in response body:

{"message"=>"The request is invalid.", "modelState"=>{"internalIPAddress"=>["IP address 10.0.0.1 is not associated with server ca1altdtest50."]}}

Create server command became the most complex one. It is clear that several helper classes can be extracted from the command to allow easier testing and reusability. There are at least two issues what we need to address:

• Boostrapping. Windows support is going to complicate things even more. Perhaps we need to extract bootstrap related code into separate entity.
• CLC API Client. It is intended for client to mirror cloud capabilities. However, combinations of several requests adds complexity to Knife commands. It seems that creating a Knife-specific adapter might help.

We have implemented two async operations already, and there are more to come. However, user should have a way to track operation status later (after command has been executed). User can check operation status by requesting it with operation ID. We need to provide the following command:

knife clc operation show OPERATION_ID <OPTIONS>

Important detail - while it probably should be a separate command, we can add wait flag to knife clc operation show, so user will be able to wait until operation finishes.

Depending on several factors, CLC might report launch operation success long before SSHD becomes available. As a result, bootstrap operation might timeout, even if server was launched successfully and becomes available later.

Knife Google plugin solves the same problem by pooling for SSHD with reasonable timeout. We need provide our own solution for CLC plugin.

We already support group listing, but we don't cover the case when user does not have group structure he wants. We can address this issue by providing knife clc group create command. So, we need to:

1. Support whole range of group creation parameters
2. No need for async/sync switches (it seems like group creation call is sync)

Since feature support is a bit more complicated than one boolean flag, we need to document it as a separate activity. Examples should include several obvious ways to bootstrap a server: with specific Chef Client version, run-list and etc.

Here's an error we get:

$ knife clc server list --chef-nodes
ERROR: knife encountered an unexpected error
This may be a bug in the 'clc server list' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: Undefined method or attribute `machinename' on `node'

We need to ignore nodes with missing machine names.

Right now, API client dumps request/response logs right into STDOUT. While good for development, it should not affect anyone using Knife plugin by default. We should use provided verbosity config parameter provided by Knife and configure log levels based on that.

I've caught HTTP 405: Method not allowed during development. It was mapped to UnknownError. We need to handle that case in API client error handling code. Some debug output:

Content-Type: "application/json"
D, [2015-11-29T16:08:52.971373 #25851] DEBUG -- request: {"message":"The requested resource does not support http method 'POST'."}
I, [2015-11-29T16:08:52.971473 #25851]  INFO -- Status: 405
D, [2015-11-29T16:08:52.971552 #25851] DEBUG -- response: cache-control: "no-cache"
pragma: "no-cache"
allow: "GET,DELETE,PATCH"

In order to repro, user can try to assign public IP to a server using it's uuid URL:

client.add_public_ip('29e85508ef50458580c1a8499e1181c1?uuid=True', ports)

Command will be invoked with:

knife clc server power_on ID <OPTIONS>

We should also provide sync call support with wait flag.

Sometimes new CLC machines are being created in powered off state. That problem eventually breaks sync bootstrapping. We need to turn machine power on right before bootstrapping.

Groups provide logical structure for servers in CLC cloud. We need to provide users with knife command to display list of available groups. Note, that we need to reflect group hierarchy somehow (tree view, for example). User will invoke this command with this:

knife clc group list <OPTIONS>

What we need to do:

1. Allow user to specify a datacenter to list groups from.
2. Render groups into terminal-formatted table.
3. Render groups into terminal-formatted tree.
4. Provide display stubs for values missing in cloud response (for example, descriptions and etc.)
5. Print meaningful messages if provided datacenter parameters are invalid.

Options 2) and 3) are somewhat debatable. Tree view will surely reflect logical structure and is enough to get user going (providing we print an ID of the group). Table view might provide more info, but won't be able to reflect group structure. As an option, we might skip table view, and provide detailed group information in a separate command.

Bootstrapping command parameters differ between platform specific implementations. For example, Linux bootstrapping command doesn't support winrm_user or winrm_password parameters as well as other platform specific auth details. We clearly need to mark those parameters as platform specific during option generation. Possible solution: modify parameter descriptions to add some kind of platform tag in it:

winrm-password PASSWORD              [Windows only] Specifes password for WinRM connection

So, if tag is omitted it means that parameter supported for all platforms. If tag is present, then it is supported specified platform only.

Async interactions do not require to hold console running, and provide a quick way to launch several servers. Sync bootstrapping can sometimes take 10 minutes or even longer.

Async bootstrapping should provide a way to schedule bootstrapping automatically. Initial investigation shown that there is only one way to do it. We need to rely on public CLC package, which executes a script on launched server. Sync bootstrapping basically executes prepared shell script on a target system. The same script should be used for async implementation.

We have a prototype version of this command, but it lacks several features:

1. It displays only one public IP of the instance without allowed protocol details.
2. It does not provide a way to query for SSH/RDP credentials.

The primary purpose for these pieces of data is to be used for knife bootstrap command. We need to upgrade prototype command with those features and cover it with specs.

Note, that both credentials and IP address details are separate cloud requests. As an optional feature, we could provide configuration flags for querying additional data. Experiments show that both requests are extremely slow.

Also, commands involving server data changes should provide a hint message to this command (or even reuse display code). For example, knife clc ip create will add a public IP to the server. Command itself does not provide an information about new IP address, but user should have a way to find out about new IP address. Server launch command should provide a hint too.

We came to the point when we can start preparing for initial release. We need to write more or less descriptive README file showcasing primary commands and their parameters. Also, we need to provide a sample knife.rb file with list of all available configuration parameters for the plugin.

Prototype implemented already. Command should be invoked with this:

knife clc datacenter list

Datacenter listing does not require any additional parameters. Note, that some datacenter fields (links for example) can not be normalised into a table and can be omitted. Those details will be displayed by a separate command.

What we need to do:

1. Render datacenters into terminal-formatted table.
2. Provide display stubs for values missing in cloud response (for example, descriptions and etc.)