mirantis / launchpad

Mirantis Launchpad

Mirantis Launchpad CLI tool ("launchpad") simplifies and automates deploying Mirantis Container Runtime, Mirantis Kubernetes Engine and Mirantis Secure Registry on public clouds (like AWS or Azure), private clouds (like OpenStack or VMware), virtualization platforms (like VirtualBox, VMware Workstation, Parallels, etc.), or bare metal.

Launchpad can also provide full cluster lifecycle management. Multi-manager, high availability clusters, defined as having sufficient node capacity to move active workloads around while updating, can be upgraded with no downtime.

Documentation

Launchpad documentation can be browsed on the Mirantis Documentation site.

Example

Launchpad reads a YAML configuration file which lists cluster hosts with their connection addresses and product settings. It will then connect to each of the hosts, make the necessary preparations and finally install, upgrade or uninstall the cluster to match the desired state.

An example configuration:

apiVersion: launchpad.mirantis.com/mke/v1.3
kind: mke
spec:
  hosts:
    - role: manager
      ssh:
        address: 10.0.0.1
        user: root
    - role: worker
      ssh:
        address: 10.0.0.2
        user: ubuntu
  mke:
    version: 3.3.7

Installing a cluster:

$ launchpad apply --config launchpad.yaml


                       ..,,,,,..
              .:i1fCG0088@@@@@880GCLt;,               .,,::::::,,...
         ,;tC0@@@@@@@@@@@@@@@@@@@@@@@@@0:,     .,:ii111i;:,,..
      ,;1ttt1;;::::;;itfCG8@@@@@@@@@i @@@@0fi1t111i;,.
     .,.                  .:1L0@@   @8GCft111ii1;
                               :f0CLft1i;i1tL . @8Cti:.               .,:,.
                           .:;i1111i;itC;  @@@@@@@@@@@80GCLftt11ttfLLLf1:.
                    .,:;ii1111i:,.    , G8@@@@@@@@@@@@@@@@@@@@@@@0Lt;,
            ...,,::;;;;::,.               ,;itfLCGGG0GGGCLft1;:.



   ;1:      i1, .1, .11111i:      .1i     :1;     ,1, i11111111: ;i   ;1111;
   G@GC:  1G0@i ;@1 ;@t:::;G0.   .0G8f    L@GC:   i@i :;;;@G;;;, C@ .80i:,:;
   C8 10CGC::@i :@i :@f:;;;CG.  .0G ,@L   [email protected], ;@;     @L     L@. tLft1;.
   G8   1;  ;@i ;@i :@L11C@t   ,08fffL@L  L@.  10fi@;    .@L     L@.    .:t@1
   C0       ;@i :@i :@i   ;Gf..0C     ,8L f@.   .f0@;    .8L     L8  fft11fG;
   ..        .   .   ..     ,..,        , ..      ..      ..     ..  .,:::,

   Mirantis Launchpad (c) 2021 Mirantis, Inc.

INFO ==> Running phase: Open Remote Connection
INFO ==> Running phase: Detect host operating systems
INFO [ssh] 10.0.0.2:22: is running Ubuntu 18.04.5 LTS
INFO [ssh] 10.0.0.1:22: is running Ubuntu 18.04.5 LTS
INFO ==> Running phase: Gather Facts
INFO [ssh] 10.0.0.1:22: gathering host facts
INFO [ssh] 10.0.0.2:22: gathering host facts
INFO [ssh] 10.0.0.1:22: internal address: 172.17.0.2
INFO [ssh] 10.0.0.1:22: gathered all facts
INFO [ssh] 10.0.0.2:22: internal address: 172.17.0.3
INFO [ssh] 10.0.0.2:22: gathered all facts
...
...
INFO Cluster is now configured.  You can access your admin UIs at:
INFO MKE cluster admin UI: https://test-mke-cluster.example.com
INFO You can also download the admin client bundle with the following command: launchpad client-config

Support, Reporting Issues & Feedback

Please use Github issues to report any issues, provide feedback, or request support.

launchpad's Issues

UCP required port missing in error message

I get an error message:

INFO[0019] ==> Running phase: Install UCP components
INFO[0021] 172.16.100.101:  time="2020-06-11T14:32:44Z" level=info msg="Your Docker daemon version 19.03.8, build 5ecd622f3f (3.10.0-693.el7.x86_64) is compatible with UCP 3.3.0 (1aefc52)"
INFO[0025] 172.16.100.101:  time="2020-06-11T14:32:48Z" level=info msg="Checking required ports for connectivity"
INFO[0058] 172.16.100.101:  time="2020-06-11T14:33:21Z" level=fatal msg="unable to verify host port connectivity: unable to verify required UCP ports: the following required ports are blocked on your host: 179, 443, 2376, 6443, 6444, 10250, 12376, 12378 - 12386, 12388.  Check your firewall settings"

Note that the UCP port list contains 179, 443, 2376, 6443, 6444, 10250, 12376, 12378 - 12386, 12388 but UCP also requires port 2377 to be opened as well.

Please add the port to the list in the error message.

upgrade example: docs/upgrades.md

In the example the upgrade verbiage and yaml reference a docker engine that isn't released.

This maybe confusing to users. Currently there is only one docker engine compatible(19.03.8) with UCP 3.3.0.

Say you want to upgrade the Docker EE engine running in the cluster. You've bootsrapped the cluster with version `19.03.8` and want to upgrade to say `19.03.14`.

yaml:

  engine:
    version: 19.03.14 # was previously 19.03.8

/etc/environment not updated before pkg install

/etc/environment gets updated with environment values from launchpad.yaml after required OS packages are installed rather than before. If it was updated before OS package installations we could set HTTP proxy settings once in the launchpad.yaml rather that having to set on each target host before we run launchpad apply.

launchpad apply - ssh: handshake failed

Trying to follow the https://github.com/Mirantis/launchpad/blob/master/docs/getting-started.md.

Given I have 3 Virtual Machines:

1. xx.xx.xxx.xx4 Where I run the. launchpad
2. xx.xx.xxx.xx5 is the manager node
3. xxx.xxx.xxx.xx8 is the worker node

When running the command: ./launchpad --debug apply

This is my config file:

DEBU[0000] tracking analytics event 'Cluster Apply Started' 
DEBU[0000] opened config file from /home/my_user/cluster.yaml 
DEBU[0000] loaded configuration:
apiVersion: launchpad.mirantis.com/v1beta2
kind: UCP
metadata:
  name: ucp-kube
spec:
  hosts:
  - address: xx.xx.xxx.xx5
    role: manager
    privateInterface: eth0
    engineConfig: {}
    ssh:
      user: my_user
      port: 22
      keyPath: /home/my_user/.ssh/google_compute_engine
  - address: xxx.xxx.xxx.xx8
    role: worker
    privateInterface: eth0
    engineConfig: {}
    ssh:
      user: my_user
      port: 22
      keyPath: /home/my_user/.ssh/google_compute_engine
  ucp:
    version: 3.3.1
    imageRepo: docker.io/docker
    installFlags: [--admin-username=admin, --admin-password=passw0rd!, --default-node-orchestrator=kubernetes]
  engine:
    version: 19.03.8
    repoURL: https://repos.mirantis.com
    installURLLinux: https://get.mirantis.com/
    installURLWindows: https://get.mirantis.com/install.ps1
    channel: stable

Then I get the error message:

INFO[0000] xxx.xxx.xxx.xx8: opening SSH connection      
DEBU[0000] using SSH auth sock /tmp/ssh-0VrK4rzCq1/agent.4744 
ERRO[0000] xxx.xxx.xxx.xx8: failed to connect -> ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
ERRO[0000] xx.xx.xxx.xx5: failed to connect -> ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
INFO[0000] xxx.xxx.xxx.xx8: opening SSH connection      
DEBU[0000] using SSH auth sock /tmp/ssh-0VrK4rzCq1/agent.4744 
ERRO[0000] 1xxx.xxx.xxx.xx8: failed to connect -> ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 
INFO[0000] xx.xx.xxx.xx5: opening SSH connection        
DEBU[0000] using SSH auth sock /tmp/ssh-0VrK4rzCq1/agent.4744 
ERRO[0000] xx.xx.xxx.xx5: failed to connect -> ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain 

Doing
ssh [email protected]
ssh [email protected]

Work, so the ssh keys are correctly set up. Is there something I'm missing? The ssh connection opened by the GO client needs any additional config?

Incorrect templating of shim container

I am seeing the error below when trying to run launchpad with a private repo:

unable to run install step \"Deploy Kubelet\": unable to reconcile state of Kubelet component: unable to reconcile persistent state: image \"docker.io/repo.shanedabes.com/mirantis/ucp-containerd-shim-process:3.3.8\": not found

It looks like the templating of the image isn't working properly, the above should be repo.shanedabes.com/mirantis... instead of docker.io/repo.shanedabes.com/mirantis. This starts happening when trying to install MKE versions 3.3.8 and up, it's not seen on versions lower than that.

I am on the newest stable release of launchpad.

Thanks!

FATA failed to initialize swarm: Error response from daemon: advertise address must be a non-zero IP address or network interface (with optional port number)

apiVersion: launchpad.mirantis.com/mke/v1.3
kind: mke
metadata:
  name: my-mke
spec:
  mke:
    adminUsername: admin
    adminPassword: passw0rd!
    installFlags:
      - --default-node-orchestrator=kubernetes
      - --pod-cidr 10.0.0.0/16
  hosts:
  - role: manager
    ssh:
      address: 192.168.110.100
      keyPath: ~/.ssh/id_rsa
      user: theuser
    privateInterface: enp0s3
  - role: worker
    ssh:
      address: 192.168.110.101
      keyPath: ~/.ssh/id_rsa
      user: theuser
    privateInterface: enp0s3

Failing to initialize swarm. no way to set advertise address when initializing swarm.
NOTE : above spec from mirantis doc and changed values for my nodes.

Source for launchpad?

Hi Folks,

Is there a plan to make launchpad opensource or no?

SSH packets get dropped when deploying onto Ubuntu in Azure public cloud

When using launchpad to deploy docker ee onto Ubuntu VMs in Azure public cloud the installation hangs at the following point:

INFO[0020] ==> Running phase: Install Docker EE Engine on the hosts

After this it is no longer possible to ssh onto the VM. A packet capture on the VM (via serial console) shows that the SSH TCP SYN packets are not being ACKed. Is the docker ee install adding some firewall or iptable rules that are causing this?

$ launchpad version
version: 1.1.0-beta3
commit: 11f7d21

Fatal Error :: FATA[0075] Failed to check DTR bootstrapper image version

Hello Folks,

Thanks so much for the fix for #52. There's a new one now.

sh-4.2# ./launchpad-linux-x64 version
version: 1.0.2
commit: ad58f68

Here's my cluster.yml file (datamasked):

# API version: launchpad.mirantis.com/v1
# Ansible managed
# Source: https://bitbucket.example.com/projects/EIU/repos/ei-unix-docker-ee-playbooks/browse
# Launchpad Sales Pitch: https://www.mirantis.com/blog/mirantis-launchpad-quick-start/
# Launchpad Source: https://github.com/Mirantis/launchpad/
#
# flags for the installer are displayed via:
# docker run -t -i --rm docker/ucp:3.3.0 install --help
#


apiVersion: launchpad.mirantis.com/v1
kind: DockerEnterprise
metadata:
  name: lab-unix-ucp-ca.example.com
spec:

  dtr:
    version: 2.8.0
    #imageRepo: "docker.io/docker"
    imageRepo: "lab-unix-dtr-ca.example.com/unixinfra"
    replicaConfig: sequential # we install sequentially to adhere to the install primary, then install the replicas pattern
    installFlags:
      - --dtr-external-url lab-unix-dtr-ca.example.com
      - --ucp-username=ucpadmin
      - --ucp-password=[REDACTED]
      - --ucp-url lab-unix-ucp-ca.example.com
      - --ucp-ca "$(curl -k -s https://lab-unix-ucp-ca.example.com:443/ca)"
      - --replica-http-port 80
      - --replica-https-port 443
      - --http-proxy proxy-mwg-http.example.com:8080
      - --https-proxy proxy-mwg-http.example.com:8443
      - --no-proxy localhost,127.0.0.1,10.*,172.*,192.*,159.*,example.com
      - --debug
      - --dtr-ca "$(cat /root/docker_install/dtr-ca.pem)"
      - --dtr-cert "$(cat /root/docker_install/dtr-cert.pem)"
      - --dtr-key "$(cat /root/docker_install/dtr-key.pem)"

  ucp:
    version: 3.3.3
    #imageRepo: "docker.io/docker"
    imageRepo: "lab-unix-dtr-ca.example.com/unixinfra"
    installFlags:
      - --admin-username=ucpadmin
      - --admin-password=[REDACTED]
      - --secure-overlay
      - --default-node-orchestrator=kubernetes
      - --san lab-unix-ucp-ca.example.com
      - --skip-cloud-provider-check
      - --force-minimums
      - --external-server-cert
      - --disable-tracking
      - --disable-usage
      - --debug
      - --license '{REDACTED}'
    configData: |-
      [auth]
        backend = "ldap"
      [scheduling_configuration]
        default_node_orchestrator = "kubernetes"
        enable_admin_ucp_scheduling = true
      [cluster_config]
        ipip_mtu = "1356"
        calico_mtu = "1356"
        secure_overlay = true

  engine:
    version: 19.03.8
    installURLLinux: file:////root/docker_install/example-customized-ucp-install.sh

  hosts:
  ############
  # managers #
  ############
  - address: host660.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: manager
  - address: host661.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: manager
  - address: host662.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: manager

  ###########
  # workers #
  ###########

  - address: host663.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: worker
  - address: host664.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: worker
  - address: host665.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: worker

  ###########
  #  dtrs   #
  ###########

  - address: host666.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: dtr
  - address: host667.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: dtr
  - address: host668.example.com
    sshPort: 22
    user: root
    # sshKeyPath: ~/.ssh/launchpad_rsa
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/launchpad_rsa
    role: dtr

When I execute it via: ./launchpad-linux-x64 --debug --disable-telemetry apply --config cluster.yml | tee test-new-launchpad.log

Everything appears to ok except for the last line:

INFO[0073] ==> Running phase: Upgrade DTR components
DEBU[0073] host666.example.com: found DTR installed, using as leader
DEBU[0073] analytics disabled, not tracking event 'Upgrade DTR components'
INFO[0073] See /root/.mirantis-launchpad/cluster/lab-unix-ucp-ca.example.com/apply.log for more logs
DEBU[0073] analytics disabled, not tracking event 'Cluster Apply Failed'
FATA[0075] Failed to check DTR bootstrapper image version

Any thoughts?

Thanks!
-Patrick

Please add launchpad to homebrew

Would be nice to be able to install this tool using brew instead of downloading the packages from this repo.

Invalid memory address error when running Launchpad prior to registering

If you attempt to run launchpad prior to registering as a user, some commands will trigger an "invalid memory address" error, followed by a stack trace.

Example:

$ launchpad apply
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x848a01]

goroutine 1 [running]:
github.com/Mirantis/mcc/pkg/analytics.(*Client).TrackEvent(0xf1adb0, 0xa1c100, 0x13, 0xc0002d44e0, 0xc00005fc20, 0xc00005fc00)
	/go/src/github.com/Mirantis/mcc/pkg/analytics/analytics.go:74 +0x291
github.com/Mirantis/mcc/pkg/analytics.(*Client).RequireRegisteredUser(0xf1adb0, 0xa18d07, 0xe)
	/go/src/github.com/Mirantis/mcc/pkg/analytics/user.go:12 +0x5a
github.com/Mirantis/mcc/pkg/analytics.RequireRegisteredUser(...)
	/go/src/github.com/Mirantis/mcc/pkg/analytics/analytics.go:119
github.com/Mirantis/mcc/cmd.NewApplyCommand.func1(0xc000288c80, 0x3, 0x4)
	/go/src/github.com/Mirantis/mcc/cmd/apply.go:32 +0x69
github.com/urfave/cli/v2.(*Command).Run(0xc0000a6d80, 0xc000288a80, 0x0, 0x0)
	/root/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:151 +0x57c
github.com/urfave/cli/v2.(*App).RunContext(0xc0001b4300, 0xaee020, 0xc00001e5c8, 0xc00000c080, 0x2, 0x2, 0x0, 0x0)
	/root/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:306 +0x790
github.com/urfave/cli/v2.(*App).Run(...)
	/root/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:215
main.main()
	/go/src/github.com/Mirantis/mcc/main.go:82 +0x67f

The same behavior ("invalid memory address" and stack trace) is seen when running the reset command without registering.
This gives very little indication of the real underlying issue, and will only confuse users. Instead, this error should be handled in the code, and a clear error message should be provided.

By contrast, the init command, when the user has not yet registered, outputs a much more useful error message:

$ launchpad init
FATA[0000] Registration or license acceptance is required. Please use `launchpad register` command to register

This is very clear what the specific problem is and what is required to correct it.

0.13.0-rc7 -> yaml: line NNN: could not find expected ':'

Hello,

Using 0.13.0-rc7. In the installFlags stanza in the DTR section, I'd like to pass the --dtr-ca option in the cluster.yml file:

spec:
  dtr:
    version: 2.8.0
    imageRepo: "docker.io/docker"
    #imageRepo: "docker.io/mirantis"
    replicaConfig: sequential
    installFlags:
      - --dtr-external-url https://dtr.example.com:443
      - --ucp-username=admin
      - --ucp-password=orcaorca
      - --ucp-url ucp.example.com:443
      - --ucp-insecure-tls
      - --replica-http-port 80
      - --replica-https-port 443
      - --dtr-ca="-----BEGIN CERTIFICATE-----
MIIFqjCCA5KgAwIBAgIQGHY3AQwhwJJDjWP4ysZGSjANBgkqhkiG9w0BAQUFADBm
(output redacted for brevity)
CFN1biBMa"
      - --debug

However, when I run this, I see:

[root@hostnamedocker_install]# ./launchpad-linux-x64 --debug --disable-telemetry apply --config cluster.yml
DEBU[0000] analytics disabled, not tracking event 'Cluster Apply Started'
DEBU[0000] opened config file from /root/docker_install/cluster.yml
DEBU[0000] analytics disabled, not tracking event 'Cluster Apply Failed'
FATA[0002] yaml: line 33: could not find expected ':'

Thoughts?

SSH fails to connect after launchpad apply

I receive the following errors and am unable to ever connect via SSH (~/.mirantis-launchpad/cluster/[cluster-name]/apply.log):

time="03 Apr 24 14:36 UTC" level=debug msg="preparing phase 'Check For Upgrades'"
time="03 Apr 24 14:36 UTC" level=info msg="\x1b[32m==> Running phase: Check For Upgrades\x1b[0m"
time="03 Apr 24 14:36 UTC" level=debug msg="phase 'Check For Upgrades' took 0s"
time="03 Apr 24 14:36 UTC" level=debug msg="preparing phase 'Open Remote Connection'"
time="03 Apr 24 14:36 UTC" level=info msg="\x1b[32m==> Running phase: Open Remote Connection\x1b[0m"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using default known_hosts file ~/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: checking identity file /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using an unencrypted private key from /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:36 UTC" level=debug msg="[ssh] [sanitized]:22: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=error msg="[SSH] [sanitized]: attempt 1 of 60.. failed to connect: not connected: client connect: ssh dial: dial tcp [sanitized]:22: connect: connection timed out"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using known_hosts file from ssh config /terraTrain/.ssh/known_hosts"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: failed to get ssh agent client: connect ssh agent: SSH_AUTH_SOCK is not set"
time="03 Apr 24 14:38 UTC" level=debug msg="[SSH] [sanitized]: using cached auth method for /terraTrain/key-pair"

Instances are up in Openstack and I am able to connect with key pair on my local machine -- but I should also be able to connect using my TerraTrain lab and configuration setup by launchpad 🤷

What am I missing? Thanks.

launchpad init generates a config with unsupported kind: UCP

Description of the issue:
./launchpad init generates a config with kind: UCP which causes error since only kind: DockerEnterprise is supported according to [1], a test also confirms it.

Affected version:
version: 0.14.0
commit: c7d9682

Test case*

./launchpad init > cluster.yaml
./launchpad apply
FATA[0000] Key: 'ClusterConfig.Kind' Error:Field validation for 'Kind' failed on the 'eq' tag 
echo $?
1

Generated config

./launchpad init
apiVersion: launchpad.mirantis.com/v1beta3
kind: UCP
metadata:
  name: my-ucp-cluster
spec:
  hosts:
  - address: 10.0.0.1
    role: manager
    engineConfig: {}
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/id_rsa
  - address: 10.0.0.2
    role: worker
    engineConfig: {}
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/id_rsa
  - address: 10.0.0.3
    role: dtr
    engineConfig: {}
    ssh:
      user: root
      port: 22
      keyPath: ~/.ssh/id_rsa
  ucp:
    version: 3.3.2
  dtr:
    version: 2.8.2
    imageRepo: ""
    installFlags: []
    replicaConfig: sequential
    metadata: null
  engine:
    version: 19.03.12

[1] https://github.com/Mirantis/launchpad/blob/master/docs/configuration-file.md#kind

Ability to pass launchpad config using stdin

It would be very useful to be able to pass the launchpad yaml config into the command through standard in, for example when taking input from a computed source like terraform outputs. Instead of having to write to file, we could do something like below:

terraform output launchpad_yaml | launchpad apply -c -

Error\">Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query - deploying onto Windows Workers

After deploying azure infrastructure successfully though the terraform example (Linux master and Windows Worker) as per the recent PR by Steven Richard, the Launchpad yaml info is successfully created. However when you run launchpad apply to deploy MCR and MKE the deployment fails with the following error:

INFO [winrm] 20.108.20.83:5986: mirantis container runtime not installed
ERRO [winrm] 20.108.20.83:5986: failed to resolve internal address: command failed (received output to stderr on windows)
INFO See /Users/jas.atwal/.mirantis-launchpad/cluster/jas-mke-cluster/apply.log for more logs
FATA failed on 1 hosts:

• [winrm] 20.108.20.83:5986: [winrm] 20.108.20.83:5986: failed to resolve internal address: command failed (received output to stderr on windows)

The apply.log states this:

time="16 Jul 21 08:10 BST" level=debug msg="[winrm] 20.108.20.83:5986: executing powershell.exe -NonInteractive -ExecutionPolicy Bypass -NoProfile -EncodedCommand JABQAHIAbwBnAHIAZQBzAHMAUAByAGUAZgBlAHIAZQBuAGMAZQA9ACcAUwBpAGwAZQBuAHQAbAB5AEMAbwBuAHQAaQBuAHUAZQAnADsAIAAoAEcAZQB0AC0ATgBlAHQASQBQAEEAZABkAHIAZQBzAHMAIAAtAEEAZABkAHIAZQBzAHMARgBhAG0AaQBsAHkAIABJAFAAdgA0ACAALQBJAG4AdABlAHIAZgBhAGMAZQBBAGwAaQBhAHMAIAAnAEUAdABoAGUAcgBuAGUAdAAnACkALgBJAFAAQQBkAGQAcgBlAHMAcwA="
time="16 Jul 21 08:10 BST" level=debug msg="[winrm] 20.108.20.83:5986: (stderr) #< CLIXML"
time="16 Jul 21 08:10 BST" level=debug msg="[winrm] 20.108.20.83:5986: (stderr) <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04\"><S S="Error">Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query for instances of the x000D__x000A<S S="Error">ROOT/StandardCimv2/MSFT_NetIPAddress class on the CIM server: SELECT * FROM MSFT_NetIPAddress WHERE ((InterfaceAlias x000D__x000A<S S="Error">LIKE 'Ethernet')) AND ((AddressFamily = 2)). Verify query parameters and retry.x000D__x000A<S S="Error">At line:1 char:42_x000D__x000A_<S S="Error">+ ... Continue'; (Get-NetIPAddress -AddressFamily IPv4 -InterfaceAlias 'Eth ...x000D__x000A<S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x000D__x000A<S S="Error"> + CategoryInfo : ObjectNotFound: (MSFT_NetIPAddress:String) [Get-NetIPAddress], CimJobException_x000D__x000A_<S S="Error"> + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetIPAddress_x000D__x000A_<S S="Error"> x000D__x000A"
time="16 Jul 21 08:10 BST" level=trace msg="[winrm] 20.108.20.83:5986: trying vEthernet (Ethernet) as a private interface alias"
time="16 Jul 21 08:10 BST" level=debug msg="[winrm] 20.108.20.83:5986: executing powershell.exe -NonInteractive -ExecutionPolicy Bypass -NoProfile -EncodedCommand JABQAHIAbwBnAHIAZQBzAHMAUAByAGUAZgBlAHIAZQBuAGMAZQA9ACcAUwBpAGwAZQBuAHQAbAB5AEMAbwBuAHQAaQBuAHUAZQAnADsAIAAoAEcAZQB0AC0ATgBlAHQASQBQAEEAZABkAHIAZQBzAHMAIAAtAEEAZABkAHIAZQBzAHMARgBhAG0AaQBsAHkAIABJAFAAdgA0ACAALQBJAG4AdABlAHIAZgBhAGMAZQBBAGwAaQBhAHMAIAAnAHYARQB0AGgAZQByAG4AZQB0ACAAKABFAHQAaABlAHIAbgBlAHQAKQAnACkALgBJAFAAQQBkAGQAcgBlAHMAcwA="
time="16 Jul 21 08:11 BST" level=debug msg="[winrm] 20.108.20.83:5986: (stderr) #< CLIXML"
time="16 Jul 21 08:11 BST" level=debug msg="[winrm] 20.108.20.83:5986: (stderr) <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04\"><S S="Error">Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query for instances of the x000D__x000A<S S="Error">ROOT/StandardCimv2/MSFT_NetIPAddress class on the CIM server: SELECT * FROM MSFT_NetIPAddress WHERE ((InterfaceAlias x000D__x000A<S S="Error">LIKE 'vEthernet (Ethernet)')) AND ((AddressFamily = 2)). Verify query parameters and retry.x000D__x000A<S S="Error">At line:1 char:42_x000D__x000A_<S S="Error">+ ... Continue'; (Get-NetIPAddress -AddressFamily IPv4 -InterfaceAlias 'vEt ...x000D__x000A<S S="Error">+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~x000D__x000A<S S="Error"> + CategoryInfo : ObjectNotFound: (MSFT_NetIPAddress:String) [Get-NetIPAddress], CimJobException_x000D__x000A_<S S="Error"> + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetIPAddress_x000D__x000A_<S S="Error"> x000D__x000A"
time="16 Jul 21 08:11 BST" level=error msg="[winrm] 20.108.20.83:5986: failed to resolve internal address: command failed (received output to stderr on windows)"
time="16 Jul 21 08:11 BST" level=debug msg="phase 'Gather Facts' took 0s"
time="16 Jul 21 08:11 BST" level=debug msg="tracking analytics event 'Gather Facts'"
time="16 Jul 21 08:11 BST" level=debug msg="tracking analytics event 'Detect host operating systems'"
time="16 Jul 21 08:11 BST" level=debug msg="tracking analytics event 'Check For Upgrades'"
time="16 Jul 21 08:11 BST" level=debug msg="tracking analytics event 'Cluster Apply Failed'"
time="16 Jul 21 08:11 BST" level=info msg="See /Users/jas.atwal/.mirantis-launchpad/cluster/jas-mke-cluster/apply.log for more logs "
time="16 Jul 21 08:11 BST" level=fatal msg="failed on 1 hosts:\n - [winrm] 20.108.20.83:5986: [winrm] 20.108.20.83:5986: failed to resolve internal address: command failed (received output to stderr on windows)"

Steps to reproduce:

1. Successfully deploy a master node and a windows worker node using the azure terraform example based on the PR request by Steven Richards
2. Create the corresponding launchpad.yml file and run launchpad apply
3. Notice the above error, which is a blocker and stopping the successful deployment of MKE on Azure.

Versions:
Terraform v1.0.2

on darwin_amd64

• provider registry.terraform.io/hashicorp/azurerm v2.32.0
• provider registry.terraform.io/hashicorp/local v2.0.0
• provider registry.terraform.io/hashicorp/random v3.0.0
• provider registry.terraform.io/hashicorp/tls v3.0.0

terraform providers

Providers required by configuration:

├── provider[registry.terraform.io/hashicorp/tls] 3.0.0
├── provider[registry.terraform.io/hashicorp/azurerm] 2.32.0
├── provider[registry.terraform.io/hashicorp/local] 2.0.0
├── provider[registry.terraform.io/hashicorp/random] 3.0.0

Launchpad

version: 1.3.0
commit: 706d2b4

MCR

v20.10.0

MKE

v3.4.3

Could this be related to this kubernetes issue?

Failed to check DTR bootstrapper image version

$ launchpad version
version: 1.0.2
commit: ad58f68

We are seeing the following FATAL error when deploying DTR without internet access:

INFO[0856] ==> Running phase: Upgrade DTR components
DEBU[0856] host.domain.com: found DTR installed, using as leader
DEBU[0856] analytics disabled, not tracking event 'Upgrade DTR components'
INFO[0856] See /home/deploy/.mirantis-launchpad/cluster/launchpad-de/apply.log for more logs                                                                                     DEBU[0856] analytics disabled, not tracking event 'Cluster Apply Failed'
FATA[0856] Failed to check DTR bootstrapper image version

I notice in this release the option --disable-upgrade-check is missing which is available in 1.1.0-beta5, could this option enable us to work around the issue?

Does the TF setup work with AWS as the cloud provider

I am currently trying to set up UCP on AWS using the provided TF example.
I have difficulties to set up the AWS ESB and ALB.

According to the documentation, labels need to be set. Apart from the fact that the docs contradict each other on the Key Values the hostname FQDN dns or just hostname,

• https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/mke/install-mke.html#install-on-aws
• https://docs.mirantis.com/docker-enterprise/v3.1/dockeree-products/mke/deploy-apps-with-kubernetes/persistent-storage/using-aws-ebs-storage.html

My question is basically should the example work as is with "cloud-provider=aws" or do I need to change the TF, label tags and so on?

/etc/systemd/system/docker.service.d/http-proxy.conf "\n" in value

A literal \n is being added to the end of each environment variable value and no newline between Environment lines in /etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="https_proxy=http://proxy:8080\n"Environment="NO_PROXY='10.0.0.*'\n"

Version:

$ launchpad version
version: 1.0.0-rc1
commit: c3f0a0b

custom imageRepo DTR parameter prefixing pulls with "docker"

Hi Folks,

# ./launchpad version
version: 1.0.1
commit: 47d6e89

snippet of dtr yaml segment in cluster.yml:

  dtr:
    version: 2.8.0
    #imageRepo: "docker.io/docker"
    imageRepo: "dtr.example.com/nix"
    replicaConfig: sequential # we install sequentially to adhere to the install primary, then install the replicas pattern
    installFlags:
      - --dtr-external-url dtr.example.com

Executing:

./launchpad --debug --disable-telemetry apply --config cluster.yml | tee launchpad-apply.log

the UCP installation is fine. However, when it attempts to pull the DTR images, I see this (snippet):

INFO[0090] host1.example.com: pulling imagedtr.example.com/nix/docker/dtr-notary-signer:2.8.0
WARN[0090] host1.example.com: failed to pull image dtr.example.com/nix/docker/dtr-garant:2.8.0:
Error response from daemon: pull access denied fordtr.example.com/nix/docker/dtr-garant, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
WARN[0090] host1.example.com: failed to pull image dtr.example.com/nix/docker/dtr-garant:2.8.0 - retrying

See how it's prefixing the image with a "docker/"? It reads:

WARN[0090] host1.example.com: failed to pull image dtr.example.com/nix/docker/dtr-garant:2.8.0:

it should read:

WARN[0090] host1.example.com: failed to pull image dtr.example.com/nix/dtr-garant:2.8.0:

Thoughts?

Thanks!

Macro is not triggered always for the first time

Hi!
I am using your soft and it's very cool!
But I am facing some little issue :(
I am using Launchpad X as stream deck while streaming for scene switching etc.
Anyway it seems sometimes it needs 2-3 taps for triggering the macro.
I set it like this:

Am I doing something wrong or it's "feature" or can I evade it somehow?
Thanks for help!

Tilde notation to reference user's home dir does not work in ClusterConfig.Spec.Hosts[0].SSHKeyPath

launchpad-linux-x64 apply fails to find SSH key file if the path contains ~ to reference users's home directory (like in example [2]).
The tool fails to find the key file but the error message [0] in this case is obscure (worth to return obvious message). OS is [1]

[0]

FATA[0002] Key: 'ClusterConfig.Spec.Hosts[0].SSHKeyPath' Error:Field validation for 'SSHKeyPath' failed on the 'file' tag
Key: 'ClusterConfig.Spec.Hosts[1].SSHKeyPath' Error:Field validation for 'SSHKeyPath' failed on the 'file' tag 

[1]

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

[2] https://github.com/Mirantis/launchpad/blame/master/docs/configuration-file.md#L19

Error syncing load balancer: failed to ensure load balancer: Multiple untagged security groups found

After installing MKE with the provided TF example, I created a nginx Ingress controller with Service Type Load balancer:

In AWS Console I saw the CLB was created but in K8s it failed with this event:

Error syncing load balancer: failed to ensure load balancer: Multiple untagged security groups found for instance i-0e16ead5ecfcda342; ensure the k8s security group is tagged

The resolution was to add

  tags = map(
    "${var.kube_cluster_tag}", "shared"
  )

To the Security Group and VPC Tags.

launchpad apply fails when adding DTR node

When additional DTR node is added to docker ee cluster already provisioned with DTR launchpad apply fails. From the log:

time="16 Sep 20 14:04 BST" level=debug msg="UCP health check response code: 503, expected 200"
time="16 Sep 20 14:04 BST" level=info msg="Performing health check against UCP: x.x.x.x:443; elapsed: 8m20s"
time="16 Sep 20 14:04 BST" level=info msg="Performing health check against UCP: x.x.x.x:443; elapsed: 8m25s"
time="16 Sep 20 14:04 BST" level=info msg="Performing health check against UCP: x.x.x.x:443; elapsed: 8m30s"
time="16 Sep 20 14:04 BST" level=info msg="Performing health check against UCP: x.x.x.x:443; elapsed: 8m35s"
time="16 Sep 20 14:04 BST" level=info msg="Performing health check against UCP: x.x.x.x:443; elapsed: 8m40s"
time="16 Sep 20 14:04 BST" level=info msg="Performing health check against UCP: x.x.x.x:443; elapsed: 8m45s"
time="16 Sep 20 14:04 BST" level=debug msg="tracking analytics event 'Validating UCP Health'"
time="16 Sep 20 14:04 BST" level=info msg="See /home/blah/.mirantis-launchpad/cluster/launchpad-de/apply.log for more logs "
time="16 Sep 20 14:04 BST" level=debug msg="tracking analytics event 'Cluster Apply Failed'"
time="16 Sep 20 14:04 BST" level=fatal msg="failed to determine health of UCP: polling failed with 5 attempts 30s apart: unexpected response code"

I can see the node is added in the UCP console but the type is "kubernetes" whereas the original DTR node is "mixed"

Terraform examples issue a deprecation warning

When using the provided examples for Terraform, the following warning is displayed:

Warning: Interpolation-only expressions are deprecated

  on modules/master/main.tf line 29, in locals:
  29:   subnet_count = "${length(var.subnet_ids)}"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a
mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.

(and 6 more similar warnings elsewhere)

This is a minor issue, and hopefully easy to fix. But as a new user, it causes me unnecessary noise and friction when inexperienced with Terraform and Launchpad and MKE, and trying to focus on the task at hand.

Incorrect cloud provider type for OpenStack terraform template

launchpad/examples/terraform/openstack/output.tf

Lines 62 to 65 in ca0e887

	cloud = {
	provider = "openstack"
	configFile = var.provider_config_file_path
	}

openstack is unsupported cloud provider type.
Tested with launchpad 1.5.2 and receive error
level=error msg="invalid Kubernetes config: cloud provider value must be one of [aws azure vsphere gce external]

For OpenStack provider should be external

cloud = {
    provider = "external"
    configFile = var.provider_config_file_path
}

debug output message has a typo for "succesfully"

sample output:

INFO[0132] 192.168.100.5: joined succesfully

Should read successfully

engine not upgrading with cluster.yml

Installed with engine 19.03.08

Updated cluster.yml but the change was not applied when running launchpad apply

engine
version: 19.03.14 # was previously 19.03.8

minimum engine version check protection for UCP enabled cluster

Recommend adding a minimum engine version check logic based on the requirements of UCP.

In this example a cluster was created with `UCP 3.3.0' & '19.03.8'

Used cluster.yaml to downgrade '19.03.5'. This was allowed by launchpad.

The issue here is that the minimum requirement is '19.03.8'

This condition should be prevented to "help" the user.

Please release statically compiled binaries

When using launchpad from inside an alpine container, we are unable to execute the binary due to it depending on libc.

iptables warnings

firewalld is logging the following warnings when installing UCP 3.3.3 using launchpad:

Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Nov 26 08:54:16 myvm firewalld[9577]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

Can these warnings be safely ignored?

$ launchpad version
version: 1.1.0-rc4
commit: 301a9cc

Thanks.

Version on DL is not showing 1.0

Official download version(s) not showing 1.0

is the following expected?

Checked mac & linux DLs

$ launchpad version
version: 0.11.3
commit: 4056b8e

Launchpad chokes on Terraform's output (AWS examples)

Reference documentation: https://github.com/Mirantis/launchpad/blob/master/examples/terraform/aws/README.md

The example states to run Launchpad as follows: terraform output mke_cluster | launchpad apply --config -
The problem is that Terraform's output contains EOT markers:

user@machine:~/launchpad/examples/terraform/aws$ terraform output mke_cluster 
<<EOT
"apiVersion": "launchpad.mirantis.com/mke/v1.1"
"kind": "mke"
"spec":
  "hosts":
[snip]
...
[snip]
EOT
user@machine:~/launchpad/examples/terraform/aws$ 

Launchpad does not like the EOT markers.

A workaround is:

1. terraform output mke_cluster | tee launchpad.yaml
2. edit launchpad.yaml and remove both EOT lines
3. launchpad apply

Mirantis windows worker node issue (Stuck on down(active) status))

Hi Folks,

@kke There is an error on cluster message after adding the windows node to the cluster, although Linux worker nodes are working absolutely fine, windows server 2019 data center node is added but stuck on cluster message error "Node-local UCP component state unknown", What is the reason behind this error. On CLI with command docker docker node ls nodes are ready and active but from MKE web UI it is down(active), Kindly respond asap.

MKE VERSION: 3.4.4
DOCKER VERSION: 20.10.7

Attached is the error ss:

"launchpad init" creates a template with incompatible versions

Version used:

# launchpad version
version: 1.5.7-tp4
commit: 3c5a5336b9dc025676d4250380c401680aa632f8

The template that is created mentions (among other things):

  mke:
    version: 3.7.9
  mcr:
    version: 20.10.13

but first 20.10.13 is not even provided anymore by Mirantis' repos, and by updating this value to 20.10.21 (still available), then an error message is displayed later which states "MKE 3.7.9 is not compatible with Docker 20.10".

Another small issue is that launchpad 1.5.7 displays for each command "A new version (v1.5.7) of launchpad is available [etc.]" :)

Thank you!

Consistency needed in release names and URLs

Hello,

I am trying to develop an asdf plugin to download and install launchpad versions. I've been having some problems with the consistency in the download urls starting with the 1.5 releases.

1.5.0/launchpad-linux-x64
1.5.1/launchpad-linux-x64
1.5.2/launchpad-linux-x64
v1.5.3/launchpad-linux-x64
v1.5.4/launchpad-linux-x64
v1.5.6/launchpad_linux_amd64_1.5.6

The urls have changed three times since 1.5, with the architecture either being x64 or amd64, the version either starting with a v or not and the filenames either using hypens or underscores. Add to this windows downloads either containing "windows" or "win".

Will there be some consistency in the release url names from now on? Could the current scheme be followed from now on?

Thanks.

AWS terraform example seems oudated and shed some errors

Trying the aws onboarding example from scratch I see 2 errors and 1 warning. I have noticed that the last commit was from 2021, maybe a review and update of the script is required.
FYI @ben Dorman

➜  aws git:(master) terraform apply
╷
│ Warning: Argument is deprecated
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 3, in provider "aws":
│    3:   shared_credentials_file = var.aws_shared_credentials_file
│ 
│ Use shared_credentials_files instead.
│ 
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: Attribute Deprecated
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 3, in provider "aws":
│    3:   shared_credentials_file = var.aws_shared_credentials_file
│ 
│ Use shared_credentials_files instead.
│ 
│ (and one more similar warning elsewhere)
╵
╷
│ Error: error configuring Terraform AWS Provider: failed to get shared config profile, kaas
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 1, in provider "aws":
│    1: provider "aws" {
│ 

launchpad fails to get mirantis repos set up on target nodes

I have set up a couple of hosts on a vsphere environment in order to get MKE installed. The hosts fulfil all the basic requirements but once I start launchpad with this configuration file:

apiVersion: launchpad.mirantis.com/mke/v1.3
kind: mke
metadata:
  name: my-mke-cluster
spec:
  hosts:
  - ssh:
      address: 172.31.60.113
      user: mkarg
      port: 22
      keyPath: ~/.ssh/id_rsa
    role: manager
  - ssh:
      address: 172.31.60.123
      user: mkarg
      port: 22
      keyPath: ~/.ssh/id_rsa
    role: worker
  mke:
    version: 3.6
    adminUsername: admin
    adminPassword: password
  mcr:
    version: 20.10.0
  cluster:
    prune: false

it gets to running installer and gets stuck there. On the target hosts we can see this:

  13576 pts/1    Ss+    0:00          \_ bash /home/mkarg/installerLinux529712060
  13876 pts/1    S+     0:00              \_ bash /home/mkarg/installerLinux529712060
  14007 pts/1    S+     0:00                  \_ sudo -E sh -c add-apt-repository 'deb [arch=amd64] https://repos.mirantis.com/ubuntu jammy stable' >/dev/null
  14008 pts/2    Ss     0:00                      \_ sudo -E sh -c add-apt-repository 'deb [arch=amd64] https://repos.mirantis.com/ubuntu jammy stable' >/dev/null
  14009 pts/2    S+     0:00                          \_ sh -c add-apt-repository 'deb [arch=amd64] https://repos.mirantis.com/ubuntu jammy stable' >/dev/null
  14010 pts/2    S+     0:00                              \_ /usr/bin/python3 /usr/bin/add-apt-repository deb [arch=amd64] https://repos.mirantis.com/ubuntu jammy stable

Logs on the launchpad node reveal this:

INFO [ssh] 172.31.60.123:22: running installer
DEBU [ssh] 172.31.60.123:22: executing `DOCKER_URL=https://repos.mirantis.com CHANNEL=stable VERSION=20.10.0 bash /home/mkarg/installerLinux1128266021`
DEBU [ssh] 172.31.60.113:22: + sudo -E sh -c 'apt-get update -qq'
DEBU [ssh] 172.31.60.123:22: + sudo -E sh -c 'apt-get update -qq'
DEBU [ssh] 172.31.60.113:22: + sudo -E sh -c 'apt-get install -y -qq apt-transport-https ca-certificates curl software-properties-common >/dev/null'
DEBU [ssh] 172.31.60.113:22: curl: (22) The requested URL returned error: 404
DEBU [ssh] 172.31.60.113:22: + sudo -E sh -c 'curl -fsSL https://repos.mirantis.com/ubuntu/gpg | apt-key add -qq - >/dev/null'
DEBU [ssh] 172.31.60.123:22: + sudo -E sh -c 'apt-get install -y -qq apt-transport-https ca-certificates curl software-properties-common >/dev/null'
DEBU [ssh] 172.31.60.113:22: Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
DEBU [ssh] 172.31.60.113:22: + sudo -E sh -c 'add-apt-repository '\''deb [arch=amd64] https://repos.mirantis.com/ubuntu jammy stable'\'' >/dev/null'
DEBU [ssh] 172.31.60.123:22: curl: (22) The requested URL returned error: 404
DEBU [ssh] 172.31.60.123:22: + sudo -E sh -c 'curl -fsSL https://repos.mirantis.com/ubuntu/gpg | apt-key add -qq - >/dev/null'
DEBU [ssh] 172.31.60.123:22: Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
DEBU [ssh] 172.31.60.123:22: + sudo -E sh -c 'add-apt-repository '\''deb [arch=amd64] https://repos.mirantis.com/ubuntu jammy stable'\'' >/dev/null'

I tried to download the repos manually and that works fine on the hosts, so it's very likely not a connectivity issue.
Any help is greatly appreciated!
Thanks.

terraform 'map' function has been deprecated and replaced with 'tomap' function.

The 'map' function has been deprecated and replaced with 'tomap' function. All terraform files that contain this function should be updated to reflect this and also so that the examples terraform files work out of the box.

https://www.terraform.io/docs/language/functions/tomap.html

With the current examples you will not be able to continue with the deployment of infra. on AWS for example until such a change is made.

Http_proxy environment variable adding /n to Http-proxy.conf file

Using launchpad version 0.12.1 when I add a http_proxy environment variable to a host during the install it adds a /n to the Http-proxy.conf file.

I have run dos2unix on the cluster.yaml file to make sure it unix format and have run launchpad from MacOS and Linux.

This is on a Ubuntu 18.04.4 system

cluster.yml: changing engine version doesn't affect installation/upgrade/downgrade

tried two different combinations. Seems to default to latest 19.03.8.

  engine:
    version: 19.03.05 

  engine:
    version: 19.03.5 

Deleting a worker node then re-adding fails via

Hello,

Using the launchpad v1.2.0 binary:

Scenario. Removing a node via the:

cluster:
    prune: True

section in the launchpad.yml file. The debuging output shows me the command run on the manager node running launchpad:

docker node rm --force 6sux44bz58xf2zq4tf4c95b3u

This is OK, but I think there needs to be a command to run on the remote target:

$ docker swarm leave --force

because if I try to re-add the node by running launchpad with the node in play back in the file again, it fails, because it still reports it's node as the stale 6sux44bz58xf2zq4tf4c95b3u node id. i.e. on the node to be added back before launchpad is run:

$ docker info --format "{{.Swarm.NodeID}}"
6sux44bz58xf2zq4tf4c95b3u

This was repeatable 100% of the time I tried it.

Cheers,
-Patrick

TLS config: error while exec-ing into the container

Hi there,

# export http_proxy=http://127.0.0.1:3128
# export https_proxy=http://127.0.0.1:3128
#  ./launchpad --debug download-bundle -u [REDACTED] -p [REDACTED] -c cluster.yml 
DEBU[0000] opened config file from /root/cluster.yml
FATA[0002] error getting TLS config: error while exec-ing into the container: Process exited with status 1
# ./launchpad version
version: 0.11.3
commit: 4056b8e

I don't have too much to add beyond that. The debug flag is not revealing much,

Thanks!

launchpad V1.3.0 seg faults when https_proxy is not set to check for downloads

Hi Folks,

Host is RHEL 7.9

# ./launchpad-linux-x64 version
version: 1.3.0
commit: 706d2b4
# uname -a
Linux localhost 3.10.0-1160.25.1.el7.x86_64 #1 SMP Tue Apr 13 18:55:45 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

steps to replicate:

# export https_proxy=proxy:8080
# ./launchpad-linux-x64 download-launchpad
FATA[0000] No upgrade available

unset the proxy:

# ./launchpad-linux-x64 apply --debug --disable-upgrade-check --accept-license --disable-telemetry --config launchpad.yml
DEBU analytics disabled, not tracking event 'Cluster Apply Started'
DEBU opened config file from /root/docker_install/launchpad.yml
DEBU migrated configuration from launchpad.mirantis.com/v1.2 to launchpad.mirantis.com/mke/v1.3
INFO Note: The configuration has been migrated from a previous version
INFO       to see the migrated configuration use: launchpad describe config
DEBU loaded configuration:
apiVersion: launchpad.mirantis.com/mke/v1.4
kind: mke+msr
metadata:
:
:  REDACTED COMPANY INFO, but emits the YAML configuration file
: 
DEBU preparing phase 'Check For Upgrades'
INFO ==> Running phase: Check For Upgrades
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x754e55]

goroutine 1 [running]:
github.com/Mirantis/mcc/pkg/docker/hub.LatestTag(0xa55d20, 0x8, 0xa510b8, 0x3, 0xffffffffffffff00, 0x0, 0x0, 0x0, 0x0)
        github.com/Mirantis/mcc/pkg/docker/hub/hub.go:37 +0x2d5
github.com/Mirantis/mcc/pkg/product/mke/phase.(*UpgradeCheck).Run(0xc0004ec320, 0xb8529d, 0xeddce0)
        github.com/Mirantis/mcc/pkg/product/mke/phase/upgrade_check.go:25 +0xa6
github.com/Mirantis/mcc/pkg/phase.(*Manager).Run(0xc0000e7970, 0x0, 0x0)
        github.com/Mirantis/mcc/pkg/phase/manager.go:91 +0x293
github.com/Mirantis/mcc/pkg/product/mke.(*MKE).Apply(0xc0004e4540, 0xa50000, 0x5, 0x0)
        github.com/Mirantis/mcc/pkg/product/mke/apply.go:58 +0x929
github.com/Mirantis/mcc/cmd.NewApplyCommand.func1(0xc0002e1c00, 0x0, 0x0)
        github.com/Mirantis/mcc/cmd/apply.go:72 +0x28f
github.com/urfave/cli/v2.(*Command).Run(0xc000021320, 0xc0002e1740, 0x0, 0x0)
        github.com/urfave/cli/[email protected]/command.go:164 +0x4dd
github.com/urfave/cli/v2.(*App).RunContext(0xc0002f0480, 0xb1b1d0, 0xc0000260a0, 0xc00001e080, 0x8, 0x8, 0x0, 0x0)
        github.com/urfave/cli/[email protected]/app.go:306 +0x810
github.com/urfave/cli/v2.(*App).Run(...)
        github.com/urfave/cli/[email protected]/app.go:215
main.main()
        command-line-arguments/main.go:95 +0x476
# 

However, if I set the http_proxy variable the command works. Why is the launchpad binary checking for upgrades despite the --disable-upgrade-checks flag being set?

Thanks!