saltstack-formulas / opensds-formula

Integration with Open Software Defined Storage - Capri release - SuSE/CentOS/Ubuntu

opensds-formula

Deploy official releases of OpenSDS (www.opensds.io) using Salt on CENTOS-7, UBUNTU-18, and OPENSUSE-15. This is an experimental solution using repeatable patterns to deploy cloud-native stack using infrastructure as code. This formula compliments the OpenSDS-Installer project.

See the full Salt Formulas installation and usage instructions.

Architectural View

Available META states

opensds

Runs all the other states in the formula. Used by the OpenSDS-installer/salt module.

opensds.infra

Deploy os profile (PATHS) and environmental dependencies (packages, nginx, docker, etc) via salt.

opensds.telemetry -----------------

Deploy prometheus and grafana via salt.

opensds.keystone -----------------

Deploy devstack with keystone configuration for hotpot and gelato.

opensds.config -----------------

Deploy opensds configuration file

opensds.hotpot -----------------

Deploy opensds hotpot

opensds.auth

Deploy authentication service (default keystone).

opensds.database -----------------

Deploy database service (default etcd).

opensds.dock

Deploy osdsdock service.

opensds.sushi

Deploy osdsnbp service.

opensds.gelato -----------------

Deploy multi-cloud service.

opensds.dashboard -----------------

Deploy Dashboard service.

opensds.freespace -----------------

Free some disk space

Site-specific Data Collection ================

The site.j2 and pillar.example files contain required pillars!

You may review and cautiously update site.j2 to reflect site requirements and preferences.

Prerequisite

Prepare your environment by running the salt.formulas state from https://github.com/saltstack-formulas/salt-formula. See pillar.example and/or opensds-installer/salt README.

opensds-formula's Issues

[CentOS] curl#6 - "Could not resolve host: mirrorlist.centos.org;

Make blockbox fails. My internet connection is okay; name resolution works?

          ID: opensds backend block daemon cinder software build make blockbox
    Function: cmd.run
        Name: make blockbox
      Result: False
     Comment: Command "make blockbox" run
     Started: 09:37:46.534821
    Duration: 129760.935 ms
     Changes:   
              ----------
              pid:
                  25817
              retcode:
                  2
              stderr:
                  The command '/bin/sh -c /opt/loci/scripts/install.sh' returned a non-zero code: 1
                  make: *** [base] Error 1
              stdout:
                  docker build https://git.openstack.org/openstack/loci.git \
                  	--build-arg PROJECT=cinder \
                  	--build-arg FROM=centos:7 \
                  	--build-arg PROJECT_REF=master \
                  	--tag debian-cinder:latest
                  Sending build context to Docker daemon  250.9kB
                  
                  Step 1/26 : ARG FROM=ubuntu:xenial
                  Step 2/26 : FROM ${FROM}
                  7: Pulling from library/centos
                  a02a4930cb5d: Pulling fs layer
                  a02a4930cb5d: Download complete
                  a02a4930cb5d: Pull complete
                  Digest: sha256:184e5f35598e333bfa7de10d8fb1cebb5ee4df5bc0f970bf2b1e7c7345136426
                  Status: Downloaded newer image for centos:7
                   ---> 1e1148e4cc2c
                  Step 3/26 : ENV PATH=/var/lib/openstack/bin:$PATH
                   ---> Running in f251a479b1f4
                  Removing intermediate container f251a479b1f4
                   ---> ea5dd140fc6e
                  Step 4/26 : ARG PROJECT
                   ---> Running in c9b5f5bcb6d2
                  Removing intermediate container c9b5f5bcb6d2
                   ---> e645f35066e8
                  Step 5/26 : ARG WHEELS=loci/requirements:master-ubuntu
                   ---> Running in c5591a391c16
                  Removing intermediate container c5591a391c16
                   ---> 7eefb2db8cd6
                  Step 6/26 : ARG PROJECT_REPO=https://git.openstack.org/openstack/${PROJECT}
                   ---> Running in 1c4ffb8b68a6
                  Removing intermediate container 1c4ffb8b68a6
                   ---> 79303fa7bc48
                  Step 7/26 : ARG PROJECT_REF=master
                   ---> Running in d72d7e855495
                  Removing intermediate container d72d7e855495
                   ---> b5246ca44fba
                  Step 8/26 : ARG DISTRO
                   ---> Running in 2b0152dff469
                  Removing intermediate container 2b0152dff469
                   ---> 61a9404f23c8
                  Step 9/26 : ARG PROFILES
                   ---> Running in 68bf940f2aa0
                  Removing intermediate container 68bf940f2aa0
                   ---> dbb1f3e7b0cc
                  Step 10/26 : ARG PIP_PACKAGES=""
                   ---> Running in 7cafb9d76843
                  Removing intermediate container 7cafb9d76843
                   ---> 7b9e11f53f2f
                  Step 11/26 : ARG PIP_ARGS=""
                   ---> Running in e46625a239bc
                  Removing intermediate container e46625a239bc
                   ---> dce293f86882
                  Step 12/26 : ARG PIP_WHEEL_ARGS=$PIP_ARGS
                   ---> Running in beb16116ba7d
                  Removing intermediate container beb16116ba7d
                   ---> 1855d7a232a9
                  Step 13/26 : ARG DIST_PACKAGES=""
                   ---> Running in eb8f572573cc
                  Removing intermediate container eb8f572573cc
                   ---> de684274d720
                  Step 14/26 : ARG PLUGIN=no
                   ---> Running in 0da837edccb1
                  Removing intermediate container 0da837edccb1
                   ---> c56634d7e0f4
                  Step 15/26 : ARG PYTHON3=no
                   ---> Running in 9f27713d4576
                  Removing intermediate container 9f27713d4576
                   ---> 1e7873b9544c
                  Step 16/26 : ARG EXTRA_BINDEP=""
                   ---> Running in 1643051e45a6
                  Removing intermediate container 1643051e45a6
                   ---> a92b0c4694d2
                  Step 17/26 : ARG EXTRA_PYDEP=""
                   ---> Running in a106c729f3d5
                  Removing intermediate container a106c729f3d5
                   ---> 0c2726ce6efc
                  Step 18/26 : ARG UID=42424
                   ---> Running in b5d69f7c78fc
                  Removing intermediate container b5d69f7c78fc
                   ---> 5c004ec16d9c
                  Step 19/26 : ARG GID=42424
                   ---> Running in a3e3b14fe58d
                  Removing intermediate container a3e3b14fe58d
                   ---> 10bfe58e93c7
                  Step 20/26 : ARG NOVNC_REPO=https://github.com/novnc/novnc
                   ---> Running in 13386469d2d4
                  Removing intermediate container 13386469d2d4
                   ---> c50c1b95d134
                  Step 21/26 : ARG NOVNC_REF=v1.0.0
                   ---> Running in a5b09a2647c4
                  Removing intermediate container a5b09a2647c4
                   ---> 5005e3b8632b
                  Step 22/26 : ARG SPICE_REPO=https://gitlab.freedesktop.org/spice/spice-html5.git
                   ---> Running in ffd93b26b8fb
                  Removing intermediate container ffd93b26b8fb
                   ---> 099ac0eff250
                  Step 23/26 : ARG SPICE_REF=spice-html5-0.1.6
                   ---> Running in d360f460eff6
                  Removing intermediate container d360f460eff6
                   ---> 915d2777c532
                  Step 24/26 : COPY scripts /opt/loci/scripts
                   ---> 72be9c3646e4
                  Step 25/26 : ADD bindep.txt pydep.txt $EXTRA_BINDEP $EXTRA_PYDEP /opt/loci/
                   ---> 56eb751d3f64
                  Step 26/26 : RUN /opt/loci/scripts/install.sh
                   ---> Running in 5cc0bb7ac0e8
                  ?[91m++ awk -F= '/^ID=/ {gsub(/\"/, "", $2); print $2}' /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release
                  ?[0m?[91m+ distro=centos
                  + export distro=centos
                  + distro=centos
                  + [[ no == \n\o ]]
                  + dpkg_python_packages=("python" "virtualenv")
                  + rpm_python_packages=("python" "python-virtualenv")
                  + case ${distro} in
                  + yum upgrade -y
                  ?[0mLoaded plugins: fastestmirror, ovl
                  Determining fastest mirrors
                  Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
                  14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
                  ?[91m
                  
                   One of the configured repositories failed (Unknown),
                   and yum doesn't have enough cached data to continue. At this point the only
                   safe thing yum can do is fail. There are a few ways to work "fix" this:
                  
                       1. Contact the upstream for the repository and get them to fix the problem.
                  
                       2. Reconfigure the baseurl/etc. for the repository, to point to a working
                          upstream. This is most often useful if you are using a newer
                          distribution release than is supported by the repository (and the
                          packages for the previous distribution release still work).
                  
                       3. Run the command with the repository temporarily disabled
                              yum --disablerepo=<repoid> ...
                  
                       4. Disable the repository permanently, so yum won't use it by default. Yum
                          will then just ignore the repository until you permanently enable it
                          again or use --enablerepo for temporary usage:
                  
                              yum-config-manager --disable <repoid>
                          or
                              subscription-manager repos --disable=<repoid>
                  
                       5. Configure the failing repository to be skipped, if it is unavailable.
                          Note that yum will try to contact the repo. when it runs most commands,
                          so will have to try and fail each time (and thus. yum will be be much
                          slower). If it is a very temporary problem though, this is often a nice
                          compromise:
                  
                              yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
                  
                  Cannot find a valid baseurl for repo: base/7/x86_64
                  ?[0m

osdslet daemon failure

Running osdslet as daemon fails with error logged

[root@localhost opensds]# nohup /opt/opensds-linux-amd64/bin/osdslet >/var/log/opensds/osdslet.out 2> /var/log/opensds/osdslet.err &

[root@localhost opensds]# cat osdslet.localhost.root.log.ERROR.20181214-213525.2829 
Log file created at: 2018/12/14 21:35:25
Running on machine: localhost
Binary: Built with gc go1.10.4 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1214 21:35:25.442800    2829 keystone.go:60] When get auth client:You must provide a password to authenticate

[ubuntu] some systemd tweaks needed

The installation of OpenSDS is good but systemd status is bit funny for multicloud and nbp. This is not a major issue but should be corrected.

root@ubuntu1804:/root/opensds-installer/salt# systemctl list-unit-files | grep opensds
opensds-multi-cloud.service                                      enabled        
opensds-nbp.service                                              enabled        
opensds-opensds.service                                          enabled        
root@ubuntu1804:/root/opensds-installer/salt# systemctl status opensds-multi-cloud.service
* opensds-multi-cloud.service - OpenSDS multi-cloud service
   Loaded: loaded (/usr/lib/systemd/system/opensds-multi-cloud.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-01-23 05:16:20 MST; 18min ago
     Docs: https://opensds.io
  Process: 31233 ExecStop=/usr/bin/killall opensds-multi-cloud (code=exited, status=1/FAILURE)
  Process: 28610 ExecStart=/usr/local/bin/docker-compose up -d (code=exited, status=0/SUCCESS)
 Main PID: 28610 (code=exited, status=0/SUCCESS)

Jan 23 05:16:09 ubuntu1804.localdomain docker-compose[28610]: Creating multi-cloud_api_1       ...
Jan 23 05:16:09 ubuntu1804.localdomain docker-compose[28610]: Creating multi-cloud_s3_1        ...
Jan 23 05:16:09 ubuntu1804.localdomain docker-compose[28610]: Creating multi-cloud_backend_1   ...
Jan 23 05:16:16 ubuntu1804.localdomain docker-compose[28610]: [311B blob data]
Jan 23 05:16:17 ubuntu1804.localdomain docker-compose[28610]: [91B blob data]
Jan 23 05:16:17 ubuntu1804.localdomain docker-compose[28610]: Creating multi-cloud_datamover_1 ...
Jan 23 05:16:20 ubuntu1804.localdomain docker-compose[28610]: [110B blob data]
Jan 23 05:16:20 ubuntu1804.localdomain killall[31233]: opensds-multi-cloud: no process found
Jan 23 05:16:20 ubuntu1804.localdomain systemd[1]: opensds-multi-cloud.service: Control process exited, code=exited status=1
Jan 23 05:16:20 ubuntu1804.localdomain systemd[1]: opensds-multi-cloud.service: Failed with result 'exit-code'.
root@ubuntu1804:/root/opensds-installer/salt# systemctl status opensds-nbp.service 
* opensds-nbp.service - OpenSDS nbp service
   Loaded: loaded (/usr/lib/systemd/system/opensds-nbp.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-01-23 05:11:16 MST; 23min ago
     Docs: https://opensds.io
 Main PID: 19424 (code=exited, status=1/FAILURE)

Jan 23 05:11:15 ubuntu1804.localdomain systemd[1]: Started OpenSDS nbp service.
Jan 23 05:11:15 ubuntu1804.localdomain kubectl[19424]: The connection to the server localhost:8080 was refused - did you specify the right host or port?
Jan 23 05:11:16 ubuntu1804.localdomain systemd[1]: opensds-nbp.service: Main process exited, code=exited, status=1/FAILURE
Jan 23 05:11:16 ubuntu1804.localdomain systemd[1]: opensds-nbp.service: Failed with result 'exit-code'.
root@ubuntu1804:/root/opensds-installer/salt# systemctl status opensds-opensds.service 
* opensds-opensds.service - OpenSDS opensds service
   Loaded: loaded (/usr/lib/systemd/system/opensds-opensds.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-01-23 05:06:08 MST; 29min ago
     Docs: https://opensds.io
 Main PID: 17548 (osdslet)
   CGroup: /system.slice/opensds-opensds.service
           `-17548 /usr/local/bin/osdslet

Jan 23 05:25:02 ubuntu1804.localdomain osdslet[17548]: E0123 05:25:02.570268   17548 controller.go:191] when search profile in db:Wrong resource uuid provided!
Jan 23 05:25:02 ubuntu1804.localdomain osdslet[17548]: E0123 05:25:02.570648   17548 etcd.go:1012] When get volume in db:Wrong resource uuid provided!
Jan 23 05:25:02 ubuntu1804.localdomain osdslet[17548]: E0123 05:25:02.570910   17548 etcd.go:2238] When update volume status in db:specified volume(86d86894-d473-45b3-a3d8-9515686f5fc4) can't find
Jan 23 05:25:02 ubuntu1804.localdomain osdslet[17548]: E0123 05:25:02.570924   17548 volume.go:294] Delete volume failed: specified volume(86d86894-d473-45b3-a3d8-9515686f5fc4) can't find
Jan 23 05:25:02 ubuntu1804.localdomain osdslet[17548]: 2019/01/23 05:25:02.570 [D] 127.0.0.1 - - [23/Jan/2019 05:25:02] "DELETE /v1beta/e93b4c0934da416eb9c8d120c5d04d96/block/volumes/86d86894-d473-45b3-a3d8
Jan 23 05:25:02 ubuntu1804.localdomain osdslet[17548]: 
Jan 23 05:25:05 ubuntu1804.localdomain osdslet[17548]: 2019/01/23 05:25:05.780 [D] 127.0.0.1 - - [23/Jan/2019 05:25:05] "GET /v1beta/e93b4c0934da416eb9c8d120c5d04d96/pools?limit=50&offset=0&sortDir=desc&sor
Jan 23 05:25:05 ubuntu1804.localdomain osdslet[17548]: 
Jan 23 05:25:16 ubuntu1804.localdomain osdslet[17548]: 2019/01/23 05:25:16.512 [D] 127.0.0.1 - - [23/Jan/2019 05:25:16] "GET /v1beta/e93b4c0934da416eb9c8d120c5d04d96/block/volumes?sortDir=desc&sortKey=id&li
Jan 23 05:25:16 ubuntu1804.localdomain osdslet[17548]: 

[BUG] make blockbox fails on CentOS7

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Pillar / config used

---

Bug details

Describe the bug

make lockbox fails on CentOS7

ID: opensds backend block daemon cinder software build make blockbox
    Function: cmd.run
        Name: make blockbox
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "make blockbox" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "make blockbox" run"
              Command "make blockbox" run
     Started: 15:23:41.096769
    Duration: 5691818.388 ms
     Changes:
              ----------
              pid:
                  16785
              retcode:
                  2
              stderr:
                  The command '/bin/sh -c /opt/loci/scripts/install.sh' returned a non-zero code: 1
                  make: *** [base] Error 1
....
                  Collecting python-editor===1.0.4
                    Downloading python_editor-1.0.4-py2-none-any.whl (4.9 kB)
                  Collecting python-dateutil===2.8.0
                    Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
                  Collecting pytz===2019.2
                    Downloading pytz-2019.2-py2.py3-none-any.whl (508 kB)
                  ?[91mERROR: Could not find a version that satisfies the requirement tabulate>=0.8.5 (from cinder==16.0.0.0b2.dev98) (from versions: 0.7.6b0, 0.7.7)
                  ?[0m?[9

osdslet container error on Ubuntu/Centos

root@ubuntu1804:~# docker ps -a
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                      PORTS                                             NAMES
21f3bdbfb215        opensdsio/dashboard:latest             "/bin/sh -c /opt/das…"   9 hours ago         Up 2 hours                                                                    dashboard
b7e411103870        opensdsio/opensds-dock:latest          "/usr/bin/osdsdock"      9 hours ago         Up 11 minutes                                                                 osdsdock
1083acf9f4d1        gcr.io/etcd-development/etcd:v3.2.24   "etcd -name etcd0 -a…"   9 hours ago         Up 12 minutes               2379-2380/udp, 0.0.0.0:2379-2380->2379-2380/tcp   osdsdb-etcd
3e7d23ad9355        opensdsio/opensds-controller:latest    "/usr/bin/osdslet"       10 hours ago        Exited (2) 10 minutes ago                                                     opensds-controller


root@ubuntu1804:~# docker logs 3e7d23ad9355
2018/12/11 01:43:55 [ERROR] Read configuration failed, use default value
E1211 01:43:58.623265       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 1 time(s)
E1211 01:44:01.624393       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2380: connect: connection refused, retry 2 time(s)
E1211 01:44:04.624930       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 3 time(s)
E1211 01:44:04.624954       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: dial tcp 127.0.0.1:2379: connect: connection refused

goroutine 1 [running]:
github.com/opensds/opensds/pkg/db/drivers/etcd.Init(0xc0001b0700, 0x2, 0x2, 0x2)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/client.go:70 +0x110
github.com/opensds/opensds/pkg/db/drivers/etcd.NewClient(0xc0001b0700, 0x2, 0x2, 0x1)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/etcd.go:64 +0x43
github.com/opensds/opensds/pkg/db.Init(0xc000123e70)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/db.go:45 +0x1a6
main.main()
	/root/gopath/src/github.com/opensds/opensds/cmd/osdslet/osdslet.go:50 +0x6d
2018/12/11 11:07:00 [ERROR] Read configuration failed, use default value
E1211 11:07:03.580288       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 1 time(s)
E1211 11:07:06.582396       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 2 time(s)
E1211 11:07:09.582704       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 3 time(s)
E1211 11:07:09.582767       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: dial tcp 127.0.0.1:2379: connect: connection refused

goroutine 1 [running]:
github.com/opensds/opensds/pkg/db/drivers/etcd.Init(0xc00014e020, 0x2, 0x2, 0x2)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/client.go:70 +0x110
github.com/opensds/opensds/pkg/db/drivers/etcd.NewClient(0xc00014e020, 0x2, 0x2, 0x1)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/etcd.go:64 +0x43
github.com/opensds/opensds/pkg/db.Init(0xc000121e70)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/db.go:45 +0x1a6
main.main()
	/root/gopath/src/github.com/opensds/opensds/cmd/osdslet/osdslet.go:50 +0x6d

root@ubuntu1804:~# curl 127.0.0.1:2379
curl: (56) Recv failure: Connection reset by peer

root@ubuntu1804:~# telnet 127.0.0.1 2379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

root@ubuntu1804:~# netstat -a | grep 2379 | wc -l
15

cinderbox jinja rendering error (on archlinux)

Seen on archlinux

run salt: this takes a while, please be patient ...
SaltRenderError: Jinja variable 'dict object' has no attribute 'platform'
[CRITICAL] Rendering SLS 'base:opensds.backend.block.config.cinderbox' failed: Jinja variable 'dict object' has no attribute 'platform'
local:
    Data failed to compile:

Make cinder blockbox fails on CentOS.

Needs to be customized for CentOS

          ID: opensds backend block daemon cinder software build make blockbox
    Function: cmd.run
        Name: make blockbox
      Result: False
     Comment: Command "make blockbox" run
     Started: 03:14:00.080141
    Duration: 3173.534 ms
     Changes:
              ----------
              pid:
                  24647
              retcode:
                  2
              stderr:
                  unable to prepare context: unable to 'git clone' to temporary context directory: error initializing submodules: usage: git submodule [--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]
                     or: git submodule [--quiet] status [--cached] [--recursive] [--] [<path>...]
                     or: git submodule [--quiet] init [--] [<path>...]
                     or: git submodule [--quiet] deinit [-f|--force] [--] <path>...
                     or: git submodule [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
                     or: git submodule [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
                     or: git submodule [--quiet] foreach [--recursive] <command>
                     or: git submodule [--quiet] sync [--recursive] [--] [<path>...]
                  : exit status 1
                  make: *** [base] Error 1
              stdout:
                  docker build https://git.openstack.org/openstack/loci.git \
                        --build-arg PROJECT=cinder \
                        --build-arg FROM=debian:stretch \
                        --build-arg PROJECT_REF=master \
                        --tag debian-cinder:latest

autoremove old packages on ubuntu

Cleanup old packages

The following packages were automatically installed and are no longer required:
  gyp libjs-async libjs-inherits libjs-node-uuid libpython-all-dev libuv1-dev node-abbrev node-ansi node-ansi-color-table node-archy
  node-async node-balanced-match node-block-stream node-brace-expansion node-builtin-modules node-combined-stream node-concat-map
  node-cookie-jar node-delayed-stream node-forever-agent node-form-data node-fs.realpath node-fstream node-fstream-ignore
  node-github-url-from-git node-glob node-graceful-fs node-hosted-git-info node-inflight node-inherits node-ini node-is-builtin-module
  node-isexe node-json-stringify-safe node-lockfile node-lru-cache node-mime node-minimatch node-mkdirp node-mute-stream node-node-uuid
  node-nopt node-normalize-package-data node-npmlog node-once node-osenv node-path-is-absolute node-pseudomap node-qs node-read
  node-read-package-json node-request node-retry node-rimraf node-semver node-sha node-slide node-spdx-correct node-spdx-expression-parse
  node-spdx-license-ids node-tar node-tunnel-agent node-underscore node-validate-npm-package-license node-which node-wrappy node-yallist
  python-all python-all-dev python-dbus python-gi python-keyring python-keyrings.alt python-secretstorage python-wheel python-xdg
Use 'apt autoremove' to remove them.

tracker: blockbox container failures [ctypes]

Tracking: https://bugs.launchpad.net/openstack-loci/+bug/1815581

Blocking Salt issues on Suse

Blocking issues:

• saltstack-formulas/golang-formula#14
• saltstack-formulas/docker-formula#198

minor issues:

• saltstack-formulas/ceph-formula#20

[BUG] gelato on freeBSD - go not found

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Pillar / config used

---

Bug details

Describe the bug

There is a failed state on FreeBSD.

 ID: opensds gelato daemon multi-cloud software build make docker
    Function: cmd.run
        Name: make docker
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "make docker" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "make docker" run"
              Command "make docker" run
     Started: 03:10:19.502800
    Duration: 129112.585 ms
     Changes:
              ----------
              pid:
                  33404
              retcode:
                  1
              stderr:
                  /bin/sh: go: not found
              stdout:
                  mkdir -p  /build
                  CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o /build/api github.com/opensd
s/multi-cloud/api/cmd
                  *** Error code 127

                  Stop.
                  make: stopped in /usr/local/golang/packages/src/github.com/opensds/multi-cloud

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

[centos] multi-cloud systemd service fails

This was initial deployment of OpenSDS on CentOS7

----------
          ID: opensds gelato daemon systemd multi-cloud service started
    Function: service.running
        Name: opensds-multi-cloud
      Result: False
     Comment: Service opensds-multi-cloud has been enabled, and is dead
     Started: 17:40:58.915387
    Duration: 297.844 ms
     Changes:   
              ----------
              opensds-multi-cloud:
                  True

Logs

[root@localhost multi-cloud]# docker-compose up -d
Creating network "multicloud_default" with the default driver
ERROR: Failed to program FILTER chain: iptables failed: iptables --wait -I FORWARD 
-o br-0a7584956716 -j DOCKER: iptables v1.4.21: 
Couldn't load target `DOCKER':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
 (exit status 2)

[BUG] gelato "make docker" fails Centos

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Pillar / config used

---

Bug details

Describe the bug

See journeymidnight/radoshttpd#2

Steps to reproduce the bug

          ID: opensds gelato daemon multi-cloud software build make docker
    Function: cmd.run
        Name: make docker
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "make docker" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "make docker" run"
              Command "make docker" run
     Started: 18:03:00.684861
    Duration: 134571.248 ms
     Changes:   
              ----------
              pid:
                  1269
              retcode:
                  2
              stderr:
                  # github.com/opensds/multi-cloud/vendor/github.com/journeymidnight/radoshttpd/rados
                  vendor/github.com/journeymidnight/radoshttpd/rados/striper.go:61: cannot use sp.ioctx (type _Ctype_rados_ioctx_t) as type _Ctype_rados_striper_t in argument to func literal
                  make: *** [s3] Error 2
              stdout:
                  mkdir -p  /usr/local/golang/packages/src/github.com/opensds/multi-cloud/build
                  CGO_ENABLED=0 GOOS=linux go build -ldflags '-w -s -extldflags "-static"' -o /usr/local/golang/packages/src/github.com/opensds/multi-cloud/build/api github.com/opensds/multi-cloud/api/cmd
                  CGO_ENABLED=0 GOOS=linux go build -ldflags '-w -s -extldflags "-static"' -o /usr/local/golang/packages/src/github.com/opensds/multi-cloud/build/backend github.com/opensds/multi-cloud/backend/cmd
                  CGO_ENABLED=1 GOOS=linux go build -ldflags '-w -s -extldflags "-dynamic"' -o /usr/local/golang/packages/src/github.com/opensds/multi-cloud/build/s3 github.com/opensds/multi-cloud/s3/cmd

Expected behaviour

Attempts to fix the bug

Additional context

update version

v0.5.2

[archlinux] docker container running exception

 ID: opensds dashboard daemon dashboard container running
    Function: docker_container.running
        Name: dashboard
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/site-packages/salt/state.py", line 1933, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1939, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/site-packages/salt/states/docker_container.py", line 1695, in running
                  image_id = _resolve_image(ret, image, client_timeout)
                File "/usr/lib/python2.7/site-packages/salt/states/docker_container.py", line 193, in _resolve_image
                  image_id = __salt__['docker.resolve_image_id'](image)
                File "/usr/lib/python2.7/site-packages/salt/modules/dockermod.py", line 1911, in resolve_image_id
                  inspect_result = inspect_image(name)
                File "/usr/lib/python2.7/site-packages/salt/modules/dockermod.py", line 1843, in inspect_image
                  ret = _client_wrapper('inspect_image', name)
                File "/usr/lib/python2.7/site-packages/salt/modules/dockermod.py", line 423, in wrapper
                  __context__['docker.client'] = _get_client(timeout=timeout, **kwargs)
                File "/usr/lib/python2.7/site-packages/salt/modules/dockermod.py", line 387, in _get_client
                  ret = docker.APIClient(**client_kwargs)
                File "/usr/lib/python2.7/site-packages/docker/api/client.py", line 128, in __init__
                  config_dict=self._general_configs, credstore_env=credstore_env,
              TypeError: load_config() got an unexpected keyword argument 'config_dict'
     Started: 04:03:00.169246
    Duration: 8.655 ms
     Changes:   

osdsdock - Error: context deadline exceeded

Seems etcd-io/etcd#8793 is causing osdsdock container failures.

	/root/gopath/src/github.com/opensds/opensds/cmd/osdsdock/osdsdock.go:51 +0x70
E1226 00:28:06.994061       1 utils.go:100] Get etcd client:context deadline exceeded, retry 1 time(s)
E1226 00:28:09.995494       1 utils.go:100] Get etcd client:context deadline exceeded, retry 2 time(s)
E1226 00:28:12.996046       1 utils.go:100] Get etcd client:context deadline exceeded, retry 3 time(s)
E1226 00:28:12.996095       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: context deadline exceeded

goroutine 1 [running]:
github.com/opensds/opensds/pkg/db/drivers/etcd.Init(0xc00000c060, 0x2, 0x2, 0x2)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/client.go:70 +0x110
github.com/opensds/opensds/pkg/db/drivers/etcd.NewClient(0xc00000c060, 0x2, 0x2, 0x1)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/etcd.go:64 +0x43
github.com/opensds/opensds/pkg/db.Init(0xc000137070)
	/root/gopath/src/github.com/opensds/opensds/pkg/db/db.go:45 +0x1a6
main.main()
	/root/gopath/src/github.com/opensds/opensds/cmd/osdsdock/osdsdock.go:51 +0x70

[ubuntu] two errors during cleanup

To be fixed

          ID: opensds sushi daemon nbp systemd service removed
    Function: cmd.run
        Name: make clean
      Result: False
     Comment: Command "make clean" run
     Started: 12:22:44.917712
    Duration: 11.427 ms
     Changes:   
              ----------
              pid:
                  30397
              retcode:
                  2
              stderr:
                  make: *** No rule to make target 'clean'.  Stop.
              stdout:


          ID: opensds hotpot daemon opensds systemd service removed
    Function: cmd.run
        Name: make clean
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1913, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1898, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/cmd.py", line 872, in run
                  cret = mod_run_check(cmd_kwargs, onlyif, unless, creates)
                File "/usr/lib/python2.7/dist-packages/salt/states/cmd.py", line 350, in mod_run_check
                  cmd = __salt__['cmd.retcode'](entry, ignore_retcode=True, python_shell=True, **cmd_kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 2088, in retcode
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 603, in _run
                  proc = salt.utils.timed_subprocess.TimedProc(cmd, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/utils/timed_subprocess.py", line 44, in __init__
                  args = salt.utils.stringutils.to_bytes(args)
                File "/usr/lib/python2.7/dist-packages/salt/utils/stringutils.py", line 63, in to_bytes
                  return to_str(s, encoding, errors)
                File "/usr/lib/python2.7/dist-packages/salt/utils/stringutils.py", line 118, in to_str
                  raise TypeError('expected str, bytearray, or unicode')
              TypeError: expected str, bytearray, or unicode
     Started: 12:22:45.373892
    Duration: 2.495 ms
     Changes:   

cannot make dashboard - missing commands

make fails

          ID: opensds dashboard repo build from source
    Function: cmd.run
        Name: make
      Result: False
     Comment: Command "make" run
     Started: 13:34:32.669826
    Duration: 21.219 ms
     Changes:   
              ----------
              pid:
                  29581
              retcode:
                  2
              stderr:
                  ./image_builder.sh: line 21: npm: command not found
                  ./image_builder.sh: line 22: npm: command not found
                  ./image_builder.sh: line 23: ng: command not found
                  cp: cannot stat './dist/*': No such file or directory
                  make: *** [dashboard] Error 1
              stdout:
                  chmod +x ./image_builder.sh \
                    && ./image_builder.sh
                  Makefile:25: recipe for target 'dashboard' failed

long-running cmd.run segmentation fault: (Salt/Cent7)

Salt always segmentation faults on CentOS after stack.sh exits the long running cmd.run state. WORKAROUND: Rerun the high.state (2nd time always works)*

[ERROR   ] retcode: 128
./install.sh: line 137: 18741 Segmentation fault      salt-call state.highstate --local ${DEBUGG_ON}

When I tried to collect logs (-ldebug) the segmentation fault does not occur - thats bizarre!

We need to test devstack-formula on Cent7 to try to replicate.

[root@localhost salt]# salt-call --versions
Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: 5.6.0
       dateutil: 2.7.5
      docker-py: 3.6.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.6.0
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.14.4.el7.x86_64
         system: Linux
        version: CentOS Linux 7.5.1804 Core
```

[centos] [nbp] cannot connect to localhost:8080

Jan 22 17:55:43 localhost.localdomain systemd[1]: Started OpenSDS nbp service.
-- Subject: Unit opensds-nbp.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit opensds-nbp.service has finished starting up.
-- 
-- The start-up result is done.
Jan 22 17:55:43 localhost.localdomain polkitd[675]: Unregistered Authentication Agent for unix-process:11154:495524 (system bus name :1.187, 
Jan 22 17:55:43 localhost.localdomain kubectl[11160]: The connection to the server localhost:8080 was refused - did you specify the right hos
Jan 22 17:55:43 localhost.localdomain systemd[1]: opensds-nbp.service: main process exited, code=exited, status=1/FAILURE
Jan 22 17:55:43 localhost.localdomain killall[11168]: opensds-nbp: no process found
Jan 22 17:55:43 localhost.localdomain systemd[1]: opensds-nbp.service: control process exited, code=exited status=1
Jan 22 17:55:43 localhost.localdomain systemd[1]: Unit opensds-nbp.service entered failed state.
Jan 22 17:55:43 localhost.localdomain systemd[1]: opensds-nbp.service failed.

Salt 2019.2 support missing

Please use Salt 2018.3 or 2017.7 with this formula.

Various states consumed by opensds-formula may be impacted by NON-BACKWARD-COMPATIBLE CHANGE TO YAML RENDERER

[centos] [dashboard] wrong osfamilymap nginx path

Looks like nginx path is wrong in osfamilymap.yaml for CentOS

----------
          ID: opensds dashboard container dashboard running
    Function: docker_container.running
        Name: dashboard
      Result: False
     Comment: Created container 'dashboard'. Failed to start container 'dashboard': 'Error 400: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/etc/nginx/sites-available/default\\\" to rootfs \\\"/var/lib/docker/overlay2/f42989007c5b2b2a27ebc025f570fd6669287a868ead1a5867f5a46b6b766aa6/merged\\\" at \\\"/var/lib/docker/overlay2/f42989007c5b2b2a27ebc025f570fd6669287a868ead1a5867f5a46b6b766aa6/merged/etc/nginx/sites-available/default\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type'.
     Started: 17:31:12.546618
    Duration: 437409.91 ms
     Changes:   
              ----------
              container_id:
                  ----------
                  added:
                      c66ed18ece2ef754c0edb9f74ed44b6636a7c1a0f44efa77daa4f945fdb1706b
              image:
                  ----------
                  Layers:
                      ----------
                      Pulled:
                          - b775667b1fd0
                          - 4b105072aa89
                          - 83ef5c84ec9c
                          - f567c7054a96
                          - 0fe156bcb4b1
                          - 07fa96bb7d5e
                          - 4dd021921c9a
                          - 42dae22f8cd7
                          - e1a8721afe6e
                          - f77ff8405034
                          - 099ee7dd0a81
                          - fb9906355a0f
                          - 3fbdf05660f5
                  Status:
                      Downloaded newer image for opensdsio/dashboard:bali
                  Time_Elapsed:
                      2.55150794983
                  retcode:
                      0

Rendering Error when ipv6 disabled

If ipv6 is disabled then the associated grains are not available: i.e. grains.ipv6[-1]

    Rendering SLS 'base:opensds.dashboard' failed: Jinja variable list object has no element -1
/var/cache/salt/minion/files/base/devstack/defaults.yaml(53):

Failed to start container 'dashboard' on Suse

          ID: opensds dashboard daemon dashboard container running
    Function: docker_container.running
        Name: dashboard
      Result: False
     Comment: Created container 'dashboard'. Failed to start container 'dashboard': 'Error 400: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/etc/nginx/sites-available/default\\\" to rootfs \\\"/var/lib/docker/btrfs/subvolumes/cf8113a0f42659e88bfda8ad2ca647d1152249b98b310e9561a8c0e83750eafb\\\" at \\\"/var/lib/docker/btrfs/subvolumes/cf8113a0f42659e88bfda8ad2ca647d1152249b98b310e9561a8c0e83750eafb/etc/nginx/conf.d/default.conf\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type'.
     Started: 10:34:05.299510
    Duration: 31699.527 ms
     Changes:
              ----------
              container_id:
                  ----------
                  added:
                      db64a3caa3805bc57e68d77abe7d8a94878e2e5d240e7043011222babdbb2a25
              image:
                  ----------
                  Layers:
                      ----------
                      Pulled:
                          - f789259667da
                          - cd784148e348
                          - 7e5cdc0df0c4
                          - 44249028334b
                          - 7c0ea5070fa4
                          - 437435d1a872
                          - 9f5a4538dd6e
                          - 2b554a36c05f
                  Status:
                      Downloaded newer image for opensdsio/dashboard:latest
                  Time_Elapsed:
                      2.373220920562744
                  retcode:
                      0

[BUG] hotpot daemon make fails on FreeBSD

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Pillar / config used

---

Bug details

Describe the bug

          ID: opensds hotpot daemon opensds software build make
    Function: cmd.run
        Name: make
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command "make" run"
              Attempt 2: Returned a result of "False", with the following comment: "Command "make" run"
              Command "make" run
     Started: 02:58:35.651361
    Duration: 128082.605 ms
     Changes:
              ----------
              pid:
                  32580
              retcode:
                  1
              stderr:
                  make: exec(go) failed (No such file or directory)
              stdout:
                  mkdir -p /build/out
                  go build -o /build/out/bin/osdsdock github.com/opensds/opensds/cmd/osdsdock
                  *** Error code 1

                  Stop.
                  make: stopped in /usr/local/golang/packages/src/github.com/opensds/opensds

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

Bugs in formula

Number of issues to be addressed - various formulae.

Also some states not running in expected order (i.e. Devstack).

Summary for local
--------------
Succeeded: 116 (changed=54)
Failed:     22
--------------
Total states run:     138
Total run time:   309.985 s

golang intermittent problem?

[ERROR   ] Alternative for link-godoc not installed: 
[ERROR   ] Alternative for link-gofmt not installed: 
[ERROR   ] Alternative for link-go not installed: 

docker formula is not detecting docker.container.skip_translate: ports ???

[ERROR   ] Failed to translate input. Additional info follows:
invalid:
    ----------
    ports:
        '{u'50050/tcp': OrderedDict([(u'HostPort', u'50050'), (u'HostIp', u'192.168.1.1')])}' is not a valid port definition
If you feel this information is incorrect, the skip_translate argument can be used to skip input translation for the argument(s) identified as invalid. See the documentation for details.
[ERROR   ] Failed to translate input. Additional info follows:
.. etc ...

file.line this always gives troubles.

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 1905, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1831, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/states/file.py", line 3933, in line
    backup=backup, quiet=quiet, indent=indent)
  File "/usr/lib/python2.7/site-packages/salt/modules/file.py", line 2017, in line
    raise CommandExecutionError("Wrong conditions? "
CommandExecutionError: Wrong conditions? Unable to ensure line without knowing where to put it before and/or after.

devstack ... Where is string "Service" coming from? Bad Pillars?

 [ERROR   ] stderr: usage: openstack service create [-h] [-f {json,shell,table,value,yaml}]
                                [-c COLUMN] [--max-width <integer>]
                                [--fit-width] [--print-empty] [--noindent]
                                [--prefix PREFIX] [--name <name>]
                                [--description <description>]
                                [--enable | --disable]
                                <type>
openstack service create: error: unrecognized arguments: --type Service opensdsv0.3.0

opensds/files/macros.jinja minor bug

          ID: opensds dock block generate lvm driver file
    Function: file.managed
        Name: /etc/opensds/driver/lvm.yaml
      Result: False
     Comment: Unable to manage file: Jinja variable 'spaces' is undefined
     Started: 22:52:38.763556
    Duration: 30.78 ms
     Changes:   

lvm why did lvm.files.create state not run?

          ID: lvm_pv_create_/dev/loop0
    Function: lvm.pv_present
        Name: /dev/loop0
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/site-packages/salt/state.py", line 1905, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1831, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/site-packages/salt/states/lvm.py", line 66, in pv_present
                  changes = __salt__['lvm.pvcreate'](name, **kwargs)
                File "/usr/lib/python2.7/site-packages/salt/modules/linux_lvm.py", line 228, in pvcreate
                  raise CommandExecutionError('{0} does not exist'.format(device))
              CommandExecutionError: /dev/loop0 does not exist
     Started: 22:52:36.938774
    Duration: 377.737 ms
     Changes:  

etcd formula Check RedHat...

 ID: etcd-docker-python-docker-py-package
    Function: pkg.installed
        Name: python-docker-py
      Result: False
     Comment: The following packages failed to install/update: python-docker-py
     Started: 22:52:04.979023
    Duration: 2930.643 ms
     Changes:   

k8s Business logic issue with opensds installer

          ID: opensds nbp plugins k8s start
    Function: cmd.run
        Name: source /etc/profile && kubectl create -f deploy/kubernetes
      Result: False
     Comment: Command "source /etc/profile && kubectl create -f deploy/kubernetes" run
     Started: 22:52:04.895546
    Duration: 26.312 ms
     Changes:   
              ----------
              pid:
                  22219
              retcode:
                  127
              stderr:
                  /bin/bash: kubectl: command not found
              stdout: