GithubHelp home page GithubHelp logo

meta-akraino's People

Contributors

gaolitao avatar jackiehjm avatar qliu1 avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

meta-akraino's Issues

Packge porting: product-manifest

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

The spec:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob;f=product-manifest.spec;h=f36527e0afc7eac85b8cf2302dea6289bc393c8f;hb=HEAD

In build 185:

[newadmin@controller-1 ~]$ rpm -qa|grep product-manifest
product-manifest-185-1.el7.centos.ta.noarch
[newadmin@controller-1 ~]$ rpm -ql product-manifest
/etc/product-release
[newadmin@controller-1 ~]$ cat /etc/product-release
release=TA-1
build=185

image: add caas-helm

The docker build for caas-helm failed, need to fix or workaround and add it into the image.

build failure:

| Cloning into 'helm'...
| make: *** No rule to make target `bootstrap'.  Stop.
| Removing intermediate container fd5247f1d2dd
| The command '/bin/sh -c yum install -y --setopt=skip_missing_names_on_install=False       ${build_packages} &&  mkdir -p $go_install_dir &&  if [ $(uname -m) == 'aarch64' ]; then HOST_ARCH=arm64; else HOST_ARCH=amd64; fi &&  curl -fsSL -k https://dl.google.com/go/go${go_version}.linux-${HOST_ARCH}.tar.gz | tar zx --strip-components=1 -C ${go_install_dir} &&  mkdir -p $GOPATH/src/k8s.io &&  cd $GOPATH/src/k8s.io &&  git clone https://github.com/kubernetes/helm.git &&  cd helm &&  make GIT_TAG=v${HELM_VERSION} bootstrap build-cross &&  mkdir -p ${binaries} &&  mv ./_dist/linux-${HOST_ARCH}/tiller ${binaries}/ &&  mv ./_dist/linux-${HOST_ARCH}/helm ${binaries}/ &&  rm -rf $GOPATH &&  rm -rf $go_install_dir &&  rm -rf /build &&  rm -rf /root/.glide' returned a non-zero code: 2
| WARNING: exit code 2 from a shell command.
| ERROR: Function failed: do_compile (log file is located at /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/caas-helm/2.14.3-3-r0/temp/log.do_compile.80879)
NOTE: recipe caas-helm-2.14.3-3-r0: task do_compile: Failed
ERROR: Task (/buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-akraino/recipes-akraino/caas-helm/caas-helm_git.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 477 tasks of which 476 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-akraino/recipes-akraino/caas-helm/caas-helm_git.bb:do_compile
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

In build 185:

[newadmin@controller-1 ~]$ rpm -qa|grep helm
caas-helm-2.14.3-3.el7.centos.ta.x86_64
[newadmin@controller-1 ~]$ rpm -ql caas-helm-2.14.3-3.el7.centos.ta.x86_64
/etc/ansible/roles/helm
/etc/ansible/roles/helm/meta
/etc/ansible/roles/helm/meta/main.yaml
/etc/ansible/roles/helm/tasks
/etc/ansible/roles/helm/tasks/config.yml
/etc/ansible/roles/helm/tasks/main.yml
/etc/ansible/roles/helm/templates
/etc/ansible/roles/helm/templates/tiller-service.yml
/etc/ansible/roles/helm/templates/tiller.yml
/opt/openstack-ansible/playbooks/helm.yaml
/usr/bin/helm
/var/lib/caas/images/tiller:2.14.3-3.tar

image: do_rootfs failed with conflicts between mailcap and nginx

do_rootfs failed with:

Error: Transaction check error:
  file /etc/nginx/mime.types conflicts between attempted installs of mailcap-2.1.48-r0.corei7_64 and nginx-1.15.2-r0.corei7_64

Error Summary
-------------


ERROR: akraino-image-rec-1.0-r0 do_rootfs: Function failed: do_rootfs
ERROR: Logfile of failure stored in: /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/intel_x86_64-oe-linux/akraino-image-rec/1.0-r0/temp/log.do_rootfs.64735
NOTE: recipe akraino-image-rec-1.0-r0: task do_rootfs: Failed
ERROR: Task (/buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-akraino/recipes-core/images/akraino-image-rec.bb:do_rootfs) failed with exit code '1'
NOTE: Tasks Summary: Attempted 15972 tasks of which 15963 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-akraino/recipes-core/images/akraino-image-rec.bb:do_rootfs
Summary: There were 2 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

Package porting: iptables-services

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

In build 185:

[newadmin@controller-1 ~]$ rpm -qa|grep iptables-services
iptables-services-1.4.21-28.el7.x86_64
[newadmin@controller-1 ~]$ rpm -ql iptables-services
/etc/sysconfig/ip6tables
/etc/sysconfig/iptables
/usr/lib/systemd/system/ip6tables.service
/usr/lib/systemd/system/iptables.service
/usr/libexec/initscripts/legacy-actions/ip6tables
/usr/libexec/initscripts/legacy-actions/ip6tables/panic
/usr/libexec/initscripts/legacy-actions/ip6tables/save
/usr/libexec/initscripts/legacy-actions/iptables
/usr/libexec/initscripts/legacy-actions/iptables/panic
/usr/libexec/initscripts/legacy-actions/iptables/save
/usr/libexec/iptables
/usr/libexec/iptables/ip6tables.init
/usr/libexec/iptables/iptables.init

Bootstrapping playbooks failed on TASK [partfs_rootdisk : delete temporary partition]

Error:

TASK [partfs_rootdisk : delete temporary partition] ****************************
Sunday 09 February 2020  09:41:22 -0500 (0:00:00.029)       0:00:08.706 *******
fatal: [controller-1]: FAILED! => {"changed": false, "err": "Warning: Partition /dev/sda2 is being used. Are you sure you want to continue?\n", "failed": true, "msg": "Error while running parted script: /usr/sbin/parted -s -m -a optimal /dev/sda -- rm 2", "out": "", "rc": 1}

ceph: downgrade the version to 12.2.11

Do we need to downgrade the version?

In wrl akraino image:
ceph-python-13.2.2-r0.corei7_64
ceph-13.2.2-r0.corei7_64

In Akraino REC build 185:
[newadmin@controller-1 ~]$ rpm -qa|grep ceph
ceph-osd-12.2.11-0.el7.x86_64
ceph-12.2.11-0.el7.x86_64
ceph-common-12.2.11-0.el7.x86_64
libcephfs2-12.2.11-0.el7.x86_64
python-cephfs-12.2.11-0.el7.x86_64
ceph-base-12.2.11-0.el7.x86_64
ceph-selinux-12.2.11-0.el7.x86_64
ceph-mon-12.2.11-0.el7.x86_64
ceph-radosgw-12.2.11-0.el7.x86_64
ceph-ansible-3.0.24-1.el7.noarch
ceph-mds-12.2.11-0.el7.x86_64
ceph-mgr-12.2.11-0.el7.x86_64

Package update to build185: caas-logging

caas-logging includes: caas-logging, caas-elasticsearch and caas-fluentd
In current wrl akraino image:
caas-elasticsearch-7.3.0+0-r0.corei7_64
caas-fluentd-1.6.3+0-r0.corei7_64

In build185:
caas-logging-c16.gdbb5262-1.el7.centos.ta.noarch
caas-elasticsearch-7.3.0-1.el7.centos.ta.x86_64
caas-fluentd-1.6.3-0.el7.centos.ta.x86_64

Package porting: uwsgi

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

The recipe exists in:
meta-cloud-services/meta-openstack/recipes-extended/uwsgi/uwsgi_git.bb

the version is 2.0.14, but it's 2.0.17 in build 185:

[newadmin@controller-1 ~]$ rpm -ql uwsgi-2.0.17.1-2.el7.x86_64
/etc/uwsgi.d
/etc/uwsgi.ini
/usr/lib/systemd/system/uwsgi.service
/usr/lib/tmpfiles.d/uwsgi.conf
/usr/sbin/uwsgi
/usr/share/doc/uwsgi-2.0.17.1
/usr/share/doc/uwsgi-2.0.17.1/CHANGELOG
/usr/share/doc/uwsgi-2.0.17.1/README
/usr/share/doc/uwsgi-2.0.17.1/README.Fedora
/usr/share/licenses/uwsgi-2.0.17.1
/usr/share/licenses/uwsgi-2.0.17.1/LICENSE
[newadmin@controller-1 ~]$ rpm -ql uwsgi-plugin-common-2.0.17.1-2.el7.x86_64
/usr/lib64/uwsgi
/usr/lib64/uwsgi/cgi_plugin.so
/usr/lib64/uwsgi/http_plugin.so
[newadmin@controller-1 ~]$ rpm -ql uwsgi-plugin-python2-2.0.17.1-2.el7.x86_64
/usr/lib64/uwsgi/python_plugin.so

And I tried to build but failed with:

NOTE: Running task 1185 of 1191 (/buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-cloud-services/meta-openstack/recipes-extended/uwsgi/uwsgi_git.bb:do_install)
NOTE: recipe uwsgi-2.0.14+gitAUTOINC+af44211739-r0: task do_install: Started
ERROR: uwsgi-2.0.14+gitAUTOINC+af44211739-r0 do_install: 'python setup.py install --root=/buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/uwsgi/2.0.14+gitAUTOINC+af44211739-r0/image     --prefix=/usr     --install-lib=/usr/lib/python2.7/site-packages     --install-data=/usr/share' execution failed.
ERROR: uwsgi-2.0.14+gitAUTOINC+af44211739-r0 do_install: Function failed: do_install (log file is located at /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/uwsgi/2.0.14+gitAUTOINC+af44211739-r0/temp/log.do_install.51278)
ERROR: Logfile of failure stored in: /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/uwsgi/2.0.14+gitAUTOINC+af44211739-r0/temp/log.do_install.51278
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/python/python_2.7.15.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/python/python-native_2.7.15.bb:do_populate_sysroot', 'virtual:native:/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/python/python-setuptools_40.0.0.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.44.3.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/meta-openembedded/meta-oe/recipes-devtools/yajl/yajl_2.1.0.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/meta-openembedded/meta-python/recipes-devtools/python/python-six_1.11.0.bb:do_populate_sysroot', 'virtual:native:/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/meta-openembedded/meta-python/recipes-devtools/python/python-pip_18.0.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/gcc/gcc-cross_8.2.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/gcc/gcc-runtime_8.2.bb:do_populate_sysroot', 'virtual:native:/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', '/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/oe-core/meta/recipes-core/glibc/glibc_2.28.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: ['pseudo-native']
| NOTE: Skipping as already exists in sysroot: ['python', 'python-native', 'python-setuptools-native', 'e2fsprogs', 'yajl', 'python-six', 'pkgconfig-native', 'quilt-native', 'python-pip', 'gcc-cross-x86_64', 'gcc-runtime', 'glibc', 'zlib', 'sqlite3', 'gdbm', 'libxcrypt', 'ncurses', 'openssl', 'readline', 'libffi', 'bzip2', 'db', 'gnu-config-native', 'bzip2-native', 'db-native', 'xz-native', 'automake-native', 'autoconf-native', 'openssl-native', 'zlib-native', 'sqlite3-native', 'libtool-native', 'expat-native', 'gdbm-native', 'readline-native', 'unzip-native', 'attr', 'opkg-utils', 'util-linux', 'flex-native', 'binutils-cross-x86_64', 'mpfr-native', 'libmpc-native', 'texinfo-dummy-native', 'linux-libc-headers', 'gmp-native', 'libgcc', 'gettext-minimal-native', 'm4-native', 'ncurses-native', 'bash-completion', 'libpam', 'libselinux', 'flex', 'cracklib', 'libpcre', 'libsepol']
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_install
| running install
| core/utils.c: In function 'uwsgi_substitute':
| core/utils.c:1727:3: error: 'strncat' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
|    strncat(dst, with, with_len);
|    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
| core/utils.c:1707:20: note: length computed here
|   size_t with_len = strlen(with);
|                     ^~~~~~~~~~~~
| core/utils.c:1732:2: error: 'strncat' output truncated before terminating nul copying as many bytes from a string as its length [-Werror=stringop-truncation]
|   strncat(dst, ptr, strlen(ptr));
|   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| cc1: all warnings being treated as errors
| ERROR: 'python setup.py install --root=/buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/uwsgi/2.0.14+gitAUTOINC+af44211739-r0/image     --prefix=/usr     --install-lib=/usr/lib/python2.7/site-packages     --install-data=/usr/share' execution failed.
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_install (log file is located at /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/uwsgi/2.0.14+gitAUTOINC+af44211739-r0/temp/log.do_install.51278)
NOTE: recipe uwsgi-2.0.14+gitAUTOINC+af44211739-r0: task do_install: Failed
ERROR: Task (/buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-cloud-services/meta-openstack/recipes-extended/uwsgi/uwsgi_git.bb:do_install) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1185 tasks of which 1177 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-cloud-services/meta-openstack/recipes-extended/uwsgi/uwsgi_git.bb:do_install
Summary: There were 2 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

Failed on TASK [bootstrap-host : Deploy user_secrets file]

TASK [bootstrap-host : Deploy user_secrets file] *******************************
fatal: [controller-1]: FAILED! => {"changed": false, "msg": "Unsupported parameters for (copy) module: original_basename Supported parameters include: _original_basename, attributes, backup, checksum, content, delimiter, dest, directory_mode, follow, force, group, local_follow, mode, owner, regexp, remote_src, selevel, serole, setype, seuser, src, unsafe_writes, validate"}

PLAY RECAP *********************************************************************
controller-1               : ok=23   changed=11   unreachable=0    failed=1    skipped=13   rescued=0    ignored=0

[2020-02-04 00:56:16,845   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook setup-playbook.yml failed

image: add caas-hyperdanm

The docker build for caas-hyperdanm failed, need to fix or workaround and add it into the image.

The Error:

| Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=extras&infra=container error was
| 14: HTTP Error 502 - Bad Gateway
| Removing intermediate container 8abf0ede9b12
| The command '/bin/sh -c yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm &&  yum install -y --setopt=skip_missing_names_on_install=False       ${build_packages}       ${danm_dependencies}       bash       golang &&  set -x &&  mkdir -p ${go_install_dir} &&  curl -fsSL -k https://dl.google.com/go/go${go_version}.linux-amd64.tar.gz | tar zx --strip-components=1 -C ${go_install_dir} &&  glide_version="$(curl -sSLf https://glide.sh/version)" &&  mkdir -p ${glide_install_dir} &&  curl -fsSL https://github.com/Masterminds/glide/releases/download/${glide_version}/glide-${glide_version}-linux-amd64.tar.gz | tar zx --strip-components=1 -C ${glide_install_dir} &&  mkdir -p $GOPATH/src/github.com/nokia &&  git clone https://github.com/nokia/danm.git $GOPATH/src/github.com/nokia/danm &&  cd $GOPATH/src/github.com/nokia/danm &&  git fetch --tags &&  git checkout ${DANM_VERSION} &&  glide install --strip-vendor &&  go get -d github.com/vishvananda/netlink &&  go get github.com/containernetworking/plugins/pkg/ns &&  go get github.com/golang/groupcache/lru &&  rm -rf $GOPATH/src/k8s.io/code-generator &&  git clone -b ${code_generator_version} --depth 1 https://github.com/kubernetes/code-generator.git $GOPATH/src/k8s.io/code-generator &&  go install k8s.io/code-generator/cmd/deepcopy-gen &&  go install k8s.io/code-generator/cmd/client-gen &&  go install k8s.io/code-generator/cmd/lister-gen &&  go install k8s.io/code-generator/cmd/informer-gen &&  deepcopy-gen --alsologtostderr --input-dirs github.com/nokia/danm/crd/apis/danm/v1 -O zz_generated.deepcopy --bounding-dirs github.com/nokia/danm/crd/apis &&  client-gen --alsologtostderr --clientset-name versioned --input-base "" --input github.com/nokia/danm/crd/apis/danm/v1 --clientset-path github.com/nokia/danm/crd/client/clientset &&  lister-gen --alsologtostderr --input-dirs github.com/nokia/danm/crd/apis/danm/v1 --output-package github.com/nokia/danm/crd/client/listers &&  informer-gen --alsologtostderr --input-dirs github.com/nokia/danm/crd/apis/danm/v1 --versioned-clientset-package github.com/nokia/danm/crd/client/clientset/versioned --listers-package github.com/nokia/danm/crd/client/listers --output-package github.com/nokia/danm/crd/client/informers &&  mkdir -p ${binaries} &&  for component in $components; do       go install -a -ldflags '-extldflags "-static"' github.com/nokia/danm/cmd/${component};       cp $GOPATH/bin/${component} ${binaries}/${component};     done' returned a non-zero code: 1
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile (log file is located at /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/caas-hyperdanm/4.0.0-0-r0/temp/log.do_compile.31537)
NOTE: recipe caas-hyperdanm-4.0.0-0-r0: task do_compile: Failed
ERROR: Task (/buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-akraino/recipes-akraino/caas-danm/caas-hyperdanm_git.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 477 tasks of which 471 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /buildarea2/jhuang0/wrl_akraino_191031/src_extra_layers/meta-akraino/recipes-akraino/caas-danm/caas-hyperdanm_git.bb:do_compile
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Package update to build185: mariadb and mariadb-server-galera

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

In build185:
[newadmin@controller-1 ~]$ rpm -ql mariadb-server-galera-10.1.20-2.el7.x86_64
/etc/my.cnf.d/galera.cnf
/etc/sysconfig/clustercheck
/usr/bin/clustercheck
/usr/bin/galera_new_cluster
/usr/bin/galera_recovery
/usr/share/doc/mariadb-server-galera-10.1.20
/usr/share/doc/mariadb-server-galera-10.1.20/README.wsrep
/usr/share/licenses/mariadb-server-galera-10.1.20
/usr/share/licenses/mariadb-server-galera-10.1.20/LICENSE.clustercheck
/usr/share/mariadb/systemd/use_galera_new_cluster.conf
/usr/share/selinux/packages/mariadb/mariadb-server-galera.pp

Failed on TASK [Setup hostname for Installation controller.]

TASK [Setup hostname for Installation controller.] *****************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "hostname module cannot be used on platform Linux ()"}
        to retry, use: --limit @/opt/openstack-ansible/playbooks/presetup-playbook.retry

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=1

[2020-01-13 08:53:41,975   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook presetup-playbook.yml failed

Failed to start installation phase bootstrapping

[2020-02-04 13:30:14,914   DEBUG cmbackendhandler( 46)] set_property called for setting cloud.installation_phase="setup-ended"
[2020-02-04 13:30:14,915   DEBUG cmbackendhandler( 46)] set_property called for setting cloud.installation_phase="bootstrapping-started"
^[[0;35mVariable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml  --skip-tags pip,install-apt,install-yum,memcached_server-install,haproxy_package-install,galera-server-pip-packages,galera-gpg-keys,galera-repos,galera-pre-yum-packages,galera-client-install,galera_server-install,rabbitmq-yum-packages,rabbitmq-plugin-config,common-log,bird,rsyslog-yum-packages,glance-install,cinder-install,nova-install,nova-pip-packages,nova-novnc-git,calico-pip-packages,dragonflow-pip-packages,neutron-pkg-install,heat-pkg-install,horizon-pkg-install,swift-pkg-install,swift-chk-hashes,ironic-l2,ironic-pkg-install,ceph-install,package-install -e @/etc/openstack_deploy/env.d/baremetal.yml -e @/etc/openstack_deploy/user_variables.yml -e @/etc/openstack_deploy/user_secrets.yml"^[[0m
 [WARNING]:  * Failed to parse /opt/openstack-ansible/inventory/inventory.sh
with script plugin: Inventory script (/opt/openstack-
ansible/inventory/inventory.sh) had an execution error: Traceback (most recent
call last):   File "/usr/local/bin/cmcli", line 6, in <module>     from
pkg_resources import load_entry_point   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 3098, in <module>     @_call_aside
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3082,
in _call_aside     f(*args, **kwargs)   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 3111, in
_initialize_master_working_set     working_set = WorkingSet._build_master()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 564, in
_build_master     ws = cls()   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 557, in __init__
self.add_entry(entry)   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 613, in add_entry     for dist in
find_distributions(entry, True):   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 1961, in find_on_path     for dist in
factory(fullpath):   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 2031, in distributions_from_metadata
root, entry, metadata, precedence=DEVELOP_DIST,   File "/usr/lib/python2.7
/site-packages/pkg_resources/__init__.py", line 2470, in from_location
py_version=py_version, platform=platform, **kw   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 2845, in _reload_version
md_version = _version_from_file(self._get_metadata(self.PKG_INFO))   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2434, in
_version_from_file     line = next(iter(version_lines), '')   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2624, in
_get_metadata     for line in self.get_metadata_lines(name):   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1407, in
get_metadata_lines     return yield_lines(self.get_metadata(name))   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1403, in
get_metadata     value = self._get(self._fn(self.egg_info, name))   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1515, in
_get     with open(path, 'rb') as stream: IOError: [Errno 13] Permission
denied: '/usr/lib/python2.7/site-packages/Flask_Bootstrap-3.3.7.1-py2.7.egg-
info/PKG-INFO'
 [WARNING]:  * Failed to parse /opt/openstack-ansible/inventory/inventory.sh
with ini plugin: /opt/openstack-ansible/inventory/inventory.sh:17: Expected
key=value host variable assignment, got: [
 [WARNING]: Unable to parse /opt/openstack-ansible/inventory/inventory.sh as an
inventory source
[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use
'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions.
 This feature will be removed in a future release. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is
 discouraged. The module documentation details page may explain more about this
 rationale.. This feature will be removed in a future release. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: The use of 'static' has been deprecated. Use
'import_tasks' for static inclusion, or 'include_tasks' for dynamic inclusion.
This feature will be removed in a future release. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Could not match supplied host pattern, ignoring: management

PLAY [Report installation progress] ********************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: baremetal-
infra_hosts

PLAY [Power-off all the hosts provided in user_config.yml before starting deployemt.] ***
skipping: no hosts matched

PLAY [Report installation progress] ********************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: shared-infra_hosts

PLAY [Configure NTP service on controllers] ************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: base
PLAY [Report installation progress] ********************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring: rsyslog

PLAY [Install rsyslog] *********************************************************
skipping: no hosts matched

PLAY [Report installation progress] ********************************************
skipping: no hosts matched
 [WARNING]: Could not match supplied host pattern, ignoring:
baremetal_management_nodes

PLAY [Installation and setup of Ironic] ****************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************

Tuesday 04 February 2020  13:30:16 +0000 (0:00:00.063)       0:00:00.063 ******
===============================================================================
[2020-02-04 13:30:16,156   DEBUG cmbackendhandler( 46)] set_property called for setting cloud.installation_phase="bootstrapping-ended"
[2020-02-04 13:30:16,157   DEBUG cmbackendhandler( 46)] set_property called for setting cloud.installation_phase="provisioning-started"
^[[0;35mVariable files: "-e @/etc/openstack_deploy/user_secrets.yml -e @/etc/openstack_deploy/user_variables.yml  --skip-tags pip,install-apt,install-yum,memcached_server-install,haproxy_package-install,galera-server-pip-packages,galera-gpg-keys,galera-repos,galera-pre-yum-packages,galera-client-install,galera_server-install,rabbitmq-yum-packages,rabbitmq-plugin-config,common-log,bird,rsyslog-yum-packages,glance-install,cinder-install,nova-install,nova-pip-packages,nova-novnc-git,calico-pip-packages,dragonflow-pip-packages,neutron-pkg-install,heat-pkg-install,horizon-pkg-install,swift-pkg-install,swift-chk-hashes,ironic-l2,ironic-pkg-install,ceph-install,package-install -e @/etc/openstack_deploy/env.d/baremetal.yml -e @/etc/openstack_deploy/user_variables.yml -e @/etc/openstack_deploy/user_secrets.yml"^[[0m
 [WARNING]:  * Failed to parse /opt/openstack-ansible/inventory/inventory.sh
with script plugin: Inventory script (/opt/openstack-
ansible/inventory/inventory.sh) had an execution error: Traceback (most recent
call last):   File "/usr/local/bin/cmcli", line 6, in <module>     from
pkg_resources import load_entry_point   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 3098, in <module>     @_call_aside
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3082,
in _call_aside     f(*args, **kwargs)   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 3111, in
_initialize_master_working_set     working_set = WorkingSet._build_master()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 564, in
_build_master     ws = cls()   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 557, in __init__
self.add_entry(entry)   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 613, in add_entry     for dist in
find_distributions(entry, True):   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 1961, in find_on_path     for dist in
factory(fullpath):   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 2031, in distributions_from_metadata
root, entry, metadata, precedence=DEVELOP_DIST,   File "/usr/lib/python2.7
/site-packages/pkg_resources/__init__.py", line 2470, in from_location
py_version=py_version, platform=platform, **kw   File "/usr/lib/python2.7/site-
packages/pkg_resources/__init__.py", line 2845, in _reload_version
md_version = _version_from_file(self._get_metadata(self.PKG_INFO))   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2434, in
_version_from_file     line = next(iter(version_lines), '')   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2624, in
_get_metadata     for line in self.get_metadata_lines(name):   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1407, in
get_metadata_lines     return yield_lines(self.get_metadata(name))   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1403, in
get_metadata     value = self._get(self._fn(self.egg_info, name))   File
"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1515, in
_get     with open(path, 'rb') as stream: IOError: [Errno 13] Permission
denied: '/usr/lib/python2.7/site-packages/Flask_Bootstrap-3.3.7.1-py2.7.egg-
info/PKG-INFO'
 [WARNING]:  * Failed to parse /opt/openstack-ansible/inventory/inventory.sh
with ini plugin: /opt/openstack-ansible/inventory/inventory.sh:17: Expected
key=value host variable assignment, got: [
 [WARNING]: Unable to parse /opt/openstack-ansible/inventory/inventory.sh as an
inventory source
[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use
'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions.
 This feature will be removed in a future release. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is
 discouraged. The module documentation details page may explain more about this
 rationale.. This feature will be removed in a future release. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: 'include' for playbook includes. You should use
'import_playbook' instead. This feature will be removed in version 2.8.
Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
ERROR! the role 'ceph-defaults' was not found in /opt/openstack-ansible/playbooks/roles:/etc/ansible/roles:/opt/openstack-ansible/playbooks/roles:/opt/openstack-ansible/playbooks/roles:/opt/openstack-ansible/playbooks
The error appears to have been in '/opt/openstack-ansible/playbooks/site.yml': line 74, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  roles:
    - ceph-defaults
      ^ here
[2020-02-04 13:30:17,282   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook provisioning-playbook.yml failed

Add missing packages: xfsprogs and e2fsprogs-resize2fs

The command 'xfs_growfs' in pkg xfsprogs and 'resize2fs' in pkg e2fsprogs-resize2fs are reruiqed in cloud-init and ansible role partfs_rootdisk.

root@intel-x86-64~# cloud-init status --long
status: error
time: Sun, 16 Feb 2020 00:32:14 +0000
detail:
('resizefs', ProcessExecutionError(u"Unexpected error while running command.\nCommand: ('resize2fs', '/dev/disk/by-partuuid/366b634d-02')\nExit code: -\nReason: [Errno 2] No such file or directory\nStdout: -\nStderr: -",))
('locale', AttributeError("'SysConf' object has no attribute '_a_to_u'",))

os-set-config doesn't work

os-set-config is used in setting the externel network in start-menu and the boostrapping, but it doesn't work, the root cause is:

os-net-config uses ifcfg files (in /etc/sysconfig/network-scripts) or ifupdown
config files (in /etc/network/interfaces) to do the network settings, but these
do not work in wrlinux right now, systemd-networkd is used by default
and it uses the config files in /etc/systemd/network which is not supported
by os-net-config.

The error in boostrap:

TASK [bootstrap-host : Run os-net-config command] ******************************
fatal: [controller-1]: FAILED! => {"changed": true, "cmd": ["os-net-config", "--detailed-exit-codes", "-v", "-c", "/etc/os-net-config/config.json"], "delta": "0:00:00.682815", "end": "2020-01-17 19:47:32.002752", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2020-01-17 19:47:31.319937", "stderr": "[2020/01/17 07:47:31 PM] [INFO] Using config file at: /etc/os-net-config/config.json\n[2020/01/17 07:47:31 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml\n[2020/01/17 07:47:31 PM] [INFO] ENI net config provider created.\n[2020/01/17 07:47:31 PM] [WARNING] Config file failed schema validation at network_config/2:\n    {'persist_mapping': False, 'routes': [{'ip_netmask': '', 'next_hop': '172.20.1.26'}], 'addresses': [{'ip_netmask': '172.20.1.26/29'}], 'nic_mapping': None, 'device': 'bond0', 'use_dhcp': False, 'type': 'vlan', 'vlan_id': 144, 'mtu': 9000} is not valid under any of the given schemas\n  Sub-schemas tested and not matching:\n  - items/oneOf/vlan/routes/items/ip_netmask/oneOf: '' is not valid under any of the given schemas\n  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf: '' is not valid under any of the given schemas\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv4_cidr_string/pattern: '' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/(3[0-2]|[1-2][0-9]|[0-9])$'\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv6_cidr_string/pattern: '' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9])$'\n  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf: '' is not valid under any of the given schemas\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/0/type: '' is not of type 'object'\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/1/type: '' is not of type 'object'\n  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/2/type: '' is not of type 'object'\n[2020/01/17 07:47:31 PM] [INFO] nic1 mapped to: eth4\nTraceback (most recent call last):\n  File \"/usr/bin/os-net-config\", line 10, in <module>\n    sys.exit(main())\n  File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 204, in main\n    provider.add_object(obj)\n  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 86, in add_object\n    self.add_linux_bond(obj)\n  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 167, in add_linux_bond\n    raise NotImplementedError(\"add_linux_bond is not implemented.\")\nNotImplementedError: add_linux_bond is not implemented.", "stderr_lines": ["[2020/01/17 07:47:31 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/01/17 07:47:31 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml", "[2020/01/17 07:47:31 PM] [INFO] ENI net config provider created.", "[2020/01/17 07:47:31 PM] [WARNING] Config file failed schema validation at network_config/2:", "    {'persist_mapping': False, 'routes': [{'ip_netmask': '', 'next_hop': '172.20.1.26'}], 'addresses': [{'ip_netmask': '172.20.1.26/29'}], 'nic_mapping': None, 'device': 'bond0', 'use_dhcp': False, 'type': 'vlan', 'vlan_id': 144, 'mtu': 9000} is not valid under any of the given schemas", "  Sub-schemas tested and not matching:", "  - items/oneOf/vlan/routes/items/ip_netmask/oneOf: '' is not valid under any of the given schemas", "  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf: '' is not valid under any of the given schemas", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv4_cidr_string/pattern: '' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/(3[0-2]|[1-2][0-9]|[0-9])$'", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv6_cidr_string/pattern: '' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9])$'", "  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf: '' is not valid under any of the given schemas", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/0/type: '' is not of type 'object'", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/1/type: '' is not of type 'object'", "  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/2/type: '' is not of type 'object'", "[2020/01/17 07:47:31 PM] [INFO] nic1 mapped to: eth4", "Traceback (most recent call last):", "  File \"/usr/bin/os-net-config\", line 10, in <module>", "    sys.exit(main())", "  File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 204, in main", "    provider.add_object(obj)", "  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 86, in add_object", "    self.add_linux_bond(obj)", "  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 167, in add_linux_bond", "    raise NotImplementedError(\"add_linux_bond is not implemented.\")", "NotImplementedError: add_linux_bond is not implemented."], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************
controller-1               : ok=19   changed=10   unreachable=0    failed=1    skipped=13   rescued=0    ignored=0

[2020-01-17 19:47:32,079   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook setup-playbook.yml failed

Failed to start bootstrapping playbooks: Permission denied

"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1515, in
_get     with open(path, 'rb') as stream: IOError: [Errno 13]
Permission denied: '/usr/lib/python2.7/site-packages/Flask_Bootstrap-3.3.7.1-py2.7.egg-info/PKG-INFO'
Permission denied: '/usr/lib/python2.7/site-packages/docker_registry_core-2.0.3-py2.7.egg-info/PKG-INFO'

image: add ntp

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

In build185:

[newadmin@controller-1 ~]$ rpm -ql ntp-4.2.6p5-28.el7.centos.x86_64
/etc/dhcp/dhclient.d
/etc/dhcp/dhclient.d/ntp.sh
/etc/ntp.conf
/etc/ntp/crypto
/etc/ntp/crypto/pw
/etc/sysconfig/ntpd
/usr/bin/ntpstat
/usr/lib/systemd/ntp-units.d/60-ntpd.list
/usr/lib/systemd/system/ntpd.service
/usr/sbin/ntp-keygen
/usr/sbin/ntpd
/usr/sbin/ntpdc
/usr/sbin/ntpq
/usr/sbin/ntptime
/usr/sbin/tickadj
/usr/share/doc/ntp-4.2.6p5
/usr/share/doc/ntp-4.2.6p5/COPYRIGHT
/usr/share/doc/ntp-4.2.6p5/ChangeLog
/usr/share/doc/ntp-4.2.6p5/NEWS
/usr/share/man/man5/ntp.conf.5.gz
/usr/share/man/man5/ntp_acc.5.gz
/usr/share/man/man5/ntp_auth.5.gz
/usr/share/man/man5/ntp_clock.5.gz
/usr/share/man/man5/ntp_decode.5.gz
/usr/share/man/man5/ntp_misc.5.gz
/usr/share/man/man5/ntp_mon.5.gz
/usr/share/man/man8/ntp-keygen.8.gz
/usr/share/man/man8/ntpd.8.gz
/usr/share/man/man8/ntpdc.8.gz
/usr/share/man/man8/ntpq.8.gz
/usr/share/man/man8/ntpstat.8.gz
/usr/share/man/man8/ntptime.8.gz
/usr/share/man/man8/tickadj.8.gz
/var/lib/ntp
/var/lib/ntp/drift
/var/log/ntpstats

Package update to build185: caas-install

caas-install includes: caas-infra-charts, caas-instantiate and caas-utils
In current wrl akraino image:
caas-infra-charts-1.0.0+20-r0.corei7_64.rpm
caas-instantiate-1.0.0+13-r0.corei7_64.rpm
caas-utils-1.0.0+5-r0.corei7_64.rpm

In build 185:
caas-infra-charts-1.0.0-34.el7.centos.ta.noarch
caas-instantiate-1.0.0-20.el7.centos.ta.noarch
caas-utils-1.0.0-10.el7.centos.ta.noarch

Bootstrapping playbooks failed on TASK [partfs_rootdisk : Stop all cloud-init services and disable them]

Error:

PLAY [Configure Root disk partitions] ******************************************

TASK [partfs_rootdisk : Gather variables for each operating system] ************
Sunday 09 February 2020  02:34:43 -0500 (0:00:00.291)       0:00:08.069 *******
ok: [controller-1] => (item=/etc/ansible/roles/partfs_rootdisk/vars/redhat-7.yml)

TASK [partfs_rootdisk : Stop all cloud-init services and disable them] *********
Sunday 09 February 2020  02:34:43 -0500 (0:00:00.041)       0:00:08.111 *******
failed: [controller-1] (item=cloud-config) => {"changed": false, "failed": true, "item": "cloud-config", "msg": "Could not find the requested service cloud-config: host"}
failed: [controller-1] (item=cloud-final) => {"changed": false, "failed": true, "item": "cloud-final", "msg": "Could not find the requested service cloud-final: host"}
failed: [controller-1] (item=cloud-init) => {"changed": false, "failed": true, "item": "cloud-init", "msg": "Could not find the requested service cloud-init: host"}
failed: [controller-1] (item=cloud-init-local) => {"changed": false, "failed": true, "item": "cloud-init-local", "msg": "Could not find the requested service cloud-init-local: host"}

PLAY RECAP *********************************************************************
controller-1               : ok=18   changed=9    unreachable=0    failed=1

Sunday 09 February 2020  02:34:44 -0500 (0:00:00.528)       0:00:08.639 *******
===============================================================================
Report installation progress -------------------------------------------- 2.81s
Gathering Facts --------------------------------------------------------- 1.46s
partfs_rootdisk : Stop all cloud-init services and disable them --------- 0.53s
ntp : Copy the ntp.conf template file ----------------------------------- 0.53s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.34s
ntp : add ntp keys ------------------------------------------------------ 0.34s
Set Timezone ------------------------------------------------------------ 0.31s
Report installation progress -------------------------------------------- 0.29s
Report installation progress -------------------------------------------- 0.28s
ntp : enable ntpdate at boot time --------------------------------------- 0.27s
ntp : Start/stop ntp service -------------------------------------------- 0.25s
ntp : Clear step-tickers ------------------------------------------------ 0.23s
ntp : create redundant fallback ntp servers ----------------------------- 0.22s
ntp : restart ntp ------------------------------------------------------- 0.20s
Set Timezone ------------------------------------------------------------ 0.17s
ntp : command ----------------------------------------------------------- 0.15s
initial_poweroff_hosts : set_fact --------------------------------------- 0.05s
partfs_rootdisk : Gather variables for each operating system ------------ 0.04s
ntp : Add the OS specific variables ------------------------------------- 0.03s
ntp : command ----------------------------------------------------------- 0.01s
[2020-02-09 02:34:44,200   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

Provisioning playbooks failed on TASK [baremetal_provision : Verify remote node ssh ports active. Waiting for 60mins max.]

Steps to reproduce:

  1. Install Akraino centos build 185 on controller-1
  2. Set external network
  3. Copy the user_conf.yaml and akraino-image-rec-intel-x86-64-20200212092844.rootfs.wic.qcow2
    to controller-1, and replace the /opt/images/guest-images.img
  4. Start deployment from the start-menu.

Error:

# in /srv/deployment/log/cm.log
TASK [baremetal_provision : Checking for any deployment failures] **************
Wednesday 12 February 2020  09:44:22 -0500 (0:09:47.171)       0:10:40.261 ****

TASK [baremetal_provision : Verify node provisioning state. Waiting for 60mins max.] ***
Wednesday 12 February 2020  09:44:22 -0500 (0:00:00.028)       0:10:40.290 ****
skipping: [controller-2]
skipping: [controller-3]

TASK [baremetal_provision : Checking for any deployment failures] **************
Wednesday 12 February 2020  09:44:22 -0500 (0:00:00.036)       0:10:40.326 ****

TASK [baremetal_provision : set_fact] ******************************************
Wednesday 12 February 2020  09:44:22 -0500 (0:00:00.023)       0:10:40.350 ****
ok: [controller-2]
ok: [controller-3]

TASK [baremetal_provision : set_fact] ******************************************
Wednesday 12 February 2020  09:44:22 -0500 (0:00:00.055)       0:10:40.406 ****

TASK [baremetal_provision : Verify remote node ssh ports active. Waiting for 60mins max.] ***
Wednesday 12 February 2020  09:44:22 -0500 (0:00:00.025)       0:10:40.431 ****
FAILED - RETRYING: Verify remote node ssh ports active. Waiting for 60mins max. (3 retries left).
FAILED - RETRYING: Verify remote node ssh ports active. Waiting for 60mins max. (3 retries left).
FAILED - RETRYING: Verify remote node ssh ports active. Waiting for 60mins max. (2 retries left).
FAILED - RETRYING: Verify remote node ssh ports active. Waiting for 60mins max. (2 retries left).
FAILED - RETRYING: Verify remote node ssh ports active. Waiting for 60mins max. (1 retries left).
FAILED - RETRYING: Verify remote node ssh ports active. Waiting for 60mins max. (1 retries left).
fatal: [controller-3 -> localhost]: FAILED! => {"attempts": 3, "changed": false, "elapsed": 3604, "failed": true, "msg": "Timeout when waiting for search string OpenSSH in 172.20.1.28:22"}
fatal: [controller-2 -> localhost]: FAILED! => {"attempts": 3, "changed": false, "elapsed": 3604, "failed": true, "msg": "Timeout when waiting for search string OpenSSH in 172.20.1.27:22"}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
controller-1               : ok=28   changed=23   unreachable=0    failed=0
controller-2               : ok=29   changed=14   unreachable=0    failed=1
controller-3               : ok=25   changed=12   unreachable=0    failed=1

Wednesday 12 February 2020  13:44:57 -0500 (4:00:34.885)       4:11:15.316 ****
===============================================================================
baremetal_provision : Verify remote node ssh ports active. Waiting for 60mins max.  14434.89s
baremetal_provision : Verify node provisioning state. Waiting for 60mins max. - 587.17s
baremetal_provision : Calculate md5sum of Golden image ----------------- 17.74s
baremetal_provision : Sleep for 15 seconds ----------------------------- 15.04s
baremetal_provision : Convert QCOW2 image to RAW ------------------------ 2.69s
baremetal_provision : Create baremetal ironic nodes --------------------- 1.64s
baremetal_provision : Start Baremetal deployment ------------------------ 1.40s
baremetal_provision : Disable shellinabox console logging for target nodes --- 1.37s
baremetal_provision : Ensure nodes are powered off ---------------------- 1.31s
baremetal_provision : Configure Baremetal deployment -------------------- 1.27s
baremetal_provision : Power off nodes ----------------------------------- 1.07s
baremetal_provision : Setup tmux ---------------------------------------- 0.77s
baremetal_provision : Block traffic on infra_internal ------------------- 0.54s
Remove all entries in /etc/exports -------------------------------------- 0.45s
Create access entries for BMC ports of all nodes in /etc/exports from /remote_image_share_root/ --- 0.44s
Create nginx conf file -------------------------------------------------- 0.41s
baremetal_provision : Generate config-drive files ----------------------- 0.35s
Report installation progress -------------------------------------------- 0.35s
baremetal_provision : set_fact ------------------------------------------ 0.33s
Create nginx service for provisioning ----------------------------------- 0.33s
[2020-02-12 13:44:57,510   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook provisioning-playbook.yml failed

Package porting: m2crypto

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

python-m2crypto is already in meta-python and meta-virtualization, but failed to build:

ERROR: python-m2crypto-0.22.3-r0 do_configure: Function failed: do_configure (log file is located at /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/python-m2crypto/0.22.3-r0/temp/log.do_configure.25319)
ERROR: Logfile of failure stored in: /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/python-m2crypto/0.22.3-r0/temp/log.do_configure.25319
Log data follows:
| DEBUG: Executing shell function do_configure
| running clean
| running build_ext
| building 'M2Crypto.__m2crypto' extension
| swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap.c
| swig -python -I/buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/python-m2crypto/0.22.3-r0/recipe-sysroot/usr/include -I/buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/python-m2crypto/0.22.3-r0/recipe-sysroot/usr/include/python2.7 -I/usr/include -I/usr/include/openssl -includeall -modern -builtin -outdir build/lib.linux-x86_64-2.7/M2Crypto -o SWIG/_m2crypto_wrap.c SWIG/_m2crypto.i
| /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/python-m2crypto/0.22.3-r0/recipe-sysroot/usr/include/sys/cdefs.h:157: Warning 305: Bad constant value (ignored).
| Unable to open file build/lib.linux-x86_64-2.7/M2Crypto/_m2crypto.py: No such file or directory
| error: command 'swig' failed with exit status 1
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_configure (log file is located at /buildarea2/jhuang0/wrl_akraino_191031/prj_wrl1018_akraino/tmp-glibc/work/corei7-64-oe-linux/python-m2crypto/0.22.3-r0/temp/log.do_configure.25319)
NOTE: recipe python-m2crypto-0.22.3-r0: task do_configure: Failed
ERROR: Task (/buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/meta-virtualization/recipes-devtools/python/python-m2crypto_0.22.3.bb:do_configure) failed with exit code '1'
NOTE: Tasks Summary: Attempted 892 tasks of which 886 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /buildarea2/jhuang0/wrl_akraino_191031/src_wrl1018/layers/meta-virtualization/recipes-devtools/python/python-m2crypto_0.22.3.bb:do_configure
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

image: add nginx

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

In build185:

[newadmin@controller-1 ~]$ rpm -qa|grep nginx
nginx-mod-http-image-filter-1.12.2-3.el7.x86_64
nginx-1.12.2-3.el7.x86_64
nginx-mod-stream-1.12.2-3.el7.x86_64
nginx-mod-http-geoip-1.12.2-3.el7.x86_64
nginx-all-modules-1.12.2-3.el7.noarch
nginx-mod-mail-1.12.2-3.el7.x86_64
nginx-mod-http-xslt-filter-1.12.2-3.el7.x86_64
nginx-mod-http-perl-1.12.2-3.el7.x86_64
nginx-filesystem-1.12.2-3.el7.noarch
[newadmin@controller-1 ~]$ rpm -ql nginx-1.12.2-3.el7.x86_64
/etc/logrotate.d/nginx
/etc/nginx/fastcgi.conf
/etc/nginx/fastcgi.conf.default
/etc/nginx/fastcgi_params
/etc/nginx/fastcgi_params.default
/etc/nginx/koi-utf
/etc/nginx/koi-win
/etc/nginx/mime.types
/etc/nginx/mime.types.default
/etc/nginx/nginx.conf
/etc/nginx/nginx.conf.default
/etc/nginx/scgi_params
/etc/nginx/scgi_params.default
/etc/nginx/uwsgi_params
/etc/nginx/uwsgi_params.default
/etc/nginx/win-utf
/usr/bin/nginx-upgrade
/usr/lib/systemd/system/nginx.service
/usr/lib64/nginx/modules
/usr/sbin/nginx
/usr/share/doc/nginx-1.12.2
/usr/share/doc/nginx-1.12.2/CHANGES
/usr/share/doc/nginx-1.12.2/README
/usr/share/doc/nginx-1.12.2/README.dynamic
/usr/share/doc/nginx-1.12.2/UPGRADE-NOTES-1.6-to-1.10
/usr/share/licenses/nginx-1.12.2
/usr/share/licenses/nginx-1.12.2/LICENSE
/usr/share/man/man3/nginx.3pm.gz
/usr/share/man/man8/nginx-upgrade.8.gz
/usr/share/man/man8/nginx.8.gz
/usr/share/nginx/html/404.html
/usr/share/nginx/html/50x.html
/usr/share/nginx/html/index.html
/usr/share/nginx/html/nginx-logo.png
/usr/share/nginx/html/poweredby.png
/usr/share/vim/vimfiles/ftdetect/nginx.vim
/usr/share/vim/vimfiles/ftplugin/nginx.vim
/usr/share/vim/vimfiles/indent/nginx.vim
/usr/share/vim/vimfiles/syntax/nginx.vim
/var/lib/nginx
/var/lib/nginx/tmp
/var/log/nginx

Playbook bootstrapping-playbook.yml failed on TASK [ntp : Copy the ntp.conf template file]

Error:

TASK [ntp : Copy the ntp.conf template file] ***********************************
Saturday 08 February 2020  07:33:18 +0000 (0:00:00.013)       0:00:03.613 *****
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 2] No such file or directory
fatal: [controller-1]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
controller-1               : ok=6    changed=4    unreachable=0    failed=1

Saturday 08 February 2020  07:33:44 +0000 (0:00:26.303)       0:00:29.917 *****
===============================================================================
ntp : Copy the ntp.conf template file ---------------------------------- 26.30s
Report installation progress -------------------------------------------- 2.70s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.32s
Report installation progress -------------------------------------------- 0.29s
ntp : command ----------------------------------------------------------- 0.15s
initial_poweroff_hosts : set_fact --------------------------------------- 0.05s
ntp : Add the OS specific variables ------------------------------------- 0.03s
ntp : Install the required packages in Debian derivatives --------------- 0.01s
[2020-02-08 07:33:44,587   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

caas-infra-charts: the roles post_install_caas_infra is missing

Error:

ERROR! the role 'post_install_caas_infra' was not found in /opt/openstack-ansible/playbooks/roles:/etc/ansible/roles:/opt/openstack-ansible/playbooks/roles:/opt/openstack-ansible/playbooks/roles:/opt/openstack-ansible/playbooks

The error appears to have been in '/opt/openstack-ansible/playbooks/install_caas_infra.yaml': line 36, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  roles:
    - post_install_caas_infra
      ^ here
[2020-02-06 11:53:54,980   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook postconfig-playbook.yml failed

In build 185:

[newadmin@controller-1 ~]$ rpm -ql caas-infra-charts|grep roles
/etc/ansible/roles/infra_chart_reconfig
/etc/ansible/roles/infra_chart_reconfig/defaults
/etc/ansible/roles/infra_chart_reconfig/defaults/main.yaml
/etc/ansible/roles/infra_chart_reconfig/tasks
/etc/ansible/roles/infra_chart_reconfig/tasks/main.yaml
/etc/ansible/roles/install_caas_infra
/etc/ansible/roles/install_caas_infra/defaults
/etc/ansible/roles/install_caas_infra/defaults/main.yaml
/etc/ansible/roles/install_caas_infra/meta
/etc/ansible/roles/install_caas_infra/meta/main.yaml
/etc/ansible/roles/install_caas_infra/tasks
/etc/ansible/roles/install_caas_infra/tasks/main.yaml
/etc/ansible/roles/post_install_caas_infra
/etc/ansible/roles/post_install_caas_infra/defaults
/etc/ansible/roles/post_install_caas_infra/defaults/main.yaml
/etc/ansible/roles/post_install_caas_infra/tasks
/etc/ansible/roles/post_install_caas_infra/tasks/main.yaml
/etc/ansible/roles/post_install_caas_infra/templates
/etc/ansible/roles/post_install_caas_infra/templates/danm-tenant-config.yaml.j2
/etc/ansible/roles/pre_install_caas_infra
/etc/ansible/roles/pre_install_caas_infra/meta
/etc/ansible/roles/pre_install_caas_infra/meta/main.yaml
/etc/ansible/roles/pre_install_caas_infra/tasks
/etc/ansible/roles/pre_install_caas_infra/tasks/main.yaml
/etc/ansible/roles/pre_install_caas_infra/tasks/prepare_ca_bundle.yaml
/etc/ansible/roles/pre_install_caas_infra/tasks/prepare_metrics.yaml

In current wrl image:

root@controller-1:~# rpm -ql caas-infra-charts-1.0.0+34-r0.corei7_64|grep roles
/etc/ansible/roles
/etc/ansible/roles/infra_chart_reconfig
/etc/ansible/roles/infra_chart_reconfig/defaults
/etc/ansible/roles/infra_chart_reconfig/defaults/main.yaml
/etc/ansible/roles/infra_chart_reconfig/tasks
/etc/ansible/roles/infra_chart_reconfig/tasks/main.yaml
/etc/ansible/roles/install_caas_infra
/etc/ansible/roles/install_caas_infra/defaults
/etc/ansible/roles/install_caas_infra/defaults/main.yaml
/etc/ansible/roles/install_caas_infra/meta
/etc/ansible/roles/install_caas_infra/meta/main.yaml
/etc/ansible/roles/install_caas_infra/tasks
/etc/ansible/roles/install_caas_infra/tasks/main.yaml
/etc/ansible/roles/pre_install_caas_infra
/etc/ansible/roles/pre_install_caas_infra/meta
/etc/ansible/roles/pre_install_caas_infra/meta/main.yaml
/etc/ansible/roles/pre_install_caas_infra/tasks
/etc/ansible/roles/pre_install_caas_infra/tasks/main.yaml
/etc/ansible/roles/pre_install_caas_infra/tasks/prepare_ca_bundle.yaml
/etc/ansible/roles/pre_install_caas_infra/tasks/prepare_metrics.yaml

Package porting: update python-keystone and add into image

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

The version in wrlinux is 12.0.0, and it's 13.0.2 in REC build185.

so we need to update the version and add it into image.

Reference info:
In build 185:

[newadmin@controller-1 ~]$ rpm -ql openstack-keystone-13.0.2-2.el7.noarch
/etc/keystone
/etc/keystone/default_catalog.templates
/etc/keystone/keystone-paste.ini
/etc/keystone/keystone.conf
/etc/keystone/logging.conf
/etc/keystone/policy.json
/etc/keystone/sso_callback_template.html
/etc/logrotate.d/openstack-keystone
/usr/bin/keystone-manage
/usr/bin/keystone-wsgi-admin
/usr/bin/keystone-wsgi-public
/usr/bin/openstack-keystone-sample-data
/usr/lib/sysctl.d/openstack-keystone.conf
/usr/share/doc/openstack-keystone-13.0.2
/usr/share/doc/openstack-keystone-13.0.2/README.rst
/usr/share/keystone
/usr/share/keystone/keystone-dist.conf
/usr/share/keystone/keystone-schema.json
/usr/share/keystone/keystone-schema.yaml
/usr/share/keystone/policy.v3cloudsample.json
/usr/share/keystone/sample_data.sh
/usr/share/keystone/wsgi-keystone.conf
/usr/share/licenses/openstack-keystone-13.0.2
/usr/share/licenses/openstack-keystone-13.0.2/LICENSE
/usr/share/man/man1/keystone-manage.1.gz
/var/lib/keystone
/var/log/keystone
/var/log/keystone/keystone.log

Failed on TASK [bootstrap-host : Run os-net-config command]

Failed on TASK [bootstrap-host : Run os-net-config command]

TASK [bootstrap-host : Run os-net-config command] ******************************
fatal: [controller-1]: FAILED! => {"changed": true, "cmd": ["os-net-config", "--detailed-exit-codes", "-v", "-c", "/etc/os-net-config/config.json"], "delta": "0:00:00.682815", "end": "2020-01-17 19:47:32.002752", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2020-01-17 19:47:31.319937", "stderr": "[2020/01/17 07:47:31 PM] [INFO] Using config file at: /etc/os-net-config/config.json\n[2020/01/17 07:47:31 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml\n[2020/01/17 07:47:31 PM] [INFO] ENI net config provider created.\n[2020/01/17 07:47:31 PM] [WARNING] Config file failed schema validation at network_config/2:\n    {'persist_mapping': False, 'routes': [{'ip_netmask': '', 'next_hop': '172.20.1.26'}], 'addresses': [{'ip_netmask': '172.20.1.26/29'}], 'nic_mapping': None, 'device': 'bond0', 'use_dhcp': False, 'type': 'vlan', 'vlan_id': 144, 'mtu': 9000} is not valid under any of the given schemas\n  Sub-schemas tested and not matching:\n  - items/oneOf/vlan/routes/items/ip_netmask/oneOf: '' is not valid under any of the given schemas\n  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf: '' is not valid under any of the given schemas\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv4_cidr_string/pattern: '' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/(3[0-2]|[1-2][0-9]|[0-9])$'\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv6_cidr_string/pattern: '' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9])$'\n  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf: '' is not valid under any of the given schemas\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/0/type: '' is not of type 'object'\n  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/1/type: '' is not of type 'object'\n  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/2/type: '' is not of type 'object'\n[2020/01/17 07:47:31 PM] [INFO] nic1 mapped to: eth4\nTraceback (most recent call last):\n  File \"/usr/bin/os-net-config\", line 10, in <module>\n    sys.exit(main())\n  File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 204, in main\n    provider.add_object(obj)\n  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 86, in add_object\n    self.add_linux_bond(obj)\n  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 167, in add_linux_bond\n    raise NotImplementedError(\"add_linux_bond is not implemented.\")\nNotImplementedError: add_linux_bond is not implemented.", "stderr_lines": ["[2020/01/17 07:47:31 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/01/17 07:47:31 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml", "[2020/01/17 07:47:31 PM] [INFO] ENI net config provider created.", "[2020/01/17 07:47:31 PM] [WARNING] Config file failed schema validation at network_config/2:", "    {'persist_mapping': False, 'routes': [{'ip_netmask': '', 'next_hop': '172.20.1.26'}], 'addresses': [{'ip_netmask': '172.20.1.26/29'}], 'nic_mapping': None, 'device': 'bond0', 'use_dhcp': False, 'type': 'vlan', 'vlan_id': 144, 'mtu': 9000} is not valid under any of the given schemas", "  Sub-schemas tested and not matching:", "  - items/oneOf/vlan/routes/items/ip_netmask/oneOf: '' is not valid under any of the given schemas", "  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf: '' is not valid under any of the given schemas", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv4_cidr_string/pattern: '' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/(3[0-2]|[1-2][0-9]|[0-9])$'", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv6_cidr_string/pattern: '' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9])$'", "  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf: '' is not valid under any of the given schemas", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/0/type: '' is not of type 'object'", "  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/1/type: '' is not of type 'object'", "  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/2/type: '' is not of type 'object'", "[2020/01/17 07:47:31 PM] [INFO] nic1 mapped to: eth4", "Traceback (most recent call last):", "  File \"/usr/bin/os-net-config\", line 10, in <module>", "    sys.exit(main())", "  File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 204, in main", "    provider.add_object(obj)", "  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 86, in add_object", "    self.add_linux_bond(obj)", "  File \"/usr/lib/python2.7/site-packages/os_net_config/__init__.py\", line 167, in add_linux_bond", "    raise NotImplementedError(\"add_linux_bond is not implemented.\")", "NotImplementedError: add_linux_bond is not implemented."], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************
controller-1               : ok=19   changed=10   unreachable=0    failed=1    skipped=13   rescued=0    ignored=0

[2020-01-17 19:47:32,079   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook setup-playbook.yml failed
[2020/01/20 04:33:32 AM] [WARNING] Config file failed schema validation at network_config/4:
    {'persist_mapping': False, 'routes': [{'ip_netmask': '', 'next_hop': '172.20.1.26'}], 'addresses': [{'ip_netmask': '172.20.1.26/29'}], 'nic_mapping': None, 'device': 'eth5', 'use_dhcp': False, 'type': 'vlan', 'vlan_id': 144, 'mtu': 9000} is not valid under any of the given schemas
  Sub-schemas tested and not matching:
  - items/oneOf/vlan/routes/items/ip_netmask/oneOf: '' is not valid under any of the given schemas
  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf: '' is not valid under any of the given schemas
  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv4_cidr_string/pattern: '' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/(3[0-2]|[1-2][0-9]|[0-9])$'
  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv6_cidr_string/pattern: '' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9])$'
  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf: '' is not valid under any of the given schemas
  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/0/type: '' is not of type 'object'
  --- items/oneOf/vlan/routes/items/ip_netmask/oneOf/param/oneOf/1/type: '' is not of type 'object'
  -- items/oneOf/vlan/routes/items/ip_netmask/oneOf/2/type: '' is not of type 'object'
[2020/01/20 04:33:32 AM] [INFO] nic1 mapped to: eth4
[2020/01/20 04:33:32 AM] [INFO] adding interface: eth6
[2020/01/20 04:33:32 AM] [INFO] adding interface: eth5
[2020/01/20 04:33:32 AM] [INFO] adding interface: eth4
[2020/01/20 04:33:32 AM] [INFO] adding vlan: vlan142
[2020/01/20 04:33:32 AM] [INFO] adding vlan: vlan144
[2020/01/20 04:33:32 AM] [INFO] adding custom route for interface: vlan144
Traceback (most recent call last):
  File "/usr/bin/os-net-config", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 204, in main
    provider.add_object(obj)
  File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 56, in add_object
    self.add_vlan(obj)
  File "/usr/lib/python2.7/site-packages/os_net_config/impl_eni.py", line 181, in add_vlan
    self._add_routes(vlan.name, vlan.routes)
  File "/usr/lib/python2.7/site-packages/os_net_config/impl_eni.py", line 193, in _add_routes
    rt = netaddr.IPNetwork(route.ip_netmask)
  File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 938, in __init__
    raise AddrFormatError('invalid IPNetwork %s' % addr)
netaddr.core.AddrFormatError: invalid IPNetwork

Playbook bootstrapping-playbook.yml failed on TASK [ntp : enable ntpdate at boot time]

Error:

TASK [ntp : enable ntpdate at boot time] ***************************************
task path: /etc/ansible/roles/ntp/tasks/main.yml:30
Saturday 08 February 2020  15:50:01 +0000 (0:00:00.222)       0:00:03.807 *****
container_name: "controller-1"
physical_host: "controller-1"
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
container_name: "controller-1"
physical_host: "controller-1"
<172.20.1.26> ESTABLISH SSH CONNECTION FOR USER: newadmin
<172.20.1.26> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=newadmin -o ConnectTimeout=5 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=64 -o ServerAliveCountMax=1024 -o Compression=no -o TCPKeepAlive=yes -o VerifyHostKeyDNS=no -o ForwardX11=no -o ForwardAgent=yes -T -o ControlPath=/home/newadmin/.ansible/cp/%h 172.20.1.26 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-jwxguqfialtfomjilqkheaowpannuajn; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<172.20.1.26> (1, '\n{"changed": true, "end": "2020-02-08 15:50:02.103187", "stdout": "", "cmd": "chkconfig ntpdate on", "failed": true, "delta": "0:00:00.003283", "stderr": "error reading information on service ntpdate: No such file or directory", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": true, "_raw_params": "chkconfig ntpdate on", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2020-02-08 15:50:02.099904", "msg": "non-zero return code"}\n', '')
fatal: [controller-1]: FAILED! => {
    "changed": true,
    "cmd": "chkconfig ntpdate on",
    "delta": "0:00:00.003283",
    "end": "2020-02-08 15:50:02.103187",
    "failed": true,
    "invocation": {
        "module_args": {
            "_raw_params": "chkconfig ntpdate on",
            "_uses_shell": true,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2020-02-08 15:50:02.099904",
    "stderr": "error reading information on service ntpdate: No such file or directory",
    "stderr_lines": [
        "error reading information on service ntpdate: No such file or directory"
    ],
    "stdout": "",
    "stdout_lines": []
}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
controller-1               : ok=6    changed=4    unreachable=0    failed=1

Package porting: python2-virtualbmc

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

In build185:

[newadmin@controller-1 ~]$ rpm -qa|grep virtualbmc
python2-virtualbmc-1.2.0-1.el7.noarch
[newadmin@controller-1 ~]$ rpm -ql python2-virtualbmc-1.2.0-1.el7.noarch
/usr/bin/vbmc
/usr/lib/python2.7/site-packages/virtualbmc
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/PKG-INFO
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/SOURCES.txt
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/dependency_links.txt
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/entry_points.txt
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/not-zip-safe
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/pbr.json
/usr/lib/python2.7/site-packages/virtualbmc-1.2.0-py2.7.egg-info/top_level.txt
/usr/lib/python2.7/site-packages/virtualbmc/__init__.py
/usr/lib/python2.7/site-packages/virtualbmc/__init__.pyc
/usr/lib/python2.7/site-packages/virtualbmc/__init__.pyo
/usr/lib/python2.7/site-packages/virtualbmc/cmd
/usr/lib/python2.7/site-packages/virtualbmc/cmd/__init__.py
/usr/lib/python2.7/site-packages/virtualbmc/cmd/__init__.pyc
/usr/lib/python2.7/site-packages/virtualbmc/cmd/__init__.pyo
/usr/lib/python2.7/site-packages/virtualbmc/cmd/vbmc.py
/usr/lib/python2.7/site-packages/virtualbmc/cmd/vbmc.pyc
/usr/lib/python2.7/site-packages/virtualbmc/cmd/vbmc.pyo
/usr/lib/python2.7/site-packages/virtualbmc/config.py
/usr/lib/python2.7/site-packages/virtualbmc/config.pyc
/usr/lib/python2.7/site-packages/virtualbmc/config.pyo
/usr/lib/python2.7/site-packages/virtualbmc/exception.py
/usr/lib/python2.7/site-packages/virtualbmc/exception.pyc
/usr/lib/python2.7/site-packages/virtualbmc/exception.pyo
/usr/lib/python2.7/site-packages/virtualbmc/log.py
/usr/lib/python2.7/site-packages/virtualbmc/log.pyc
/usr/lib/python2.7/site-packages/virtualbmc/log.pyo
/usr/lib/python2.7/site-packages/virtualbmc/manager.py
/usr/lib/python2.7/site-packages/virtualbmc/manager.pyc
/usr/lib/python2.7/site-packages/virtualbmc/manager.pyo
/usr/lib/python2.7/site-packages/virtualbmc/utils.py
/usr/lib/python2.7/site-packages/virtualbmc/utils.pyc
/usr/lib/python2.7/site-packages/virtualbmc/utils.pyo
/usr/lib/python2.7/site-packages/virtualbmc/vbmc.py
/usr/lib/python2.7/site-packages/virtualbmc/vbmc.pyc
/usr/lib/python2.7/site-packages/virtualbmc/vbmc.pyo
/usr/share/licenses/python2-virtualbmc-1.2.0
/usr/share/licenses/python2-virtualbmc-1.2.0/LICENSE

Bootstrapping playbooks failed on TASK [galera_client : Drop local .my.cnf file]

Error: fatal: [controller-1]: FAILED! => {"failed": true, "msg": "'galera_root_password' is undefined"}

logs in cm.log:

PLAY [Install galera server] ***************************************************

TASK [galera_client : Gather variables for each operating system] **************
Friday 14 February 2020  05:11:15 -0500 (0:00:00.126)       0:00:41.752 *******
ok: [controller-1] => (item=/etc/ansible/roles/galera_client/vars/redhat-7.yml)

TASK [galera_client : Drop local .my.cnf file] *********************************
Friday 14 February 2020  05:11:15 -0500 (0:00:00.044)       0:00:41.796 *******
fatal: [controller-1]: FAILED! => {"failed": true, "msg": "'galera_root_password' is undefined"}

PLAY RECAP *********************************************************************
controller-1               : ok=76   changed=35   unreachable=0    failed=1

Friday 14 February 2020  05:11:15 -0500 (0:00:00.052)       0:00:41.849 *******
===============================================================================
partfs_rootdisk : Running a script to remount volume on target dir ----- 12.54s
partfs_rootdisk : Creating LVM ------------------------------------------ 4.98s
partfs_rootdisk : Stop all cloud-init services and disable them --------- 4.59s
Report installation progress -------------------------------------------- 3.59s
partfs_rootdisk : delete temporary partition ---------------------------- 1.39s
Set Timezone ------------------------------------------------------------ 1.33s
Set Timezone ------------------------------------------------------------ 1.17s
partfs_rootdisk : Create fstab entry. ----------------------------------- 1.04s
partfs_rootdisk : Create XFS filesystem on LVM -------------------------- 1.04s
partfs_rootdisk : Create /tmp/<node_name>localstorage.json file --------- 0.54s
ntp : Copy the ntp.conf template file ----------------------------------- 0.49s
partfs_rootdisk : Create PV VG ------------------------------------------ 0.46s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.34s
partfs_rootdisk : remove physical volume (if any) ----------------------- 0.30s
Report installation progress -------------------------------------------- 0.29s
partfs_rootdisk : Create Partition for LVMs ----------------------------- 0.28s
Report installation progress -------------------------------------------- 0.28s
partfs_rootdisk : remove physical volume (if any) ----------------------- 0.28s
Report installation progress -------------------------------------------- 0.28s
ntp : enable ntpdate at boot time --------------------------------------- 0.26s
[2020-02-14 05:11:15,784   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

postconfig-playbook.yml: ansible module 'openssl_certificate' is missing

Error:

ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/etc/ansible/roles/update_cert/tasks/main.yml': line 58, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Validate certificate
  ^ here


The error appears to have been in '/etc/ansible/roles/update_cert/tasks/main.yml': line 58, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Validate certificate
  ^ here

exception type: <class 'ansible.errors.AnsibleParserError'>
exception: no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/etc/ansible/roles/update_cert/tasks/main.yml': line 58, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Validate certificate
  ^ here
[2020-02-07 02:59:05,557   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook postconfig-playbook.yml failed

The play called by playbook postconfig-playbook.yml:
/etc/ansible/roles/update_cert/tasks/main.yml

- name: Validate certificate
  openssl_certificate:
    path: "{{ pem_location }}"
    provider: assertonly
    has_expired: False
  tags:
    - runtime
  when: upload_pem == True or deploy_pem == True

In current wrl image:

root@controller-1:~# ansible-doc openssl_certificate
 [WARNING]: module openssl_certificate not found in: /home/root/.ansible/plugins/modules:/usr/share/ansible/plugins/modules:/usr/lib/python2.7/site-packages/ansible/modules

In build 185:

[newadmin@controller-1 roles]$ ansible-doc openssl_certificate |head
> OPENSSL_CERTIFICATE    (/usr/lib/python2.7/site-packages/ansible/modules/crypto/openssl_certificate.py)

        This module allows one to (re)generate OpenSSL certificates. It implements a notion of provider (ie. `selfsigned', `acme', `assertonly') for your certificate. The 'assertonly' provider is intended for use
        cases where one is only interested in checking properties of a supplied certificate. Many properties that can be specified in this module are for validation of an existing or newly generated certificate.
        The proper place to specify them, if you want to receive a certificate with these properties is a CSR (Certificate Signing Request). It uses the pyOpenSSL python library to interact with OpenSSL.

OPTIONS (= is mandatory):

- acme_accountkey
        Path to the accountkey for the `acme' provider

[newadmin@controller-1 roles]$ rpm -qf /usr/lib/python2.7/site-packages/ansible/modules/crypto/openssl_certificate.py
ansible-2.4.4.0-1.el7.ans.noarch

Playbook bootstrapping-playbook.yml failed on TASK [ntp : Install the required packages in Redhat derivatives]

Error:

PLAY [Configure NTP service on controllers] ************************************

TASK [ntp : command] ***********************************************************
Saturday 08 February 2020  05:38:02 +0000 (0:00:00.279)       0:00:04.869 *****
changed: [controller-1]

TASK [ntp : Add the OS specific variables] *************************************
Saturday 08 February 2020  05:38:02 +0000 (0:00:00.151)       0:00:05.021 *****
ok: [controller-1]

TASK [ntp : Install the required packages in Redhat derivatives] ***************
Saturday 08 February 2020  05:38:02 +0000 (0:00:00.032)       0:00:05.053 *****
fatal: [controller-1]: FAILED! => {"changed": false, "failed": true, "msg": "The Python 2 bindings for rpm are needed for this module. If you require Python 3 support use the `dnf` Ansible module instead.. The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
controller-1               : ok=7    changed=4    unreachable=0    failed=1

Saturday 08 February 2020  05:38:02 +0000 (0:00:00.359)       0:00:05.413 *****
===============================================================================
Report installation progress -------------------------------------------- 2.71s
Gathering Facts --------------------------------------------------------- 1.44s
ntp : Install the required packages in Redhat derivatives --------------- 0.36s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.33s
Report installation progress -------------------------------------------- 0.28s
ntp : command ----------------------------------------------------------- 0.15s
initial_poweroff_hosts : set_fact --------------------------------------- 0.05s
ntp : Add the OS specific variables ------------------------------------- 0.03s
[2020-02-08 05:38:02,923   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

Bootstrapping playbooks failed on TASK [partfs_rootdisk : Grow xfs filesystem on Root partition]

Error:

TASK [partfs_rootdisk : Check if Root partition can grow] **********************
Friday 14 February 2020  04:49:13 -0500 (0:00:22.473)       0:00:48.741 *******
fatal: [controller-1]: FAILED! => {"changed": true, "cmd": "growpart --dry-run $(readlink -f /dev/sda) 1", "delta": "0:00:00.003485", "end": "2020-02-14 04:49:13.148069", "failed": true, "msg": "non-zero return code", "rc": 127, "start": "2020-02-14 04:49:13.144584", "stderr": "/bin/sh: growpart: command not found", "stderr_lines": ["/bin/sh: growpart: command not found"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [partfs_rootdisk : Grow Root partition] ***********************************
Friday 14 February 2020  04:49:13 -0500 (0:00:00.153)       0:00:48.894 *******

TASK [partfs_rootdisk : Grow xfs filesystem on Root partition] *****************
Friday 14 February 2020  04:49:13 -0500 (0:00:00.013)       0:00:48.907 *******
fatal: [controller-1]: FAILED! => {"changed": true, "cmd": ["xfs_growfs", "/"], "delta": "0:00:00.007624", "end": "2020-02-14 04:49:13.312731", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2020-02-14 04:49:13.305107", "stderr": "xfs_growfs: / is not a mounted XFS filesystem", "stderr_lines": ["xfs_growfs: / is not a mounted XFS filesystem"], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************
controller-1               : ok=55   changed=28   unreachable=0    failed=1

Friday 14 February 2020  04:49:13 -0500 (0:00:00.138)       0:00:49.046 *******
===============================================================================
partfs_rootdisk : Running a script to remount volume on target dir ----- 22.47s
partfs_rootdisk : Creating LVM ------------------------------------------ 6.23s
Report installation progress -------------------------------------------- 3.58s
partfs_rootdisk : Stop all cloud-init services and disable them --------- 2.59s
Set Timezone ------------------------------------------------------------ 2.17s
Set Timezone ------------------------------------------------------------ 1.30s
partfs_rootdisk : Create XFS filesystem on LVM -------------------------- 1.16s
partfs_rootdisk : Create fstab entry. ----------------------------------- 1.02s
partfs_rootdisk : Create PV VG ------------------------------------------ 0.98s
partfs_rootdisk : Create /tmp/<node_name>localstorage.json file --------- 0.53s
ntp : Copy the ntp.conf template file ----------------------------------- 0.49s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.32s
partfs_rootdisk : delete temporary partition ---------------------------- 0.31s
partfs_rootdisk : get volume groups ------------------------------------- 0.29s
partfs_rootdisk : get volume groups ------------------------------------- 0.28s
Report installation progress -------------------------------------------- 0.28s
partfs_rootdisk : get volume groups ------------------------------------- 0.28s
Report installation progress -------------------------------------------- 0.27s
partfs_rootdisk : remove physical volume (if any) ----------------------- 0.27s
partfs_rootdisk : Create Partition for LVMs ----------------------------- 0.27s
[2020-02-14 04:49:13,382   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

Failed to load user config /etc/userconfig/user_config.yaml

root@intel-x86-64:/srv/deployment/log# cat bootstrap.log 
Mon Jan 13 07:15:16 UTC 2020 (info) main =====================================================================
Mon Jan 13 07:15:16 UTC 2020 (info) main Boot strapping the environment with /etc/userconfig/user_config.yaml
Mon Jan 13 07:15:16 UTC 2020 (info) main =====================================================================
Mon Jan 13 07:15:16 UTC 2020 (info) main Bootstrapping started
Mon Jan 13 07:15:16 UTC 2020 (info) start_db Starting redis db using /usr/bin/redis-server ./redis.conf
Mon Jan 13 07:15:16 UTC 2020 (info) start_db Wait till DB is serving
Mon Jan 13 07:15:16 UTC 2020 (info) run_cmd Running /usr/bin/redis-cli -h 127.0.0.1 --scan --pattern '*'
Mon Jan 13 07:15:16 UTC 2020 (info) run_cmd Command succeeded: 
Mon Jan 13 07:15:16 UTC 2020 (info) start_cm Starting CM server
Mon Jan 13 07:15:16 UTC 2020 (info) start_cm cmserver pid is 7873
Mon Jan 13 07:15:16 UTC 2020 (info) start_cm Wait till cmserver is ready to serve
Mon Jan 13 07:15:16 UTC 2020 (info) start_cm cmserver not ready yet, got error Failed with error: HTTPConnectionPool(host='127.0.0.1', port=61100): Max retries exceeded with url: /cm/v1.0/properties?prop-name-filter=.* (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3a5368d590>: Failed to establish a new connection: [Errno 111] Connection refused',))
Mon Jan 13 07:15:18 UTC 2020 (info) start_installation Start installation
Mon Jan 13 07:15:18 UTC 2020 (info) handle_user_config Handling user configuration from file /etc/userconfig/user_config.yaml
Mon Jan 13 07:15:18 UTC 2020 (info) run_cmd Running /usr/local/bin/cmcli --ip 127.0.0.1 --port 61100 --client-lib cmframework.lib.cmclientimpl.CMClientImpl bootstrap --config /etc/userconfig/user_config.yaml --plugin_path /opt/cmframework/userconfighandlers
Mon Jan 13 07:15:20 UTC 2020 (error) run_cmd Failed with error Failed with error: Failed to load user config /etc/userconfig/user_config.yaml: 'str' object has no attribute 'get'
Mon Jan 13 07:15:20 UTC 2020 (info) cleanup Cleaning up
Mon Jan 13 07:15:20 UTC 2020 (info) stop_process Stopping process 7873
Mon Jan 13 07:15:20 UTC 2020 (info) stop_process Shutting down process 7873 gracefully
Mon Jan 13 07:15:20 UTC 2020 (info) run_cmd Running pkill -TERM -g 7873
Mon Jan 13 07:15:20 UTC 2020 (info) run_cmd Command succeeded: 
Mon Jan 13 07:15:20 UTC 2020 (info) stop_process Waiting for process 7873 to exit
Mon Jan 13 07:15:20 UTC 2020 (info) stop_process Process 7873 exited

ERROR cmactivateserverhandler(100)] Skipping installationactivator

Error:

Failed with error: [Errno 2] No such file or directory: '/etc/lcm/playbooks/installation/postconfig/.yaml'
[2020-02-08 03:46:20,695   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Failed to run /usr/local/bin/cmcli ansible-playbooks-generate

Package update to build 185: dpdk may need to update to 18.11-4

In current wrl akraino:

root@controller-1:~# rpm -ql dpdk-17.11.3-r0.intel_x86_64
/usr
/usr/bin
/usr/bin/dpdk-pdump
/usr/bin/dpdk-pdump.map
/usr/bin/dpdk-pmdinfo
/usr/bin/dpdk-procinfo
/usr/bin/dpdk-procinfo.map
/usr/bin/dpdk-test-crypto-perf
/usr/bin/dpdk-test-crypto-perf.map
/usr/bin/dpdk-test-eventdev
/usr/bin/dpdk-test-eventdev.map
/usr/bin/testpmd
/usr/bin/testpmd.map
/usr/lib
/usr/sbin
/usr/sbin/dpdk-devbind
/usr/share
/usr/share/dpdk
/usr/share/dpdk/usertools
/usr/share/dpdk/usertools/cpu_layout.py
/usr/share/dpdk/usertools/dpdk-devbind.py
/usr/share/dpdk/usertools/dpdk-pmdinfo.py
/usr/share/dpdk/usertools/dpdk-setup.sh
/usr/share/dpdk/usertools/dpdk_nic_bind.py
/usr/share/dpdk/usertools/setup.sh

In build 185:

[newadmin@controller-1 ~]$ rpm -qa|grep dpdk
dpdk-tools-18.11-4.el7_6.x86_64
dpdk-18.11-4.el7_6.x86_64
[newadmin@controller-1 ~]$ rpm -ql dpdk-tools-18.11-4.el7_6.x86_64
/usr/sbin/dpdk-devbind
/usr/share/dpdk/usertools
/usr/share/dpdk/usertools/cpu_layout.py
/usr/share/dpdk/usertools/cpu_layout.pyc
/usr/share/dpdk/usertools/cpu_layout.pyo
/usr/share/dpdk/usertools/dpdk-devbind.py
/usr/share/dpdk/usertools/dpdk-devbind.pyc
/usr/share/dpdk/usertools/dpdk-devbind.pyo
/usr/share/dpdk/usertools/dpdk-pmdinfo.py
/usr/share/dpdk/usertools/dpdk-pmdinfo.pyc
/usr/share/dpdk/usertools/dpdk-pmdinfo.pyo
/usr/share/dpdk/usertools/dpdk-telemetry-client.py
/usr/share/dpdk/usertools/dpdk-telemetry-client.pyc
/usr/share/dpdk/usertools/dpdk-telemetry-client.pyo

Bootstrapping playbooks failed on TASK [partfs_rootdisk : Create Partition for LVMs]

Error:

TASK [partfs_rootdisk : Create Partition for LVMs] *****************************
Friday 14 February 2020  03:50:25 -0500 (0:00:00.193)       0:00:20.440 *******

TASK [partfs_rootdisk : Create Partition for LVMs] *****************************
Friday 14 February 2020  03:50:25 -0500 (0:00:00.015)       0:00:20.455 *******
fatal: [controller-1]: FAILED! => {"changed": false, "err": "Error: Partition(s) on /dev/sda are being used.\n", "failed": true, "msg": "Error while running parted script: /usr/sbin/parted -s -m -a optimal /dev/sda -- unit KiB mklabel msdos", "out": "", "rc": 1}

PLAY RECAP *********************************************************************
controller-1               : ok=45   changed=21   unreachable=0    failed=1

Friday 14 February 2020  03:50:25 -0500 (0:00:00.174)       0:00:20.629 *******
===============================================================================
partfs_rootdisk : Stop all cloud-init services and disable them --------- 5.59s
Report installation progress -------------------------------------------- 3.59s
Set Timezone ------------------------------------------------------------ 2.31s
ntp : Start/stop ntp service -------------------------------------------- 1.24s
Set Timezone ------------------------------------------------------------ 1.17s
partfs_rootdisk : Create /tmp/<node_name>localstorage.json file --------- 0.52s
ntp : Copy the ntp.conf template file ----------------------------------- 0.49s
partfs_rootdisk : delete temporary partition ---------------------------- 0.37s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.33s
Report installation progress -------------------------------------------- 0.29s
partfs_rootdisk : get volume groups ------------------------------------- 0.28s
partfs_rootdisk : get volume groups ------------------------------------- 0.28s
partfs_rootdisk : get volume groups ------------------------------------- 0.28s
Report installation progress -------------------------------------------- 0.27s
partfs_rootdisk : remove physical volume (if any) ----------------------- 0.27s
ntp : enable ntpdate at boot time --------------------------------------- 0.26s
partfs_rootdisk : Delete old LVMs if not mounted already ---------------- 0.24s
ntp : Clear step-tickers ------------------------------------------------ 0.22s
partfs_rootdisk : Remove /tmp/<node_name>localstorage.json file --------- 0.19s
partfs_rootdisk : Create Partition for LVMs ----------------------------- 0.17s
[2020-02-14 03:50:25,290   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

Package porting: galera

It is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

In build185:
[newadmin@controller-1 ~]$ rpm -ql galera-25.3.16-3.el7.x86_64
/etc/sysconfig/garb
/usr/lib/systemd/system/garbd.service
/usr/lib64/galera
/usr/lib64/galera/libgalera_smm.so
/usr/sbin/garbd
/usr/sbin/garbd-wrapper
/usr/share/doc/galera
/usr/share/doc/galera/COPYING
/usr/share/doc/galera/LICENSE.asio
/usr/share/doc/galera/LICENSE.chromium
/usr/share/doc/galera/LICENSE.crc32
/usr/share/doc/galera/README
/usr/share/doc/galera/README-MySQL

Package update to build 185: facter

In current wrl akraino image:
root@controller-1:~# rpm -qa|grep facter
facter-2.5.0-r0.corei7_64

In build 185:
[newadmin@controller-1 ~]$ rpm -ql facter-3.9.3-7.el7.x86_64
/usr/bin/facter
/usr/lib64/libfacter.so
/usr/lib64/libfacter.so.3.9.3
/usr/share/doc/facter-3.9.3
/usr/share/doc/facter-3.9.3/README.md
/usr/share/licenses/facter-3.9.3
/usr/share/licenses/facter-3.9.3/LICENSE
/usr/share/man/man8/facter.8.gz

Provisioning playbook failed on TASK [baremetal_provision : Checking for any deployment failures]

Tested with:
controller-1: Akraino build 185
Guest image: wrl-akraino-image-rec 20200208

Error:

TASK [baremetal_provision : Checking for any deployment failures] **************
Monday 10 February 2020  21:43:01 -0500 (0:06:28.102)       0:07:12.839 *******
fatal: [controller-2 -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "One or more nodes failed in deployment. [{u'provision_state': u'deploy failed', u'name': u'controller-2'}]"}
fatal: [controller-3 -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "One or more nodes failed in deployment. [{u'provision_state': u'deploy failed', u'name': u'controller-3'}]"}

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
controller-1               : ok=25   changed=20   unreachable=0    failed=0
controller-2               : ok=28   changed=14   unreachable=0    failed=1
controller-3               : ok=24   changed=12   unreachable=0    failed=1

Monday 10 February 2020  21:43:01 -0500 (0:00:00.052)       0:07:12.892 *******
===============================================================================
baremetal_provision : Verify node provisioning state. Waiting for 60mins max. - 388.10s
baremetal_provision : Sleep for 15 seconds ----------------------------- 15.04s
baremetal_provision : Calculate md5sum of Golden image ----------------- 13.50s
baremetal_provision : Create baremetal ironic nodes --------------------- 1.55s
baremetal_provision : Start Baremetal deployment ------------------------ 1.41s
baremetal_provision : Configure Baremetal deployment -------------------- 1.25s
baremetal_provision : Ensure nodes are powered off ---------------------- 1.23s
baremetal_provision : Disable shellinabox console logging for target nodes --- 0.91s
baremetal_provision : Power off nodes ----------------------------------- 0.90s
baremetal_provision : Setup tmux ---------------------------------------- 0.76s
Remove all entries in /etc/exports -------------------------------------- 0.45s
Create access entries for BMC ports of all nodes in /etc/exports from /remote_image_share_root/ --- 0.44s
Create nginx conf file -------------------------------------------------- 0.41s
Report installation progress -------------------------------------------- 0.35s
baremetal_provision : Generate config-drive files ----------------------- 0.34s
baremetal_provision : set_fact ------------------------------------------ 0.33s
Create nginx service for provisioning ----------------------------------- 0.33s
baremetal_provision : Create directories for tmux ----------------------- 0.33s
Enable nfs-server ------------------------------------------------------- 0.32s
baremetal_provision : Block traffic on infra_internal ------------------- 0.32s
[2020-02-10 21:43:02,073   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook provisioning-playbook.yml failed

Bootstrapping playbooks failed on TASK [partfs_rootdisk : Create PV VG]

Error:

TASK [partfs_rootdisk : Create PV VG] ******************************************
Friday 14 February 2020  08:43:54 -0500 (0:00:00.030)       0:00:23.501 *******
fatal: [controller-1]: FAILED! => {"changed": false, "err": "  WARNING: Failed to connect to lvmetad. Falling back to device scanning.\n  Can't open /dev/sda2 exclusively.  Mounted filesystem?\n  Can't open /dev/sda2 exclusively.  Mounted filesystem?\n", "failed": true, "msg": "Creating physical volume '/dev/sda2' failed", "rc": 5}

PLAY RECAP *********************************************************************
controller-1               : ok=52   changed=25   unreachable=0    failed=1

Friday 14 February 2020  08:43:54 -0500 (0:00:00.647)       0:00:24.148 *******
===============================================================================
partfs_rootdisk : Stop all cloud-init services and disable them --------- 5.59s
Report installation progress -------------------------------------------- 4.00s
Gathering Facts --------------------------------------------------------- 1.47s
Set Timezone ------------------------------------------------------------ 1.34s
ntp : Start/stop ntp service -------------------------------------------- 1.24s
ntp : restart ntp ------------------------------------------------------- 1.20s
Set Timezone ------------------------------------------------------------ 1.17s
partfs_rootdisk : Create PV VG ------------------------------------------ 0.65s
ntp : Copy the ntp.conf template file ----------------------------------- 0.53s
partfs_rootdisk : Create /tmp/<node_name>localstorage.json file --------- 0.41s
ntp : add ntp keys ------------------------------------------------------ 0.37s
partfs_rootdisk : delete temporary partition ---------------------------- 0.33s
initial_poweroff_hosts : Ensure all the nodes in the deployment are powered-off. --- 0.33s
Report installation progress -------------------------------------------- 0.29s
Report installation progress -------------------------------------------- 0.28s
partfs_rootdisk : get volume groups ------------------------------------- 0.27s
partfs_rootdisk : get volume groups ------------------------------------- 0.27s
partfs_rootdisk : get volume groups ------------------------------------- 0.27s
partfs_rootdisk : remove physical volume (if any) ----------------------- 0.27s
partfs_rootdisk : Delete old LVMs if not mounted already ---------------- 0.27s
[2020-02-14 08:43:54,787   ERROR cmactivateserverhandler(100)] Skipping installationactivator, got exception Playbook bootstrapping-playbook.yml failed

Package update to build 185: cloud-init

we have a very old verion of cloud-init in wrl and it doesn't work well.

root@intel-x86-64:~# rpm -qa|grep cloud-init
cloud-init-0.7.6-r0.corei7_64
cloud-init-systemd-0.7.6-r0.corei7_64

in build 185:

[root@controller-1 ~]# rpm -qa|grep cloud-init
cloud-init-18.2-1.el7.centos.2.x86_64

Package porting: authconfig

authconfig is required in:
https://gerrit.akraino.org/r/gitweb?p=ta/manifest.git;a=blob_plain;f=packages.yaml;hb=HEAD

But it's not needed in build185, so we will add it later.

It's introduced in:
https://gerrit.akraino.org/r/gitweb?p=ta/infra-ansible.git;a=commit;h=e5776805848728d0aac93078223585f725b84c5e

For reference, in CentOS:

[newadmin@controller-1 roles]$ rpm -qa|grep authconfig
authconfig-6.2.8-30.el7.x86_64
[newadmin@controller-1 roles]$ rpm -ql authconfig|egrep -v 'locale|man|doc'
/etc/pam.d/fingerprint-auth-ac
/etc/pam.d/password-auth-ac
/etc/pam.d/postlogin-ac
/etc/pam.d/smartcard-auth-ac
/etc/pam.d/system-auth-ac
/etc/sysconfig/authconfig
/usr/lib64/python2.7/site-packages/acutilmodule.so
/usr/sbin/authconfig
/usr/sbin/authconfig-tui
/usr/sbin/cacertdir_rehash
/usr/share/authconfig
/usr/share/authconfig/authconfig-tui.py
/usr/share/authconfig/authconfig-tui.pyc
/usr/share/authconfig/authconfig-tui.pyo
/usr/share/authconfig/authconfig.py
/usr/share/authconfig/authconfig.pyc
/usr/share/authconfig/authconfig.pyo
/usr/share/authconfig/authinfo.py
/usr/share/authconfig/authinfo.pyc
/usr/share/authconfig/authinfo.pyo
/usr/share/authconfig/dnsclient.py
/usr/share/authconfig/dnsclient.pyc
/usr/share/authconfig/dnsclient.pyo
/usr/share/authconfig/msgarea.py
/usr/share/authconfig/msgarea.pyc
/usr/share/authconfig/msgarea.pyo
/usr/share/authconfig/shvfile.py
/usr/share/authconfig/shvfile.pyc
/usr/share/authconfig/shvfile.pyo
/var/lib/authconfig

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.