Setup ONAP-Amsterdam CNF vagrant demo environment on Packet.net. about onap-demo

[taylor]

ISsue found: Vagrant hanging on SSH Private key insertion randomly.

Reference tickets

• hashicorp/vagrant#7610
• hashicorp/vagrant#7673
• hashicorp/vagrant#8157 (edited)

disable inserting key hashicorp/vagrant#5186
zunayed
#5186 "Warning: Authentication failure. Retrying... " after packaging box
I am able to vagrant up and have ssh running fine from a base box that I've provisioned.
I then package up the said box and keep getting errors with ssh. I've packaged numerious boxes before with this same process just yesterday. Not sure what the issue is now.

vagrant package --base "<name from VBoxManage list vms>"
Show more…
Assignees

mitchellh

Labels

bug, core

hashicorp/vagrantJan 14th, 2015 at 17:20 Added by GitHub

Taylor Carpenter [40 minutes ago]
possible key sometimes overriding the random key hashicorp/vagrant#5186 (comment)
skinneejoe
Comment on #5186 "Warning: Authentication failure. Retrying... " after packaging box
Below is an explanation of why this problem occurs, why solutions in this thread work for some and not others, how Vagrant has failed us, and a side note on how Ubuntu has failed in creating their base boxes.

If you are having trouble understanding the solutions in this thread please read my experiences below, I think it will help you.

How Vagrant authenticates to boxes Show more…
hashicorp/vagrantJun 30th, 2017 at 14:03 Added by GitHub

---

Solution: Add config.ssh.insert_key = false to Vagrantfile

[taylor]

• OpenStack (devstack) install error: Currently installed pip version 1 does not meet minimum requirements

==> control: 2018-04-18 14:49:32.325 | [ERROR] /home/vagrant/devstack/inc/python:146 Currently installed pip version 1 does not meet minimum requi
rements (>=6).                                                           
==> control: 2018-04-18 14:49:33.333 | ++ ./stack.sh:main:739                      :   err_trap       
==> control: 2018-04-18 14:49:33.335 | ++ ./stack.sh:err_trap:524                  :   local r=1                                                  
==> control: 2018-04-18 14:49:33.339 | stack.sh failed: full log in stack.sh.log.2018-04-18-144818
==> control: 2018-04-18 14:49:33.341 | Error on exit                                                                                              
==> control: 2018-04-18 14:49:33.342 | ./stack.sh: line 508: generate-subunit: command not found
The SSH command responded with a non-zero exit status. Vagrant                                                                                    
assumes that this means the command failed. The output for this command             
should be in the log above. Please read the output to determine what                                                                              
went wrong.

[taylor]

regarding the pip version

https://bugs.launchpad.net/devstack/+bug/1764760

talks about splitting version to find it. The last post is 4/17/2018

---

fixed in master http://git.openstack.org/cgit/openstack-dev/devstack/tree/inc/python?h=master#n340

old code used strip (http://git.openstack.org/cgit/openstack-dev/devstack/tree/inc/python?h=stable/newton#n142)

    local pip_version
    pip_version=$(python -c "import pip; \
                        print(pip.__version__.strip('.')[0])")
    if (( pip_version<6 )); then
        die $LINENO "Currently installed pip version ${pip_version} does not" \
            "meet minimum requirements (>=6)."
    fi

when it should have used split

    local pip_version
    pip_version=$(python -c "import pip; \
                        print(pip.__version__.split('.')[0])")
    if (( pip_version<6 )); then
        die $LINENO "Currently installed pip version ${pip_version} does not" \
            "meet minimum requirements (>=6)."
    fi

SOLUTION: create patch and apply from bootstrap.sh

pip_version_check_fix.patch

[taylor]

New Issue: No stable/newton branch on keystone repository

stable/newton is what the onap demo uses for devstack install. It is not available under the same name for keystone so devstack fails to install

2018-04-18 16:57:28.399 | + functions-common:git_clone:540           :   git checkout stable/newton                                               
2018-04-18 16:57:28.403 | error: pathspec 'stable/newton' did not match any file(s) known to git.  

Newton has been EOL: https://github.com/openstack/keystone/tree/newton-eol

Devstack does not have a newton-eol tag instead they have a stable/newton branch https://github.com/openstack-dev/devstack/tree/stable/newton

[taylor]

Issue: compute node service failed to start and devstack timesout with hostname error

    compute-1: 2018-04-19 16:40:12.341 | Didn't find service registered by hostname after 120 seconds
    compute-1: 2018-04-19 16:40:12.344 | + functions:wait_for_compute:420           :   openstack --os-cloud devstack-admin --os-region RegionOne compute service list
    compute-1: 2018-04-19 16:40:13.259 | +----+------------------+---------+----------+---------+-------+----------------------------+
    compute-1: 2018-04-19 16:40:13.259 | | ID | Binary           | Host    | Zone     | Status  | State | Updated At                 |
    compute-1: 2018-04-19 16:40:13.259 | +----+------------------+---------+----------+---------+-------+----------------------------+
    compute-1: 2018-04-19 16:40:13.259 | |  4 | nova-conductor   | control | internal | enabled | up    | 2018-04-19T16:40:07.000000 |
    compute-1: 2018-04-19 16:40:13.259 | |  6 | nova-scheduler   | control | internal | enabled | up    | 2018-04-19T16:40:12.000000 |
    compute-1: 2018-04-19 16:40:13.259 | |  7 | nova-consoleauth | control | internal | enabled | up    | 2018-04-19T16:40:09.000000 |
    compute-1: 2018-04-19 16:40:13.259 | |  8 | nova-compute     | control | nova     | enabled | up    | 2018-04-19T16:40:09.000000 |
    compute-1: 2018-04-19 16:40:13.259 | +----+------------------+---------+----------+---------+-------+----------------------------+
    compute-1: 2018-04-19 16:40:13.311 | + functions:wait_for_compute:422           :   return 124
    compute-1: 2018-04-19 16:40:13.314 | + lib/nova:is_nova_ready:1                 :   exit_trap
    compute-1: 2018-04-19 16:40:13.318 | + ./stack.sh:exit_trap:494                 :   local r=124
    compute-1: 2018-04-19 16:40:13.322 | ++ ./stack.sh:exit_trap:495                 :   jobs -p
    compute-1: 2018-04-19 16:40:13.328 | + ./stack.sh:exit_trap:495                 :   jobs=
    compute-1: 2018-04-19 16:40:13.333 | + ./stack.sh:exit_trap:498                 :   [[ -n '' ]]
    compute-1: 2018-04-19 16:40:13.338 | + ./stack.sh:exit_trap:504                 :   kill_spinner
    compute-1: 2018-04-19 16:40:13.343 | + ./stack.sh:kill_spinner:390              :   '[' '!' -z '' ']'
    compute-1: 2018-04-19 16:40:13.347 | + ./stack.sh:exit_trap:506                 :   [[ 124 -ne 0 ]]
    compute-1: 2018-04-19 16:40:13.350 | + ./stack.sh:exit_trap:507                 :   echo 'Error on exit'
    compute-1: 2018-04-19 16:40:13.351 | Error on exit
    compute-1: 2018-04-19 16:40:13.354 | + ./stack.sh:exit_trap:508                 :   generate-subunit 1524155317 696 fail
    compute-1: 2018-04-19 16:40:13.635 | + ./stack.sh:exit_trap:509                 :   [[ -z /opt/stack/logs ]]
    compute-1: 2018-04-19 16:40:13.639 | + ./stack.sh:exit_trap:512                 :   /home/vagrant/devstack/tools/worlddump.py -d /opt/stack/logs
    compute-1: 2018-04-19 16:40:14.220 | + ./stack.sh:exit_trap:518                 :   exit 124
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
root@cnfs-dev:/cnfs/demo-amsterdam/vagrant# vim Vagrantfile                         
Taglist: Exuberant ctags (http://ctags.sf.net) not found in PATH. Plugin is not loaded.
Press ENTER or type command to continue   

On compute node in screen session (screen -x under vagrant) the following was found

2018-04-19 16:38:10.764 11207 INFO nova.service [-] Starting compute node (version 15.1.1)
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service [-] Error starting thread.
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service Traceback (most recent call last):
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service   File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 722, in run_service
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service     service.start()
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 144, in start
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service     self.manager.init_host()
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1140, in init_host
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service     raise exception.PlacementNotConfigured()
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service PlacementNotConfigured: This compute is not configured to talk to the placement service. Configure the [placement] section of nova.conf and restart the service.
2018-04-19 16:38:10.766 11207 ERROR oslo_service.service
2018-04-19 16:38:10.766 11207 DEBUG oslo_concurrency.lockutils [req-5efe3368-5adc-4f6e-b3b2-fd5381939556 - -] Acquired semaphore "singleton_lock" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212
2018-04-19 16:38:10.767 11207 DEBUG oslo_concurrency.lockutils [req-5efe3368-5adc-4f6e-b3b2-fd5381939556 - -] Releasing semaphore "singleton_lock" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225

POSSIBLE SOLUTION: add placement-api to enabled services in local.conf

[taylor]

Issue: public endpoint for orchestration service not found

heat service not started.

Enabled in control.conf. destroyed and brought back up the control node

enable_plugin heat https://git.openstack.org/openstack/heat stable/ocata
ENABLED_SERVICES+=heat,h-eng,h-api,h-api-cfn,h-api-cw 

[taylor]

Issue: heat template references xxlarge flavor that is not set causing onap setup to exit

Updated setup_contoller.sh to create a xxlarge flavor and added to the create_onap script.

create_onap.sh:

sed -i "s/flavor_xxlarge:.*/flavor_xxlarge: m1.xxlarge/" onap_openstack.env

setup_controller.sh:

openstack flavor delete m1.xxlarge
openstack flavor create --public m1.xxlarge --id auto --ram 8196 --vcpus 4 --disk 100

[taylor]

Issue: ubuntu packages not available

    onap: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tiff/libtiff5_4.0.6-1ubuntu0.3_amd64.deb  404  Not Found [IP: 91.189.88.149 80]
    onap:                                 
    onap: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?                                                  
The SSH command responded with a non-zero exit status. Vagrant                       
assumes that this means the command failed. The output for this command              
should be in the log above. Please read the output to determine what                 
went wrong.        

Onap-Node needs to run apt update before installing openstack packages.

Add:

apt update -y

to the onap provision script.

[taylor]

Issue: resolution for domain portal.api.simpledemo.openecomp.org

    onap:  192.168.1.218 dcae.api.simpledemo.openecomp.org                           
    onap: + ssh -o StrictHostKeyChecking=no [email protected] -i onap 'curl sina.com.cn'                                
    onap: ssh: Could not resolve hostname portal.api.simpledemo.openecomp.org: Name or service not known                                         
The SSH command responded with a non-zero exit status. Vagrant                       
assumes that this means the command failed. The output for this command              
should be in the log above. Please read the output to determine what                 
went wrong.

[taylor]

Issue: onap vms that are created get assigned a floating ip from the NAT "public subnet" and is not SSH'able from host. Bridging does not work on Packet.net

Fix:
@denverwilliams said:

So i had to use an internal virtual box network "layer 2/switch" network for all the vms and when we bring up the openstack compute nodes we also bring up a router vm which can provide internet access to the internal network.

[taylor]

Issue: Openstack hosts not mapped to cells error on Ocata release

Solution: Run nova-manage cell_v2 simple_cell_setup after the compute nodes are up before using any OpenStack resources

[taylor]

Issue: ONAP mso-api-handler-infra failed to start service - class path issue?

17:00:36,019 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment MSOInfrastructureBPMN-1.2
.1.war (runtime-name: MSOInfrastructureBPMN-1.2.1.war) in 80492ms
17:00:55,673 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0059: Class Path entry scala-reflect.jar in /content/mso-api-handler-infra-1.2.1.war/WEB-INF/lib/scala-compiler-2.12.4.jar  does not point to a valid jar for a Class-Path reference.   17:00:55,679 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0059: Class Path entry scala-library.jar in /content/mso-api-handler-infra-1.2.1.war/WEB-INF/lib/scala-compiler-2.12.4.jar  does not point to a valid jar for a Class-Path reference.   17:00:56,353 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.deployment.unit."mso-api-handler-infra-1.2.1.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."mso-api-handler-infra-
1.2.1.war".STRUCTURE: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)                                         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                                                    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Container is down
        at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:716)
        at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:223)
        at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2401)
        at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:223)                                                        at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2401)
        at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:317)                                                      at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:208)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)                                at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
        ... 3 more

[taylor]

Copying steps from cncf/cnf-testbed#3 (comment)

Steps to setup new system and provision openstack:

Provision base dev software and pull down onap-demo code:

git clone [email protected]:cncf/onap-demo.git -b amsterdam-openstack-ocata

cd onap-demo

./vagrant/run_demo.sh

Deploy the router to new packet server:

cd router-node ; vagrant plugin install vagrant-reload ; vagrant up

Deploy openstack to new packet server and run onap creation scripts:

cd ../vagrant ; vagrant up

After this completes OpenStack (ocata release) will be running.

Some ONAP components will be up others fail.

[LexCC]

Copying steps from cncf/cnf-testbed#3 (comment)

Steps to setup new system and provision openstack:

Provision base dev software and pull down onap-demo code:

git clone [email protected]:cncf/onap-demo.git -b amsterdam-openstack-ocata

cd onap-demo

./vagrant/run_demo.sh

Deploy the router to new packet server:

cd router-node ; vagrant plugin install vagrant-reload ; vagrant up

Deploy openstack to new packet server and run onap creation scripts:

cd ../vagrant ; vagrant up

After this completes OpenStack (ocata release) will be running.

Some ONAP components will be up others fail.

I got an error
fatal: unable to access 'https://git.openstack.org/openstack/heat/': Failed to connect to git.openstack.org port 443: No route to host

[taylor]

@JieJhih, this repo is not currently being maintained and is only hear for historical purposes. Testing and development of Cloud Native Network Function use cases has moved to the CNCF CNF Testbed.

For current ONAP documentation see the ONAP Wiki (specifically the Developing ONAP page or ONAP docs site (Amsterdam, Casablanca]. The current version of the ONAP demo code is at https://github.com/onap/demo. The ONAP Mailing list are a good place to get help.

That said, it looks like the issue you are seeing is caused by git.openstack.org being replaced by a new host opendev.org. So the URL would be https://opendev.org/openstack/heat. To resolve the error the openstack install code would need to be updated to support the new URL.

The intent of the instructions was for initial analysis of the ONAP amsterdam demo which builds a OpenStack cluster inside of virtual machines. Most of the effort was making sure the host machines and networking was repeatable and updating the OpenStack code to work. This was not meant to be used long term though and it served it's purpose.

I'm not sure what your goals are with trying to get this code up and running, but in general I recommend not using it. For ONAP use the newer releases. For general CNF use case testing there are several choices including the CNCF CNF Testbed.

You can reach the CNF Testbed team on CNCF's Slack in the #cnf-testbed channel in addition to the Github repo https://github.com/cncf/cnf-testbed.

[taylor]

Closing this ticket as complete for original intent.

Open new tickets for any further issues -- noting that the repo is not currently maintained.