juniper / contrail-ansible-internal

Contrail-ansible-internal

Ansible code to provision contrail services

This ansible code help to provision contrail services running within the container. This code is supposed to run by contrailctl and not to run by user. contrailctl read its configuration file, pass the configurations to ansible as variables and run this ansible code with appropriate high level playbook found directly under playbooks/ directory.

Any changes that need to go to any internal service configuration file, any internal service management and other things related to internal services within the container should go here.

contrail-ansible-internal's Issues

setting /etc/hosts entry should be happening within the container

Currently lineinfile has an issue due to which we cannot use lineinfile for /etc/hosts in docker which is basically a docker mounted file. Because of this currently it is implemented on base_host.yml which run outside of container so external orchestrator has to handle this unless they use base_host.yml.

Ideally those things to be done inside the container.

lineinfile has option unsafe_writes which fix this, but that is only available from ansible version 2.2 onwards where current stable version available as package is 2.1. May be another option to use shell/command module with sed command.

zookeeper service is handled with lsb scripts

This should ideally be handled with supervisord in container.

hostip should be rightly configured

it should be local machine's IP but in some configs, it is possible to be used loadbalanced IP - example, hostip=control_ip

The docker container should work properly if the configuration is applied while it is running

Unlike the ansible or server manager way of deployment, the Juju charms will not know any of the nodes ipaddress (controller or analytics) upfront. The reason is that, it is deployed this in the Canonical MAAS environment and the MAAS server dynamically allocates ipaddress for these servers via dhcp (this is pretty much what will be there in the customer setup as well). This means that we should not assume that the user has to supply the ipaddress first before bringing up the container. I should be able to bring the container and then apply the configuration as and when I have all the information via ‘contrailctl’ tool. I find that this does not work properly. I have to launch the container with all the relevant ip addresses in the conf file for things to work

Not able to login to webui in a cluster with openstack as orchestrator.

Not able to login to webui in a cluster with openstack as orchestrator.

zk role need to support tags - install, configure

Since we run ansible-code with tags to run different set of code in container build, and run time, all roles should support both install, configure roles.

Currently both install and configure/provisioning code run in both the time for zookeeper.

ubuntu 14-04 contrail-status failing in controller container

== Contrail Support Services ==
supervisor-support-service: active (disabled on boot)
Traceback (most recent call last):
File "/usr/bin/contrail-status", line 492, in
main()
File "/usr/bin/contrail-status", line 480, in main
supervisor_status('support-service', options)
File "/usr/bin/contrail-status", line 403, in supervisor_status
check_status('supervisor-support-service', options)
File "/usr/bin/contrail-status", line 376, in check_status
check_svc_status(svc_name, options.debug, options.detail, options.timeout)
File "/usr/bin/contrail-status", line 317, in check_svc_status
raise Exception("%s does not exist! Cannot check supervisor status." % service_sock)
Exception: /var/run/supervisord_support_service.sock does not exist! Cannot check supervisor status.
root@server12:/#

ansible playbook to provision loadbalancer container is required to support HA contrail provisioning

ansible playbook to provision loadbalancer container is required to support HA contrail provisioning

control_ifmap_username, control_ifmap_password is not configured correctly

control_ifmap_username: "{{ control_ip }}"
control_ifmap_password: "{{ control_ip }}"

dns_ifmap_username: "{{ control_ip }}.dns"
dns_ifmap_password: "{{ control_ip }}.dns"

they can be wrong

roles may be flatter

We would not need infra role at all... would think of it.

Dedicated role should be used to setup kafka

set virtual router type to “embedded” during vrouter provisioning

From: Sachchidanand Vaidya
Date: Tuesday, October 4, 2016 at 12:38 AM
To: Harish Kumar, Andra Cismaru, Artur Dębski, Rudra Rugge
Cc: Praneet Bachheti, Adam Nieżurawski, Paweł Melon
Subject: Re: Ansible issues

Hi Harish,
Can you set virtual router type to “embedded” during vrouter provisioning. This is more
for completeness than any functionality.

Thanks
Sachin

Some variables like contrail_version, contrail_docker_registry may be able to set as shell variables

May be it make sense to allow few variables like contrail_version to be detected from shell variables if not already setup in inventory/group_vars.

Mesos-manager does not have correct IP address for api-server

It seems to pick up the compute node IP address as api server ip. It should pick up the controller api server ip.

centos hostname to be fixed

hostname on centos is hostname.domainname format that need to be fixed to hostname only

possible role of a "system" container

There may be bunch of things related to contrail needed to be done within the host like running ntp, setting some sysctl values, etc and this can be done through one container which run on all nodes

contrail-openstack-control install on docker fail in latest ubuntu trusty docker image

root@3ae8db731bd5:/# /usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install contrail-openstack-control
Reading package lists... Done
Building dependency tree
Reading state information... Done
contrail-openstack-control is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up libpam-systemd:amd64 (204-5ubuntu20.19) ...
invoke-rc.d: unknown initscript, /etc/init.d/systemd-logind not found.
dpkg: error processing package libpam-systemd:amd64 (--configure):
subprocess installed post-installation script returned error exit status 100
Errors were encountered while processing:
libpam-systemd:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

Need one(or more) fully commented sample inventory files

zookeeper should be a separate role

Currently zookeeper is installed part of config

Installing bridge-utils with docker role is not necessary

Since we are using host based networking for docker containers, installing bridgeutils is not required and thus should be removed from docker role

supervisor_*_ready fact only need to be checked on configure/provision tag, not on install tag

Currently those code are running in install time also.

This is not a big deal, just adding here as an item to work on later.

agent to configure with TLS auth

Here is the configs
DEFAULT.xmpp_auth_enable
DEFAULT.xmpp_server_cert, DEFAULT.xmpp_server_key, DEFAULT.xmpp_ca_cert

similarly, we have DEFAULT.xmpp_dns_auth_enable

control node port is different -5222 - TLS based XMPP port

ssl/security in contrail applications yet to be tested/setup

No SSL keys configured at this moment. That need to be done to support ssl based api access

provision_* scripts calls need to be migrated to contrail-ansible

Currently provision_* scripts to register various services to config is called from within entrypoint.sh script, which need to be moved to contrail-ansible and made idempotent.

vnc_api_lib.ini is populated with incorrect AUTHN_URL

vnc_api_lib.ini is not populated with incorrect AUTHN_URL as below
AUTHN_URL = v2.0/tokens

instead of,
AUTHN_URL = /v2.0/tokens

webui contrail-webui-userauth.js is not respected

I made config.orchestration.Manager = 'none'; in config.global.js and then I see default admin credentials just work - (admin/contrail123). Then I changed password in contrail-webui-userauth.js and restarted both webui and middleware but still I am not able to use updated password and old password just work.

Just wanted to follow up with webui team to see if this is expected behavior and may be file bug to them if necessary.

Have the same redis for both analytics and webui

Have the same redis for both analytics and webui as that's the deployment scenario.
And move all the redis related settings (redis password,..) to common defaults file.
Currently redis_password is specified webui defaults file.

There should be a way to understand container name once connect to it

[11/21/16, 10:30:22 PM] SIVA GURUMURTHY: One other thing I noticed and I feel will be good from the end user point of view is
when you ‘docker exec’ it will be great to have the prompt reflect the container you have logged into
[11/21/16, 10:30:44 PM] SIVA GURUMURTHY: Today I have no clue by looking at the terminal in which container I am in now
[11/21/16, 10:31:16 PM] SIVA GURUMURTHY: Need to explore if there is a way to set the hostname when you launch the docker image and set it
[11/21/16, 10:38:01 PM] harishkrishnanmk: you cannot set a hostname when you use host based networking
[11/21/16, 10:38:24 PM] harishkrishnanmk: but we can definitely set bash prompt
[11/21/16, 10:38:42 PM] harishkrishnanmk: and a command to understand the container one logged in

Parallel execution of the contrailctl command is not working

Say for example say I am deploying 3 controllers, 3 analytics node and 3 analyticsdb node in 3 machines (1 controller, 1 analytics and 1 analytics db per machine). In the Juju charm world the ipaddress of the controller,analytics and analyticsdb node itself will be available dynamically when I establish ‘relations’ between them. This means I will dynamically query and get the ipaddress and the write it into the. conf files. This means the configuration to take effect I will have multiple ‘contrailctl config’ sync commands executing simultaneously (this will be for different instances of the same charm as well as multiple instances different charms). It fails in this scenario. Juju provides the facility to scale up and scale down the number of units of a charm however the user wants and so we need this to be fixed for this to work.

nodemanager doesnt work with systemd

when try to start nodemanager with systemd got below message - same message I get when try to run it manually from commandline.

contrail-vrouter-nodemgr.service - Contrail vrouter nodemanager
Loaded: loaded (/etc/systemd/system/contrail-vrouter-nodemgr.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2016-12-12 12:42:19 UTC; 2min 39s ago
Process: 708 ExecStart=/usr/bin/contrail-nodemgr --nodetype=contrail-vrouter (code=exited, status=0/SUCCESS)
Main PID: 708 (code=exited, status=0/SUCCESS)

Dec 12 12:42:19 harish-centos-test-2.localdomain systemd[1]: Started Contrail vrouter nodemanager.
Dec 12 12:42:19 harish-centos-test-2.localdomain contrail-nodemgr[708]: Discovery server: 192.168.0.63
Dec 12 12:42:19 harish-centos-test-2.localdomain contrail-nodemgr[708]: Discovery port: 5998
Dec 12 12:42:19 harish-centos-test-2.localdomain contrail-nodemgr[708]: Collector address: []
Dec 12 12:42:19 harish-centos-test-2.localdomain contrail-nodemgr[708]: Node manager must be run as a supervisor event listener

Remove neutron stuffs from lb for now

Lets remove neutron configs from lb for now(neutron is part of openstack controller), will see what to do when we add openstack to this.

Need support vrouter_physical_interface with "dash" in it e.g br-bond0

As a result the contrail-vrouter-agent service does not come up

provision/register contrail services to config need to be idempotent

Currently the task which does this will always run which means provision_*.py will be running all the time when ansible run with the tag provision. This must be fixed

cassandra and other third party services verification needs to be moved to service.yml

Third party services(cassandra, zookeper, rabbitmq) verification needs to be moved from https://github.com/Juniper/contrail-ansible/blob/9e16dbda7aa45a1c1cf2d0f66087bc2f2796c688/playbooks/roles/contrail/config/tasks/configure.yml#L2 to service.yml and needs to be played when service tag tasks are called.

verify cassandra cluster status before configure/start contrail apps

Cassandra cluster should be up before configuring and start contrail apps which connect to cassandra.

rabbitmq is using lsb service script for service management need to migrate to supervisor

It is better to use supervisor being a single service manager within the container, so this need to be moved to supervisor.

ifmap log configuration is not right

python-memcache package needs to be installed in controller container for openstack sku builds

python-memcache package needs to be installed in controller container for openstack sku builds

https://github.com/Juniper/contrail-ansible/blob/master/playbooks/roles/contrail/config/defaults/main.yml#L51

contrail_docker_registry_insecure and may be other related vars need to have default defined

TASK [docker : configuring init-system...] *************************************
failed: [localhost](item={u'm': u'0644', u'd': u'etc/default', u'f': u'docker'}) => {"failed": true, "item": {"d": "etc/default", "f": "docker", "m": "0644"}, "msg": "AnsibleUndefinedVariable: {% if docker_daemon_args is defined %}{{docker_daemon_args}}{% endif %} {{ docker_systemd_socket }} {{docker_insecure_registries_daemon_args}} {{docker_dns_daemon_args}} {{docker_bip_daemon_args}} {{docker_remote_args}} {{docker_cluster_advertise_args}} {{docker_cluster_store_args}} {{docker_storage_driver_args}}: {% if docker_insecure_registries is defined %} {% if docker_insecure_registries is string %} --insecure-registry {{docker_insecure_registries}} {% else %} {% for ir in docker_insecure_registries %} --insecure-registry {{ir}} {% endfor %} {% endif %} {% endif %}: {%- if contrail_docker_registry_insecure|bool and contrail_docker_registry %} {{ contrail_docker_registry }}{% endif %}: 'contrail_docker_registry_insecure' is undefined"}

agent to configure metadata tls auth

webui may need to point to analyticsdb

Need to confirm with webui team

All roles code need to be pulled under contrail-ansible

All third party modules as per discussion with Ashish need to be kept under thirdparty/ directory under roles.

It would probably need little more time to do this as those roles need to be changed to work in our environment (like supervisor based service management vs init scripts). So I would like to make the movement once those change done separately to those modules.

Also need to think on a strategy to move the code to contrail-ansible with easiest path to maintain some compatibility with upstream role so getting any change in upstream module comparatively easier (I know this is going to be difficult to achieve).

waiting cassandra, kafka etc may be okay if major number of servers are fine

May be better to test cassandra/kafka or any other infra service connectivity by testing a read/write from them rather than just checking the port is up or not. In that case it may be okay just to check one node with or cluster nodes rather than checking all nodes.

redis role need to support tags - install, configure

Since we run ansible-code with tags to run different set of code in container build, and run time, all roles should support both install, configure roles.

Currently both install and configure/provisioning code run in both the time for redis.

nodemanager doesnt support systemd

Because of this nodemanager have to be disabled when systemd service management

Check on disabling setup where possible

This should reduce the execution time.

system hostname should be restored

At least redhat/centos systems are affected

keystone auth config file and vnc_api_lib ini need not be created in lb container.

keystone auth config file and vnc_api_lib ini need not be created in lb container, Current code it trying to create and failing to do it as /etc/contrail dir wont be present in lb container.

contrail-version should be optional

contrail-version should not be mandated as it is sufficient if container_image is defined.

ansible run within container to be logged somewhere

docker logs have some issues especially on centos nodes because of which docker logs doesnt print all the logs