solinea / goldstone-server Goto Github PK
View Code? Open in Web Editor NEWSolinea's open infrastructure management and monitoring server component
Home Page: http://goldstone-server.readthedocs.org/en/latest/
License: Apache License 2.0
Solinea's open infrastructure management and monitoring server component
Home Page: http://goldstone-server.readthedocs.org/en/latest/
License: Apache License 2.0
Add the HEAD and GET APIs. Three total.
The GET will return an empty list.
This will allow the client coding to proceed in parallel with the server coding, and provide a structure for the remaining tickets.
The 'node report' page is in an uncertain state after changes implemented during the new dashboard changeover.
To create this issue I:
Installed 2 addons.
Restarted the containers. Everything was working correctly.
With the addons installed, there are changes to:
Then I needed to switch branches.
And in order to switch branches these changes have to be stashed.
I stashed the changes.
I stopped the container with 'control-c' and waited for it to stop.
Then I switched branches.
Then I did a git stash pop to put the files back that are related to the addons.
Then I started the containers again and the following errors were generated.
(attached text files).
I also tried stashing/stopping/changing branches to master/stash-popping.
The result of that bin/start attempt is the 2nd text file.
Getting the following message when trying to package the RPM.
Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/lib/jenkins/workspace/goldstone-server-make-el7-rpm/master/rpm_packaging/rpmbuild_el7/BUILDROOT/goldstone-server-0.6.0-129.ga09703b.el7.centos.x86_64 error: Installed (but unpackaged) file(s) found: /opt/goldstone/addon_fabfile.py
Update docker-compose files to add a gssearch-dvc container (based on goldstone-search image).
Currently, the kibana container used for dev support comes up with no knowledge of the goldstone indices. We should provide preconfigured settings.
Currently, running fab install will accept params for the cloud credentials that create the Cloud instance in the DB. If params are not provided, then the caller will be prompted interactively for the settings.
In order to create a generic VM or docker based installation, we need to confirm that the installer can skip that part of the post-install. We will also need to confirm that the goldstone-server and celery tasks behave well prior to the user configuring a cloud connection.
The client will need to check for cloud credentials on login, and take the user to the settings page if not found, assuming they have authority to set them. If not, they should be presented with a notice to contact the goldstone admin to configure the settings.
Celery tasks should recover when the settings are available on a future run, but we should confirm that behavior.
Ran ./bin/start_dev_env.sh
and noticed the following during startup:
This environment is from a fully new set of docker images, previous docker-machine was recreated.
gsappdev_1 | [2015-12-03 16:24:42,840: ERROR/MainProcess] Received unregistered task of type u'goldstone.core.tasks.create_daily_index'.
gsappdev_1 | The message has been ignored and discarded.
gsappdev_1 |
gsappdev_1 | Did you remember to import the module containing this task?
gsappdev_1 | Or maybe you are using relative imports?
gsappdev_1 | Please see http://bit.ly/gLye1c for more information.
gsappdev_1 |
gsappdev_1 | The full contents of the message body was:
gsappdev_1 | {u'utc': True, u'chord': None, u'args': [], u'retries': 0, u'expires': None, u'task': u'goldstone.core.tasks.create_daily_index', u'callbacks': None, u'errbacks': None, u'timelimit': [None, None], u'taskset': None, u'kwargs': {}, u'eta': None, u'id': u'f904811d-f58c-426b-9d74-73eb9e0de18b'} (276b)
gsappdev_1 | Traceback (most recent call last):
gsappdev_1 | File "/venv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 455, in on_task_received
gsappdev_1 | strategies[name](message, body,
gsappdev_1 | KeyError: u'goldstone.core.tasks.create_daily_index'
The URL has to end in a /.
There will be no functionality changes.
The client shouldn't have to change, since all the API endpoints will be based off the add-on's root.
Ran fab install after failed previous install (on connecting to OpenStack auth api). Received a python 'name' TypeError on collection of OpenStack resources.
# fab uninstall
# rpm -ivh oldstone-server-0.5.0-0.5.0.el7.centos.x86_64.rpm
# cd /opt/goldstone
# . bin/activate
(goldstone)[root@localhost goldstone]# fab install
Got further this time, but received the following error.
Enter OpenStack auth URL base (ex: http://10.10.10.10:5000/v2.0): http://192.168.0.17:5000/v2.0
Collecting the static files under the web server.
[localhost] local: ./manage.py collectstatic --noinput --settings=goldstone.settings.production
Copying '/opt/goldstone/lib/python2.7/site-packages/django/contrib/admin/static/admin/css/ie.css'
. . .
Copying '/opt/goldstone/lib/python2.7/site-packages/rest_framework_swagger/static/rest_framework_swagger/lib/shred/content.js'
136 static files copied to '/var/www/goldstone/static', 91 unmodified.
Collecting information about Openstack resources.
Traceback (most recent call last):
File "/opt/goldstone/lib/python2.7/site-packages/fabric/main.py", line 743, in main
*args, **kwargs
File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 424, in execute
results['<local-only>'] = task.run(*args, **new_kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/opt/goldstone/fabfile.py", line 707, in install
stack_auth_url=stack_auth_url)
File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 171, in __call__
return self.run(*args, **kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/opt/goldstone/fabfile.py", line 632, in goldstone_init
_reconcile_hosts(settings, install_dir)
File "/opt/goldstone/fabfile.py", line 165, in _reconcile_hosts
reconcile_hosts()
File "/opt/goldstone/goldstone/nova/tasks.py", line 201, in reconcile_hosts
fqdn=[item[1] for item in incoming if item[0] == name][0])
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/manager.py", line 157, in create
return self.get_queryset().create(**kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/query.py", line 320, in create
obj = self.model(**kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/polymorphic/polymorphic_model.py", line 162, in __init__
super(PolymorphicModel, self).__init__(*args, ** kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 417, in __init__
raise TypeError("'%s' is an invalid keyword argument for this function" % list(kwargs)[0])
TypeError: 'name' is an invalid keyword argument for this function
(goldstone)[root@localhost goldstone]#
Migrate build/test from Jenkins build/test to Travis-CI.
New backend libs:
Guessing this was uploaded on a personal account without global permissions:
https://photos.google.com/album/AF1QipPsFIXlFUzuJflAowyshNoDtF3ph9hMAIdK4WGa
line 132 in current HACKING.md file.
When starting the container in a DB down state, Ctrl+C (SIGINT) does not kill the container.
This is a placeholder for @heidecke. Please fill in the detail.
Model after docker community model of contribution:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md
Other reasonably good references:
Interesting approach to collapsing commits:
Current work on "travis" branch. Pending swift object store for ES data extract.
All environment information should be self contained within the containers. Overrides should be documented where necessary, placing environment directives within docker-compose (with the defaults listed and commented out).
start_dev_env.sh requires docker be running and available, but that is not specified in the docs prior to running the script.
In configure_stack.sh it assumes the user's goldstone-server is in ~/devel. We should change this to a variable
$PROJECT_HOME/goldstone-server
The HACKING doc references and RDO image in stored in HP's cloud which will go away soon. The RDO image should be moved to a longer lived location (maybe tcloud.solinea.com?), and the documentation should be updated to point developers at the new location.
HP cloud will become unavailable on Jan 31, 2016.
Just a heads up that line 47
$ /usr/local/Cellar/pyenv-virtualenvwrapper/20140609/bin/pyenv-sh-virtualenvwrapper
refers to a script that is not currently available after implementing brew install pyenv-virtualenvwrapper
Here's the exception details:
[2015-05-15 17:55:03,843: ERROR/MainProcess] Task goldstone.core.tasks.reconcile_hosts[0ece4772-079d-4e8c-8d82-ef99ef983b75] raised unexpected: IntegrityError('null value in column "cloud_id" violates not-null constraint\nDETAIL: Failing row contains (19, 839e120a-fb2b-11e4-aea5-fa163e5566bb, null, , 2015-05-15 17:55:03.824769+00, 2015-05-15 17:55:03.830016+00).\n',)
Traceback (most recent call last):
File "/opt/goldstone/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/goldstone/goldstone/core/tasks.py", line 61, in reconcile_hosts
services()
File "/opt/goldstone/goldstone/nova/utils.py", line 183, in reconcile_nova_hosts
process_resource_type(nodetype)
File "/opt/goldstone/goldstone/core/utils.py", line 254, in process_resource_type
name=entry.get("name", ''))
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/manager.py", line 157, in create
return self.get_queryset().create(**kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/query.py", line 322, in create
obj.save(force_insert=True, using=self.db)
File "/opt/goldstone/lib/python2.7/site-packages/polymorphic/polymorphic_model.py", line 89, in save
return super(PolymorphicModel, self).save(*args, **kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 545, in save
force_update=force_update, update_fields=update_fields)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 572, in save_base
self._save_parents(cls, using, update_fields)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 597, in _save_parents
self._save_table(cls=parent, using=using, update_fields=update_fields)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 654, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 687, in _do_insert
using=using, raw=raw)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/manager.py", line 232, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/query.py", line 1514, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
cursor.execute(sql, params)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/goldstone/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
IntegrityError: null value in column "cloud_id" violates not-null constraint
DETAIL: Failing row contains (19, 839e120a-fb2b-11e4-aea5-fa163e5566bb, null, , 2015-05-15 17:55:03.824769+00, 2015-05-15 17:55:03.830016+00).
Ran ./bin/start_dev_env.sh
and noticed the following during startup:
This environment is from a fully new set of docker images, previous docker-machine was recreated.
gsappdev_1 | [2015-12-03 16:25:06,829: INFO/MainProcess] Task goldstone.core.tasks.update_persistent_graph[66bbf278-b338-4c9c-938e-01ed4204b857] succeeded in 18.996438316s: None
gsappdev_1 | Error: no such option: --prefix
gsappdev_1 | [2015-12-03 16:25:07,078: ERROR/MainProcess] Task goldstone.core.tasks.delete_indices[49e4f510-d6e9-4bd2-927f-1fc776736133] raised unexpected: CalledProcessError()
gsappdev_1 | Traceback (most recent call last):
gsappdev_1 | File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gsappdev_1 | R = retval = fun(*args, **kwargs)
gsappdev_1 | File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
gsappdev_1 | return self.run(*args, **kwargs)
gsappdev_1 | File "/home/app/goldstone/core/tasks.py", line 38, in delete_indices
gsappdev_1 | return check_call(cmd.split())
gsappdev_1 | File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
gsappdev_1 | raise CalledProcessError(retcode, cmd)
gsappdev_1 | CalledProcessError
Seen in the same startup as #158.
Testing slack integration. Please ignore this issue.
After a failed install, ran fab uninstall
from /opt/goldstone and received the following error.
Dependencies Resolved
============================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================
Removing:
goldstone-server x86_64 1431957243:0.5.0-0.5.0.el7.centos @/goldstone-server-0.5.0-0.5.0.el7.centos.x86_64 3.6 M
Transaction Summary
============================================================================================================================================
Remove 1 Package
Installed size: 3.6 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : 1431957243:goldstone-server-0.5.0-0.5.0.el7.centos.x86_64 1/1
Verifying : 1431957243:goldstone-server-0.5.0-0.5.0.el7.centos.x86_64 1/1
Removed:
goldstone-server.x86_64 1431957243:0.5.0-0.5.0.el7.centos
Complete!
[localhost] local: su - postgres -c 'dropdb goldstone'
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[localhost] local: su - postgres -c 'dropuser goldstone'
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[localhost] local: systemctl stop postgresql.service
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[localhost] local: systemctl disable postgresql.service
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
rm '/etc/systemd/system/multi-user.target.wants/postgresql.service'
Done.
Document current process or add to script where necessary.
Calling update_graph throws the exception below.
/Users/stanford/devel/goldstone-server/goldstone/core/utils.pyc in process_resource_type(nodetype)
212 nodetype_native_id_key = nodetype.native_id_key()
213 actual_cloud_instance_ids = set([x.get(nodetype_native_id_key)
--> 214 for x in actual if x])
215
216 resource_nodes = resources.nodes_of_type(nodetype)
TypeError: get() takes exactly 1 argument (2 given)
In [3]:
create a new pivotal issue when a github issue is created. would like to put it under the community epic. will probably take a few tries.
This should be a generalized solution that can be used with ‘known’ addons.
rpm_repos is legacy stuff, and needs to be cleaned up. minimally, we should remove old rpms that reference a non-existant repo. we can stand up a new repo, or find a suitable existing repo to make goldstone-server accessible without using localinstall. The ES and logstash repos should be bumped, and we shouldn't need the epel repo since there is an epel-release package available to centos7.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.