GithubHelp home page GithubHelp logo

solinea / goldstone-server Goto Github PK

View Code? Open in Web Editor NEW
12.0 12.0 9.0 27.5 MB

Solinea's open infrastructure management and monitoring server component

Home Page: http://goldstone-server.readthedocs.org/en/latest/

License: Apache License 2.0

Python 9.72% CSS 5.22% Ruby 0.15% HTML 0.42% JavaScript 83.06% Shell 1.23% Nginx 0.01% Makefile 0.18%

goldstone-server's People

Contributors

bpepple avatar jbc0 avatar joraldaz avatar jxstanford avatar leftees avatar lexjacobs avatar rsmitty avatar seemantk avatar slashk avatar stugots avatar takahigashi avatar vichargrave avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

goldstone-server's Issues

Stub API endpoints

Add the HEAD and GET APIs. Three total.

The GET will return an empty list.

This will allow the client coding to proceed in parallel with the server coding, and provide a structure for the remaining tickets.

Can't shut down and restart container with addons installed. (with branch change)

To create this issue I:

Installed 2 addons.
Restarted the containers. Everything was working correctly.

With the addons installed, there are changes to:
image

Then I needed to switch branches.
And in order to switch branches these changes have to be stashed.
I stashed the changes.
I stopped the container with 'control-c' and waited for it to stop.

Then I switched branches.
Then I did a git stash pop to put the files back that are related to the addons.

Then I started the containers again and the following errors were generated.
(attached text files).
I also tried stashing/stopping/changing branches to master/stash-popping.
The result of that bin/start attempt is the 2nd text file.

addon-stop-start.txt
addon-stop-start2.txt

Jenkins build fails due to file missing in RPM spec

Getting the following message when trying to package the RPM.

Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/lib/jenkins/workspace/goldstone-server-make-el7-rpm/master/rpm_packaging/rpmbuild_el7/BUILDROOT/goldstone-server-0.6.0-129.ga09703b.el7.centos.x86_64 error: Installed (but unpackaged) file(s) found: /opt/goldstone/addon_fabfile.py

installer should allow for unconfigured cloud connection

Currently, running fab install will accept params for the cloud credentials that create the Cloud instance in the DB. If params are not provided, then the caller will be prompted interactively for the settings.

In order to create a generic VM or docker based installation, we need to confirm that the installer can skip that part of the post-install. We will also need to confirm that the goldstone-server and celery tasks behave well prior to the user configuring a cloud connection.

The client will need to check for cloud credentials on login, and take the user to the settings page if not found, assuming they have authority to set them. If not, they should be presented with a notice to contact the goldstone admin to configure the settings.

Celery tasks should recover when the settings are available on a future run, but we should confirm that behavior.

Layout redesigned dashboard and wire up JavaScript

  • Layout redesigned dashboard with image placeholders on new viz'.
  • convert 'single-block' and 'double-block' grid layout system to bootstrap col-md-x scheme.
  • wire up JavaScript.
  • Wire up links in sidebar to render existing chart pages.
  • metrics link to nova compute page with tab to show api perf charts
  • reports page to: logs-vs-time with three tabs or buttons that refer to the three report pages
  • compliance icon for addons

error starting dev env, unregistered task of type u'goldstone.core.tasks.create_daily_index'.

Ran ./bin/start_dev_env.sh and noticed the following during startup:

This environment is from a fully new set of docker images, previous docker-machine was recreated.

gsappdev_1 | [2015-12-03 16:24:42,840: ERROR/MainProcess] Received unregistered task of type u'goldstone.core.tasks.create_daily_index'.
gsappdev_1 | The message has been ignored and discarded.
gsappdev_1 |
gsappdev_1 | Did you remember to import the module containing this task?
gsappdev_1 | Or maybe you are using relative imports?
gsappdev_1 | Please see http://bit.ly/gLye1c for more information.
gsappdev_1 |
gsappdev_1 | The full contents of the message body was:
gsappdev_1 | {u'utc': True, u'chord': None, u'args': [], u'retries': 0, u'expires': None, u'task': u'goldstone.core.tasks.create_daily_index', u'callbacks': None, u'errbacks': None, u'timelimit': [None, None], u'taskset': None, u'kwargs': {}, u'eta': None, u'id': u'f904811d-f58c-426b-9d74-73eb9e0de18b'} (276b)
gsappdev_1 | Traceback (most recent call last):
gsappdev_1 |   File "/venv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 455, in on_task_received
gsappdev_1 |     strategies[name](message, body,
gsappdev_1 | KeyError: u'goldstone.core.tasks.create_daily_index'

Merge existing add-ons into the new compliance add-on

There will be no functionality changes.

The client shouldn't have to change, since all the API endpoints will be based off the add-on's root.

  1. Merge models.py, views.py, urls.py, etc. into new repository.
  2. Adjust urls.py to base off of new API endpoint.
  3. Test interactively, and all existing unit tests.

fab install python 'name' TypeError on collecting OpenStack resources

Ran fab install after failed previous install (on connecting to OpenStack auth api). Received a python 'name' TypeError on collection of OpenStack resources.

# fab uninstall
# rpm -ivh oldstone-server-0.5.0-0.5.0.el7.centos.x86_64.rpm
# cd /opt/goldstone
# . bin/activate
(goldstone)[root@localhost goldstone]# fab install

Got further this time, but received the following error.

Enter OpenStack auth URL base (ex: http://10.10.10.10:5000/v2.0):  http://192.168.0.17:5000/v2.0
Collecting the static files under the web server.

[localhost] local: ./manage.py collectstatic --noinput   --settings=goldstone.settings.production
Copying '/opt/goldstone/lib/python2.7/site-packages/django/contrib/admin/static/admin/css/ie.css'
. . .
Copying '/opt/goldstone/lib/python2.7/site-packages/rest_framework_swagger/static/rest_framework_swagger/lib/shred/content.js'

136 static files copied to '/var/www/goldstone/static', 91 unmodified.
Collecting information about Openstack resources.
Traceback (most recent call last):
  File "/opt/goldstone/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 424, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/opt/goldstone/fabfile.py", line 707, in install
    stack_auth_url=stack_auth_url)
  File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 171, in __call__
    return self.run(*args, **kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/opt/goldstone/fabfile.py", line 632, in goldstone_init
    _reconcile_hosts(settings, install_dir)
  File "/opt/goldstone/fabfile.py", line 165, in _reconcile_hosts
    reconcile_hosts()
  File "/opt/goldstone/goldstone/nova/tasks.py", line 201, in reconcile_hosts
    fqdn=[item[1] for item in incoming if item[0] == name][0])
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/manager.py", line 157, in create
    return self.get_queryset().create(**kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/query.py", line 320, in create
    obj = self.model(**kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/polymorphic/polymorphic_model.py", line 162, in __init__
    super(PolymorphicModel, self).__init__(*args, ** kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 417, in __init__
    raise TypeError("'%s' is an invalid keyword argument for this function" % list(kwargs)[0])
TypeError: 'name' is an invalid keyword argument for this function
(goldstone)[root@localhost goldstone]#

Gather, prune, and organize static assets for dashboard redesign.

  • delete image assets no-longer needed in repo. x
  • gather new image assets for site redesign. x
  • download and test addons before moving forward. x
  • upgrade to latest version of bootstrap. x
  • gather latest update of css for site redesign. x
  • consolidate new css files into @imports in style(Dark|Light).scss x

update CONTRIBUTING.md

integrate travis-ci

Current work on "travis" branch. Pending swift object store for ES data extract.

HACKING.md docker-machine

start_dev_env.sh requires docker be running and available, but that is not specified in the docs prior to running the script.

move RDO instance to a new home

The HACKING doc references and RDO image in stored in HP's cloud which will go away soon. The RDO image should be moved to a longer lived location (maybe tcloud.solinea.com?), and the documentation should be updated to point developers at the new location.

HP cloud will become unavailable on Jan 31, 2016.

HACKING.md virtualenv setup

Just a heads up that line 47
$ /usr/local/Cellar/pyenv-virtualenvwrapper/20140609/bin/pyenv-sh-virtualenvwrapper

refers to a script that is not currently available after implementing brew install pyenv-virtualenvwrapper

reconcile_host throws exception during celery task execution

Here's the exception details:

[2015-05-15 17:55:03,843: ERROR/MainProcess] Task goldstone.core.tasks.reconcile_hosts[0ece4772-079d-4e8c-8d82-ef99ef983b75] raised unexpected: IntegrityError('null value in column "cloud_id" violates not-null constraint\nDETAIL:  Failing row contains (19, 839e120a-fb2b-11e4-aea5-fa163e5566bb, null, , 2015-05-15 17:55:03.824769+00, 2015-05-15 17:55:03.830016+00).\n',)
Traceback (most recent call last):
  File "/opt/goldstone/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/goldstone/goldstone/core/tasks.py", line 61, in reconcile_hosts
    services()
  File "/opt/goldstone/goldstone/nova/utils.py", line 183, in reconcile_nova_hosts
    process_resource_type(nodetype)
  File "/opt/goldstone/goldstone/core/utils.py", line 254, in process_resource_type
    name=entry.get("name", ''))
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/manager.py", line 157, in create
    return self.get_queryset().create(**kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/query.py", line 322, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/goldstone/lib/python2.7/site-packages/polymorphic/polymorphic_model.py", line 89, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 545, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 572, in save_base
    self._save_parents(cls, using, update_fields)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 597, in _save_parents
    self._save_table(cls=parent, using=using, update_fields=update_fields)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 654, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/base.py", line 687, in _do_insert
    using=using, raw=raw)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/manager.py", line 232, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/query.py", line 1514, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
    cursor.execute(sql, params)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/goldstone/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
IntegrityError: null value in column "cloud_id" violates not-null constraint
DETAIL:  Failing row contains (19, 839e120a-fb2b-11e4-aea5-fa163e5566bb, null, , 2015-05-15 17:55:03.824769+00, 2015-05-15 17:55:03.830016+00).

error starting dev env,

Ran ./bin/start_dev_env.sh and noticed the following during startup:

This environment is from a fully new set of docker images, previous docker-machine was recreated.

gsappdev_1 | [2015-12-03 16:25:06,829: INFO/MainProcess] Task goldstone.core.tasks.update_persistent_graph[66bbf278-b338-4c9c-938e-01ed4204b857] succeeded in 18.996438316s: None
gsappdev_1 | Error: no such option: --prefix
gsappdev_1 | [2015-12-03 16:25:07,078: ERROR/MainProcess] Task goldstone.core.tasks.delete_indices[49e4f510-d6e9-4bd2-927f-1fc776736133] raised unexpected: CalledProcessError()
gsappdev_1 | Traceback (most recent call last):
gsappdev_1 |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
gsappdev_1 |     R = retval = fun(*args, **kwargs)
gsappdev_1 |   File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
gsappdev_1 |     return self.run(*args, **kwargs)
gsappdev_1 |   File "/home/app/goldstone/core/tasks.py", line 38, in delete_indices
gsappdev_1 |     return check_call(cmd.split())
gsappdev_1 |   File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
gsappdev_1 |     raise CalledProcessError(retcode, cmd)
gsappdev_1 | CalledProcessError

Seen in the same startup as #158.

fab uninstall postgres cleanup errors when run from /opt/goldstone

After a failed install, ran fab uninstall from /opt/goldstone and received the following error.


Dependencies Resolved

============================================================================================================================================
 Package                Arch         Version                                   Repository                                              Size
============================================================================================================================================
Removing:
 goldstone-server       x86_64       1431957243:0.5.0-0.5.0.el7.centos         @/goldstone-server-0.5.0-0.5.0.el7.centos.x86_64       3.6 M

Transaction Summary
============================================================================================================================================
Remove  1 Package

Installed size: 3.6 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : 1431957243:goldstone-server-0.5.0-0.5.0.el7.centos.x86_64                                                                1/1
  Verifying  : 1431957243:goldstone-server-0.5.0-0.5.0.el7.centos.x86_64                                                                1/1

Removed:
  goldstone-server.x86_64 1431957243:0.5.0-0.5.0.el7.centos

Complete!
[localhost] local: su - postgres -c 'dropdb goldstone'
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[localhost] local: su - postgres -c 'dropuser goldstone'
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[localhost] local: systemctl stop postgresql.service
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[localhost] local: systemctl disable postgresql.service
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
rm '/etc/systemd/system/multi-user.target.wants/postgresql.service'

Done.

update_graph throws exception

Calling update_graph throws the exception below.

/Users/stanford/devel/goldstone-server/goldstone/core/utils.pyc in process_resource_type(nodetype)
    212     nodetype_native_id_key = nodetype.native_id_key()
    213     actual_cloud_instance_ids = set([x.get(nodetype_native_id_key)
--> 214                                      for x in actual if x])
    215
    216     resource_nodes = resources.nodes_of_type(nodetype)

TypeError: get() takes exactly 1 argument (2 given)

In [3]:

remove old binaries from rpm_repos folder and rework for current project environment

rpm_repos is legacy stuff, and needs to be cleaned up. minimally, we should remove old rpms that reference a non-existant repo. we can stand up a new repo, or find a suitable existing repo to make goldstone-server accessible without using localinstall. The ES and logstash repos should be bumped, and we shouldn't need the epel repo since there is an epel-release package available to centos7.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.