erigones / esdc-ce Goto Github PK

Danube Cloud :: Community Edition

License: Apache License 2.0

Python 62.97% Shell 5.68% HTML 9.54% JavaScript 13.69% CSS 2.95% Ruby 0.01% Perl 1.89% PLpgSQL 0.36% DTrace 0.05% Erlang 0.38% C 0.17% SCSS 1.67% Jinja 0.60% Awk 0.05%

automation cloud danubecloud hacktoberfest kvm monitoring orchestration server smartos virtualization

esdc-ce's People

Contributors

Stargazers

Watchers

esdc-ce's Issues

Add support for erigones image tags into POST image_manage

API calls related to image imports (either POST image_manage or POST imagestore_image_manage) ignore our image tags (deploy, resize and erigones attributes in the tags metadata).

Alerting management (Zabbix actions)

Configure Zabbix actions via esdc API/GUI.

Consider removal of empty hostgroups after this task is implemented .

Restore backup to new VM

When restoring backup, user may choose the VM, into which the data will be restored. It would be usefull if there was an option to restore to new server.
Specifically: first item in the list of VMs can be <<new VM>> and new VM will be created according to definition of original VM.
Currently, the user has to create and deploy new VM manually before the restore.
Jan

Images › repository default view

Hi,
Where: Datacenter -> Images -> Import disk Image from repository.
What: it would be nice if there was longer default time period for filtering repository images. Last month seems to be too strict and it doesn't show many relevant images.
Thx.
Jan

Update names of KVM OS types

Append VM to Linux, BSD, Windows and SunOS ostypes to indicate that these are KVM virtual machines.

Documentation (api docs and user guide) needs to be updated afterwards.

Users and groups one-way synchronization to Zabbix

Dynamically create users and user groups in Zabbix according to Groups in esdc.

Update target_hostname in PUT vm_backup to accept uuid

After #16 was implemented there is still one API call parameter that depends on VM's hostname - target_hostname in PUT /vm/(hostname_or_uuid)/backup/(bkpname). Please update this parameter so it also accepts VM's uuid as value.

Please make the new target_hostname_or_uuid required and remove the default value.

NOTE: Renaming the parameter will also affect the GUI (python or JS).

Add Control+Alt+Delete to on-screen console keyboard

The Ctrl-Alt-Del shortcut is frequently used to access the login window on a Windows machine.

Error during MON_ZABBIX_TEMPLATES_VM_NIC and MON_ZABBIX_TEMPLATES_VM_DISK settings validation

2017-01-01 13:45:31,354 ERROR [api.views]: placeholder_validator() argument after ** must be a mapping, not NoneType
Traceback (most recent call last):
  File "/opt/erigones/api/views.py", line 484, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/erigones/api/decorators.py", line 91, in handler
    return func(*args, **kwargs)
  File "/opt/erigones/api/decorators.py", line 212, in wrap
    return fun(request, *args, **kwargs)
  File "/opt/erigones/api/dc/base/views.py", line 170, in dc_settings
    return DcSettingsView(request, dc, data).response()
  File "/opt/erigones/api/api_views.py", line 98, in response
    return fun(*args, **kwargs)
  File "/opt/erigones/api/dc/base/dc_settings.py", line 38, in put
    if not ser.is_valid():
  File "/opt/erigones/api/serializers.py", line 647, in is_valid
    return not self.errors
  File "/opt/erigones/api/serializers.py", line 639, in errors
    ret = self.from_native(data, files)
  File "/opt/erigones/api/serializers.py", line 464, in from_native
    attrs = self.perform_validation(attrs)
  File "/opt/erigones/api/serializers.py", line 395, in perform_validation
    attrs = validate_method(attrs, source)
  File "/opt/erigones/api/dc/base/serializers.py", line 564, in validate_MON_ZABBIX_TEMPLATES_VM_NIC
    return validate_array_placeholders(attrs, source, VM_KWARGS.copy().update({'net': 1, 'nic_id': 2}))
  File "/opt/erigones/api/dc/base/serializers.py", line 40, in validate_array_placeholders
    placeholder_validator(i, **valid_placeholders)
TypeError: placeholder_validator() argument after ** must be a mapping, not NoneType

Synchronize user media in Zabbix according to user profile settings in esdc

User editable notes for VM and node (wiki?)

The data (notes) can be stored in existing json fields in DB.

Add vm_uuid into output of VM-related API calls

Continuation of issue #16
The field should be a read-only field similar to the hostname attribute.
Affected files identified so far:

api/vm/status/serializers.py
api/vm/backup/serializers.py
api/vm/snapshot/serializers.py
api/mon/vm/serializers.py
api/network/ip/serializers.py - this needs to be changed in issue #3 - which is already discussed in PR #4 (comment)

The monitoring synchronization is called twice during new VM deployment

Add help_text to VM tags input field

Add help_text to VM tags explaining that a new tag can be created by pressing Enter.

Reveal UUID of VM in API and GUI

add support for working with VMs with uuid as required parameter (API)
show uuid in GET vm_manage (/vm/(hostname)) and GET vm_define (/vm/(hostname)/define) output (read-only)
show uuid in VM details (GUI)

IP addr settings are not updated on notcreated VM

Hi,
there's a problem with json handling. It affects IP address changing after VM is destroyed and re-deployed. Maybe it also affects other VM settings.

Steps to reproduce:

create VM with some IP address, deploy it
stop & destroy the VM, so it goes to notcreated state
change the IP address
deploy it again

The VM will be deployed with the old IP address.

Jan

Alert listing (Zabbix events) in API and GUI

Add views for listing actual and historical alerts per VM, Node and DC (similar to task log list).

Replace raw SQL with Django Query Expressions

https://docs.djangoproject.com/en/1.8/ref/models/expressions/

Node graphs not showing when not in main DC

When adding nodem onitoring the call_mon_histoy_task() function was created. This new function is used by both Node and Vm to retrieve history. However, the dc_bound parameter, which for the node should be set to False.

Fix broken links in API documentation

The sphinx API documentation in doc/api/source currently generates broken HTML links. That is because I've moved most of the rst files into a subfolder (api).

It looks like that all broken links are generated through rst_epilog in conf.py, so the change should happen there.

You can use wget to find all 404s (e.g. wget -mkEpnv http://127.0.0.1:8000/static/api/doc/index.html)

More information (i) help texts near input fields

For example, one can use "G" for gigabytes, "M" for megabytes in disk size input.

Display release edition in GET /system/version

Add support for nics.*.allowed_ips

Switch DC link shoud be more visible and intuitive

Currently changing datacenter is not intuitive. Link to modal window to switch DC is confusing for user.

Rename compute node "offline" status to "maintenance"

Edit buttons should be more intuitive

Users are uncertain how to edit items ( VMs, Nodes etc).

It may be beneficial to redesign, rethink the button, or move it somewhere else.

Admin can not add SSH key for user.

Super admit is modify user, and add SSH key for user raises exception:

2016-12-12 16:40:28,270 ERROR [api.views]: Invalid field name(s) given in select_related: 'owner'. Choices are: dc_bound, default_dc, userprofile, auth_token
Traceback (most recent call last):
  File "/home/ricco/Erigones/esdc-ce/api/views.py", line 484, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/api/decorators.py", line 91, in handler
    return func(*args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/api/decorators.py", line 212, in wrap
    return fun(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/api/accounts/user/sshkey/views.py", line 112, in sshkey_manage
    return UserSshkeyView(request, username, title, data).response()
  File "/home/ricco/Erigones/esdc-ce/api/accounts/user/sshkey/api_views.py", line 22, in __init__
    user = get_user(request, username, exists_ok=True, noexists_fail=True)
  File "/home/ricco/Erigones/esdc-ce/api/accounts/user/utils.py", line 29, in get_user
    return get_virt_object(request, User, get_attrs={'username': username}, where=where, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/api/utils/db.py", line 140, in get_virt_object
    obj = get_object(request, model, get_attrs, sr=sr, pr=pr, extra=extra, where=where, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/api/utils/db.py", line 39, in get_object
    obj = qs.filter(where).get(**attrs)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
    sql, params = self.as_sql()
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 378, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 48, in pre_sql_setup
    self.setup_query()
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 39, in setup_query
    self.select, self.klass_info, self.annotation_col_map = self.get_select()
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 206, in get_select
    related_klass_infos = self.get_related_selections(select)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 754, in get_related_selections
    ', '.join(_get_field_choices()) or '(none)',
FieldError: Invalid field name(s) given in select_related: 'owner'. Choices are: dc_bound, default_dc, userprofile, auth_token
2016-12-12 16:40:28,275 ERROR [django.request]: Internal Server Error: /dc/user/[email protected]/profile/ssh_key/add/form/
Traceback (most recent call last):
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/gui/decorators.py", line 24, in wrap
    return fun(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/gui/decorators.py", line 81, in wrap
    return fun(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/gui/decorators.py", line 69, in wrap
    return fun(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/views/decorators/http.py", line 45, in inner
    return func(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/gui/dc/user/views.py", line 230, in dc_user_profile_sshkey_modal_form
    status = sform.save(action='create', args=(user.username, sform.cleaned_data['name']))
  File "/home/ricco/Erigones/esdc-ce/gui/forms.py", line 298, in save
    res = self.api_call(action, self._obj, self._request, args=args, data=data)
  File "/home/ricco/Erigones/esdc-ce/gui/forms.py", line 275, in api_call
    return call_api_view(request, method, cls._api_call.__func__, *args, data=dict(data), log_response=True)
  File "/home/ricco/Erigones/esdc-ce/api/utils/views.py", line 21, in call_api_view
    res = fun(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/ricco/Erigones/esdc-ce/api/views.py", line 487, in dispatch
    response = self.handle_exception(exc)
  File "/home/ricco/Erigones/esdc-ce/api/views.py", line 454, in handle_exception
    raise exc
FieldError: Invalid field name(s) given in select_related: 'owner'. Choices are: dc_bound, default_dc, userprofile, auth_token

Add task retry to mgmt callbacks

When a mgmt callback fails during execution because of an operational error (DBError, Redis/RabbitMQ connection problem) we should retry the task.

Update `update_command` to re-use old SSL key/cert

move ssl_key_file and ssl_cert_file variable into core.settings, name them e.g. UPDATE_KEY_FILE, UPDATE_CERT_FILE
check if key/cert file exists during each run (with key/cert=None) and pass them to the udpate script

Compute node monitoring and graphs (API & GUI) [meta]

Implement advanced compute node monitoring and graphs in the Danube Cloud API and GUI.
The main idea behind this task is to further integrate the monitoring system into Danube Cloud.

Create a new API call: mon_node_history similar to mon_vm_history.

Monitoring items: api/mon/node/graphs.py

CPU tab

CPU utilization
- items: ['system.cpu.util[,user,]', 'system.cpu.util[,system,]']
- stacked graph
1-minute load average
- items: ['system.cpu.load[,avg1]']
CPU jumps
- items: ['system.cpu.switches', 'system.cpu.intr']

Memory tab

Memory usage
- items: vm.memory.size[used]
Swap usage
- items: system.swap.size[,used] - erigones/esdc-factory#21

Networks tab (network interfaces)

2 graphs per each network interface:

NIC ???? bandwidth
- items: ['net.if.in[????]', 'net.if.out[????]']
- stacked graph
NIC ???? packets
- items: ['net.if.in[????,packets]', 'net.if.out[????,packets]']
- stacked graph

Storages tab (disk pools)

2 graphs per each storage pool (zpool = ????):

Storage ???? throughput
- items: ['zpool.iostat[????,nread]', 'zpool.iostat[????,nwritten]']
- stacked graph
Storage ???? I/O
- items: ['zpool.iostat[????,reads]', 'zpool.iostat[????,writes]']
- stacked graph
NEW: Display free/used space on the dataset: zfs.list[????, used] or zfs.list[????, avail]
- either a normal graph or maybe a pie chart showing distribution of the used disk space between: zfs.usedds[????], zfs.usedrefreserv[????] and zfs.usedsnap[????]

Implement update mechanism of Danube Cloud infrastructure/OS services

We need a framework for performing updates and changes of OS packages and services running in our appliances (e.g. changes in PostgreSQL configuration, OS updates, etc.). Let's use Ansible for that. Ansible playbooks should live in this repository.

Fix blank page on 403 (Forbidden)

Fix DELETE /task/(task_id)

The que.utils.delete_task() is broken (celery inspection of a task candidate for deletion). We should fix this and make the DELETE /task/(task_id) available for every PENDING task in the system.

The testing should also include use cases involving task cleanup (revoked tasks).

Target branch: v2.4

KeyError: 'get_image_manifes_url'

Critical typo found in POST imagestore_image_manage:

Traceback (most recent call last):
  File "/Users/dano/erigones/esdc-ce/api/views.py", line 484, in dispatch
    response = handler(request, *args, **kwargs)
  File "/Users/dano/erigones/esdc-ce/api/decorators.py", line 91, in handler
    return func(*args, **kwargs)
  File "/Users/dano/erigones/esdc-ce/api/decorators.py", line 212, in wrap
    return fun(request, *args, **kwargs)
  File "/Users/dano/erigones/esdc-ce/api/imagestore/image/views.py", line 63, in imagestore_image_manage
    return ImageStoreImageView(request, name, uuid, data).response()
  File "/Users/dano/erigones/esdc-ce/api/api_views.py", line 98, in response
    return fun(*args, **kwargs)
  File "/Users/dano/erigones/esdc-ce/api/imagestore/image/api_views.py", line 54, in post
    data['manifest_url'] = self.repo.get_image_manifes_url(img['uuid'])
  File "/Users/dano/erigones/esdc-ce/vms/models/base.py", line 59, in __getattr__
    return self._data[key]
KeyError: 'get_image_manifes_url'

User email address validation problem

Characters allowed in email address field when registering are not the same as characters checked by url regex.
Definitive solution TBD in comments.

Traceback (most recent call last):

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)

  File "/opt/erigones/gui/decorators.py", line 24, in wrap
    return fun(request, *args, **kwargs)

  File "/opt/erigones/gui/decorators.py", line 53, in wrap
    return fun(request, *args, **kwargs)

  File "/opt/erigones/gui/dc/user/views.py", line 67, in dc_user_list
    return render(request, 'gui/dc/user_list.html', context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/shortcuts/__init__.py", line 53, in render
    return HttpResponse(loader.render_to_string(*args, **kwargs),

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/loader.py", line 169, in render_to_string
    return t.render(context_instance)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 140, in render
    return self._render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 203, in render
    nodelist.append(node.render(context))

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render
    return nodelist.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
    bit = self.render_node(node, context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
    return node.render(context)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 447, in render
    six.reraise(*exc_info)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 433, in render
    url = reverse(view_name, args=args, kwargs=kwargs, current_app=context.current_app)

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 536, in reverse
    return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))

  File "/opt/erigones/envs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 456, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))

NoReverseMatch: Reverse for 'dc_user_profile' with arguments '()' and keyword arguments '{u'username': u'[email protected]'}' not found. 1 pattern(s) tried: ['dc/user/(?P[A-Za-z0-9\@\._-]+)/profile/$']

Add API and GUI for listing available monitoring templates and hostgroups

The API call will be used for implementing templates suggestions dropdown in GUI.

Create image from snapshot: compression options (level, type)

Remove ERIGONES_MGMT_WORKERS celery variable

Erigonesd-beat fails to start when database unavailable

When postgresql database is unreachable (or login fails), the erigonesd-beat crashes. After crash, the service stays down. Instead, it should go into infinite connect loop waiting for the database because this can be only temporary problem.

Refocus the form on checkbox when there is an form error

Observed when creating SuperAdmin and leaving DC bound checked.

Warning about unavailable nodes does not go away after node becomes available

Hi,
in GUI serverlist (/servers), if some node is unavailable, the following message appears on the top: "Some servers are not visible or accessible, because some compute nodes are not in online state. Servers are kept untouched. Full server control will be possible as soon as the compute nodes return to online state".

But when the node becomes reachable again and status of all nodes is green, the above message does not go away until the page is manually reloaded.

Also when any node becomes unavailable, the message does not appear until page is reloaded.

Insecure str.format() validation of DC settings

str.format() is a bit unsecure. So lets improve validation on DC settings, which support placeholders (this is the only place where we use format()). This should be done in api.dc.base.serializers.placeholder_validator(). Please move this function into api.validators.

Target branch: master (version: 2.3.3)

Increase swap size on mgmt and mon VMs

Limit log file size in GET /system/logs and GET /system/node/(hostname)/logs

Otherwise gunicorn/erigonesd can potentially consume all memory.

Reveal snapshot and backup IDs

Lets show the user actual snapshot names and backup paths in the API. The idea of this change is to enable users to easily identify certain snapshots/backups on the FS, which might be helpful in some (disaster) situations.

Suggested read-only attributes (let's discuss this first):

Snapshot: id - Snapshot.zfs_name
Backup: file_path - Backup.file_path

(This is a really simple change - 2 lines of code)

Automatic Zabbix host group creation and updates

Dynamically synchronize host groups in Zabbix according to monitoring hostgroups setting and DC association in esdc.

Reload of gunicorn-{sio|api} does not load new application code after upgrade

Both, gunicorn-sio and gunicorn-api use gevent workers and do not reflect new code after SIGHUP.
The problem could be solved by using gunicorn's SIGUSR2 upgrade mechanism, which was used before we've migrated to systemd. With systemd it is not possible to use SIGUSR2 and SIGTERM on the old gunicorn master.

It may be possible to solve the problem just by restarting all gunicorns via systemd.

Fix templates in API docs

There are few problems that fix for this issue needs to address:

Link to routing table is broken when not viewing individual modules
404 html template is broken for erigones theme
Add link to routing table in sphinx_rtd_theme

Update resource allocation for replicated VMs

Add option to skip reservation of vCPUs and RAM for a replicated VM. These resources should be reserved on a compute node only if the VM is promoted to master there.

dhcp_passthrough should be a non-required field in POST net_manage

$  es create /network/(name) -vlan_id 0 -netmask 255.255.255.0 -nic_tag admin -network 192.168.33.0 -dc_bound false
...
"status": "FAILURE"
 "result": {
            "dhcp_passthrough": [
                "This field is required."
            ]
        },
...

Solution: Add a default value (false) to dhcp_passthrough in the serializer (this is already documented in the API docs).

erigones / esdc-ce Goto Github PK

esdc-ce's People

Contributors

Stargazers

Watchers

Forkers

esdc-ce's Issues

CPU tab

Memory tab

Networks tab (network interfaces)

Storages tab (disk pools)

Recommend Projects

Recommend Topics

Recommend Org

Jobs