axsh / wakame-vdc

Datacenter Hypervisor - Open Source Cloud Computing / IaaS

Wakame-vdc

Wakame-vdc is a virtual datacenter or an IaaS (Infrastructure as a Service) suite.

Visit http://wakame-vdc.org for a more detailed explanation.

System requirements

• Ruby version: 2.0.0-p598
• OS: CentOS 6

Other versions or operating systems may or may not work.

Quick Start

Install Wakame-vdc on CentOS 6

Try Wakame-vdc pre-installed in VirtualBox.

Contact us

Wakame Users Group

The best way to contact us is to send a mail to the Wakame Users Group on Google Groups. Please don't hesitate to ask us any questions. If you want to try out Wakame-vdc, we'll do our best to help you. There's no such thing as a stupid question.

If you happen to be in Tokyo, you can also attend one of our Wakame Users Group events. We do them once every 3 months and they are open to everybody. Further details on http://wakameusersgroup.org.

Social Media

Twitter

• @axsh_en: The English Axsh company account

• @axsh_jp: The Japanese Axsh company account

Documentation

Documentation about Wakame-vdc can be found on http://wakame-vdc.org.

Users

Wakame-vdc is currently used by the following companies.

• Kyushu Electric Power Co., Ltd.
• National Institute of Informatics
• NTT PC Communications
  • NTT PC Web Arena VPS Cloud
• Kyocera Communication Systems Co., Ltd.
• TIS Inc.

If you already use this software, please let us know. Thank you.

Contributors

Special thanks to all contributors for submitting patches. A full list of contributors including their patches can be found at:

https://github.com/axsh/wakame-vdc/contributors

Supporters

• Servers for Continuous Integration
  • NHN Techorus Corp.

License

Copyright (c) Axsh Co. Components are included distribution under LGPL 3.0 and Apache 2.0

wakame-vdc's Issues

The curl command is considered a success even if it fails.

Issue files:

$ fgrep 'curl' -r dcmgr/lib
dcmgr/lib/dcmgr/drivers/esxi_local_store.rb:        sh("curl -s -o - #{inst_img_flat} | curl -s -u #{esxi_options[:user]}:#{esxi_options[:password]} -k -T - https://#{esxi_options[:host]}/folder/#{inst_uuid}/#{inst_img_flat.split("/").last}?dsName=#{esxi_options[:datastore]}")
dcmgr/lib/dcmgr/drivers/esxi_local_store.rb:        sh("curl -s -o - #{inst_img_meta} | curl -s -u #{esxi_options[:user]}:#{esxi_options[:password]} -k -T - https://#{esxi_options[:host]}/folder/#{inst_uuid}/#{inst_uuid}.vmdk?dsName=#{esxi_options[:datastore]}")
dcmgr/lib/dcmgr/drivers/esxi_local_store.rb:        sh("curl -s -u #{opts[:user]}:#{opts[:password]} -k -o /tmp/#{inst[:uuid]}.vmx https://#{opts[:host]}/folder/#{inst[:uuid]}/#{inst[:uuid]}.vmx?dsName=#{opts[:datastore]}")
dcmgr/lib/dcmgr/drivers/esxi_local_store.rb:        sh("curl -u #{opts[:user]}:#{opts[:password]} -k -s -T /tmp/#{inst[:uuid]}.vmx https://#{opts[:host]}/folder/#{inst[:uuid]}/#{inst[:uuid]}.vmx?dsName=#{opts[:datastore]}")
dcmgr/lib/dcmgr/drivers/ifs_iscsi.rb:        sh("curl -s http://#{@ip}:#{@web_ui_port}/iscsi/#{@vol_path}/#{@volume_id}?export=#{iqn}")
dcmgr/lib/dcmgr/drivers/ifs_iscsi.rb:        sh("curl -s http://#{@ip}:#{@web_ui_port}/iscsi?unexport=#{IQN_PREFIX}:#{@volume_id}")
dcmgr/lib/dcmgr/drivers/webdav.rb:      sh("curl -T %s %s", [src_path, abs_uri(dst_bo)])
dcmgr/lib/dcmgr/drivers/webdav.rb:      sh("curl -o %s %s", [dst_path, abs_uri(src_bo)])
dcmgr/lib/dcmgr/drivers/webdav.rb:      sh("curl -s -X DELETE %s", [abs_uri(bo)])
dcmgr/lib/dcmgr/drivers/ifs_storage.rb:      sh "curl -s #{@backup_storage[:base_uri]}/ifsutils/#{filename}?delete"
dcmgr/lib/dcmgr/drivers/esxi.rb:          sh("curl -s -u #{opts[:user]}:#{opts[:password]} -k -T #{ctx.inst_data_dir}/metadata.iso https://#{opts[:host]}/folder/#{ctx.inst[:uuid]}/metadata.iso?dsName=#{opts[:datastore]}")
dcmgr/lib/dcmgr/drivers/ifs.rb:        sh "curl -s http://#{@ip}:#{@port}/ifsutils/#{@vol_path}?mkdir"
dcmgr/lib/dcmgr/drivers/ifs.rb:          #sh "curl -X PUT -d @#{snap_file} http://#{@ip}:#{@port}/ifsutils/#{@fsid}/volumes/#{@volume_id}"
dcmgr/lib/dcmgr/drivers/ifs.rb:          sh "curl -s http://#{@ip}:#{@port}/ifsutils/#{snap_path}?duplicate=#{new_vol_path}/#{@volume_id}"
dcmgr/lib/dcmgr/drivers/ifs.rb:          sh "curl -s http://#{@ip}:#{@port}/ifsutils/#{@vol_path}/#{@volume_id}?allocate=#{@volume[:size]}"
dcmgr/lib/dcmgr/drivers/ifs.rb:        sh "curl -s http://#{@ip}:#{@port}/ifsutils/#{@vol_path}/#{@volume_id}?delete"
dcmgr/lib/dcmgr/drivers/ifs.rb:        sh "curl -s http://#{@ip}:#{@port}/ifsutils/#{@vol_path}/#{@volume[:uuid]}?duplicate=#{new_snap_path}"

Bad case:

http_code is 404. status_code is success(0).
http code 404 should be considered an error.

as follows:

$ curl -s -w "http_code=%{http_code}\n" -o  /vz/template/cache/bo-lucid7d http://localhost:8080/images/ubuntu-lucid-kvm-md-32.raw.gz
http_code=404

$ echo $?
0

Good case:

http_code is 404. status_code is fail(22).

I added curl option that -f(--fail).
--fail option mean that

$ man curl
22 HTTP  page  not  retrieved.  The  requested url was not found or returned another error with the HTTP error code being 400 or above. This return code only appears if
-f/--fail is used.

as follows:

$ curl -f -s -w "http_code=%{http_code}\n" -o  /vz/template/cache/bo-lucid7d http://localhost:8080/images/ubuntu-lucid-kvm-md-32.raw.gz
http_code=404

$ echo $?
22

After resolved the problems

I shouled run as follows.

Use case: download image file via webdav.

Failure Case:

$ file /vz/template/cache/bo-lucid7d
/vz/template/cache/bo-lucid7d: HTML document text

Success Case:

$ file /vz/template/cache/bo-lucid7d
/vz/template/cache/bo-lucid7d: gzip compressed data, was "ubuntu-lucid-kvm-md-32.raw", from Unix, last modified: Thu Jul 26 03:38:28 2012

Bug: IpLease::TYPE_RESERVED is undefined.

IpLease::TYPE_RESERVED is undefined.
https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/endpoints/11.12/networks.rb#L83
https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/models/network.rb#L99-L102

Currentry move to Dcmgr::Models::NetworkVifIpLease::TYPE_RESERVED.
we need to revert to IpLease::TYPE_RESERVED from NetworkVifIpLease::TYPE_RESERVED

rpmbuild install creates file which fills installation volume

Symptom: Downloaded image expands to fill entire volume when gunzipped
during install from instructions found at:

https://github.com/axsh/wakame-vdc/tree/master/rpmbuild

command issued:

/opt/axsh/wakame-vdc/tests/vdc.sh init

Below is output from my VM used for installation.
Note the lbnode-centos-openvz-64.tar.gz is 25GB.

[root@wakame images]# ls -al /opt/axsh/wakame-vdc/tmp/images
total 25099040
drwxr-xr-x. 2 root root        4096 Jun  4 15:00 .
drwxr-xr-x. 3 root root        4096 Jun  4 14:47 ..
-rw-r--r--. 1 root root   197103308 Jun  4 14:52 centos-6.3-openvz-md-x86_64-40G.raw.tar.gz
-rw-r--r--. 1 root root   230893964 Jun  4 14:56 lb-centos-openvz-md-64-stud.raw.gz
-rw-r--r--. 1 root root 25273397248 Jun  4 15:13 lbnode-centos-openvz-64.tar.gz

Below is the correct approximate size
$ gunzip -l lbnode-centos-openvz-64.tar.gz
         compressed        uncompressed  ratio uncompressed_name
          169807684           398735360  57.4% lbnode-centos-openvz-64.tar

Script for downloading demo images:

https://github.com/axsh/wakame-vdc/blob/master/tests/vdc.sh.d/demodata_images.sh

Parallel Curl Script:

https://github.com/axsh/wakame-vdc/blob/master/dcmgr/script/parallel-curl.sh

Output from scripted installation:
( Note echoed status from my install script logfile after demodata_images.sh finished unzipping until disk was 100% usage. Demodata_images script always returns 0 after gunzip "list" .)

Downloading image file lbnode-centos-openvz-64.tar.gz ...
content-length: 169807684 / 6
$ curl --retry 3 -s -L --range 0-28301279 -o /var/tmp/__parallel-curl.sh/lbnode-centos-openvz-64.tar.gz.0 http://dlc.wakame.axsh.jp.s3.amazonaws.com/demo/vmimage/lbnode-centos-openvz-64.tar.gz &
$ curl --retry 3 -s -L --range 28301280-56602559 -o /var/tmp/__parallel-curl.sh/lbnode-centos-openvz-64.tar.gz.1 http://dlc.wakame.axsh.jp.s3.amazonaws.com/demo/vmimage/lbnode-centos-openvz-64.tar.gz &
$ curl --retry 3 -s -L --range 56602560-84903839 -o /var/tmp/__parallel-curl.sh/lbnode-centos-openvz-64.tar.gz.2 http://dlc.wakame.axsh.jp.s3.amazonaws.com/demo/vmimage/lbnode-centos-openvz-64.tar.gz &
$ curl --retry 3 -s -L --range 84903840-113205119 -o /var/tmp/__parallel-curl.sh/lbnode-centos-openvz-64.tar.gz.3 http://dlc.wakame.axsh.jp.s3.amazonaws.com/demo/vmimage/lbnode-centos-openvz-64.tar.gz &
$ curl --retry 3 -s -L --range 113205120-141506399 -o /var/tmp/__parallel-curl.sh/lbnode-centos-openvz-64.tar.gz.4 http://dlc.wakame.axsh.jp.s3.amazonaws.com/demo/vmimage/lbnode-centos-openvz-64.tar.gz &
$ curl --retry 3 -s -L --range 141506400- -o /var/tmp/__parallel-curl.sh/lbnode-centos-openvz-64.tar.gz.5 http://dlc.wakame.axsh.jp.s3.amazonaws.com/demo/vmimage/lbnode-centos-openvz-64.tar.gz &
wait: 30377 30378 30379 30380 30381 30382
concat parts...
=> lbnode-centos-openvz-64.tar.gz
-rw-r--r--. 1 root root 169807684 Jun  4 15:00 lbnode-centos-openvz-64.tar.gz
gunzip lbnode-centos-openvz-64.tar.gz with keeping sparse area ...


FINISHED vdc.sh init

Failure a backup of instance

The tee command front "| " -- there are more than one

2012-10-04 16:01:28 JobContext thr=LocalStore[1/2] [ERROR]: Caught RuntimeError: Failed to run archive command line: ["cp -p --sparse=always %s /dev/stdout | pv -W -f -n -s %s | | tee >(md5sum > '%s') >(wc -c > '%s')> %s", ["/var/lib/wakame-vdc/tmp/instances/i-mz82vkns/i-mz82vkns", 657457152, "/var/lib/wakame-vdc/tmp/instances/i-mz82vkns/md5", "/var/lib/wakame-vdc/tmp/instances/i-mz82vkns/size", "/var/lib/wakame-vdc/tmp/instances/tmp/i-mz82vkns20121004-2529-zvebtu"]]
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/linux_local_store.rb:146:in block in upload_image' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/linux_local_store.rb:315:incall'
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/linux_local_store.rb:315:in archive_from_snapshot' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/linux_local_store.rb:125:inupload_image'
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/openvz_local_store.rb:13:in block in upload_image' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/helpers/cgroup.rb:67:incall'
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/helpers/cgroup.rb:67:in cgroup_context' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/openvz_local_store.rb:12:inupload_image'

Clean up the network schedulers

There are several network schedulers that rely on many different versions of the "vifs" parameter. We should clean those up to following 1 single standard.

Also the instance#add_nic and network_vif#attach_to_network methods are bad practice that induce circular dependencies. We should replace those by schedulers.

Something like the following should be able to schedule an instance's vifs entirely.

svc_type = Dcmgr::Scheduler.service_type(instance)

svc_type.network.schedule(instance)
instance.network_vif.each { |vif|
   svc_type.mac_address.schedule(vif)
   svc_type.ip_address.schedule(vif)
}

Container ID is checked when generating an openvz container.

When generating an openvz container, and the container of specified container ID is ending with generation, it is considered as an error.

Change the state from start to initializing.

Fix code is:
https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/rpc/hva_handler.rb#L295

Reference issue is: #101

Returns wrong available counts for IPs

19c510a breaks return value of the method available_ip_num(). The original code should work same as before. it needs to be reverted.

refactor: define create method as a singleton method in plugin/network.rb

Load balancer security groups

Summary

Load balancers should have two security groups but only have one.

Expected behaviour

The load balancers should have two security groups. One that takes care of firewall rules, and another without rules that allows ARP and IP traffic between the load balancer and its instances.

             LB - Instance group without rules
        +-----------------------------------------+
        |                                         |
        |                                         |
        |    LB firewall rules                    |
     +--|---------------------+                   |
     |  |  +----------------+ |    +----------+   |
     |  |  |                | |    |          |   |
     |  |  | Load balancer  | |    | Instance |   |
     |  |  |                | |    |          |   |
     |  |  +----------------+ |    +----------+   |
     +--|---------------------+                   |
        |                                         |
        +-----------------------------------------+

Current faulty behaviour

Currently the load balancers have only one group that handles both firewall rules and ARP/IP traffic between LB and instances. This causes all of the LB firewall rules to be applied to instances as well.

   1 group with rules and Inst-LB connection
 +-----------------------------------------+
 |                                         |
 |                                         |
 |  +----------------+      +----------+   |
 |  |                |      |          |   |
 |  | Load balancer  |      | Instance |   |
 |  |                |      |          |   |
 |  +----------------+      +----------+   |
 |                                         |
 |                                         |
 +-----------------------------------------+

Solution

Edit This line and other relevant parts of the LB api to create and maintain two groups. https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/endpoints/12.03/load_balancers.rb#L112

Estimated time to fix

2 days if we don't write tests for it

Change spec of running instance

Feature

Allow memory size or amount of cpu cores of running instances to be changed.

Reason

Some times you can have an instance up and later realize that it needs more capabilities. It can also come in handy for trying to figure out what specs you need in the first place.

Currently available functionality

You could create a backup of your instance which will save as a new machine image. You could then start this up with a new stronger spec.

Problems with this approach:

• Network settings are not retained
• It's a pain to do and there should be an easier way

New functionality needed

• An API call to handle the new spec.
• A GUI for setting new specs.
• HVA functionality that reboots the instance with the new spec.

API suggestion

PUT /instances/instance-id

Add the following two arguments to it

• cpu_cores
• memory_size

Do the following:

• Check if the host node is capable of handling the new spec.
• Update the instance in the database
• Send an event to HVA to reboot the instance with the new spec.

Request:

• PUT /instance/i-xxxxxxxx?cpu_cores=4&memory_size=1024

Response:

• instance hash

GUI suggestion

• Add memory_size and cpu_cores to edit instance form.
• It should not show specs that are more than the host node can handle.

Failure the character encoding at registration of key-pair.

Failure a encoding of "description".

as follows.:

あく.....

demodata_images.sh fatal crashes and fails to download certain images

Script: tests/vdc.sh.d/demodata_images.sh

Failed to download images #6, #9, & #10 at line 50 and Fatal Crashes at Line 78 when Unable to "cat ~.md5" file

Below are the pertinent portions of my installation logging:

(Now preparing to parallel curl ... line installed prior to line 51 for my logging)

Image #1 Download

image-cassandra.meta, cassandra.x86_64.openvz.md.raw.tar.gz ...
Downloading image file cassandra.x86_64.openvz.md.raw.tar.gz ...

Now preparing to parallel curl of cassandra.x86_64.openvz.md.raw.tar.gz

Image #2 Download

image-centos1d.meta, centos-6.3-openvz-md-x86_64-40G.raw.tar.gz ...
Downloading image file centos-6.3-openvz-md-x86_64-40G.raw.tar.gz ...

Now preparing to parallel curl of centos-6.3-openvz-md-x86_64-40G.raw.tar.gz

Image #3 Download

image-lb.meta, lb-centos-openvz-md-64-stud.raw.gz ...
Downloading image file lb-centos-openvz-md-64-stud.raw.gz ...

Now preparing to parallel curl of lb-centos-openvz-md-64-stud.raw.gz

Image #4 Download

image-lbnode.meta, lbnode-centos-openvz-64.tar.gz ...
Downloading image file lbnode-centos-openvz-64.tar.gz ...

Now preparing to parallel curl of lbnode-centos-openvz-64.tar.gz

Image #5 Download

image-lucid0.meta, ubuntu-lucid-kvm-32.raw ...
Downloading image file ubuntu-lucid-kvm-32.raw ...

Now preparing to parallel curl of ubuntu-lucid-kvm-32.raw

Image #6 Download - ************** IMAGE DOWNLOAD NOT ATTEMPTED

image-lucid1.meta, ubuntu-lucid-kvm-32.raw ...
Downloading image file ubuntu-lucid-kvm-32.raw ...

Image #7 Download

image-lucid2.meta, ubuntu-lucid-kvm-32.raw.gz ...
image-lucid5d.meta, ubuntu-lucid-kvm-md-32.raw ...
Downloading image file ubuntu-lucid-kvm-md-32.raw ...

Now preparing to parallel curl of ubuntu-lucid-kvm-md-32.raw

Image #8 Download

image-lucid5.meta, ubuntu-lucid-kvm-ms-32.raw ...
Downloading image file ubuntu-lucid-kvm-ms-32.raw ...

Now preparing to parallel curl of ubuntu-lucid-kvm-ms-32.raw

Image #9 Download - ************** IMAGE DOWNLOAD NOT ATTEMPTED

image-lucid6d.meta, ubuntu-lucid-kvm-md-32.raw ...
Downloading image file ubuntu-lucid-kvm-md-32.raw ...

Image #10 Download - ************** IMAGE DOWNLOAD NOT ATTEMPTED

image-lucid6.meta, ubuntu-lucid-kvm-ms-32.raw ...
Downloading image file ubuntu-lucid-kvm-ms-32.raw ...

Image #11 Download

image-lucid7d.meta, ubuntu-lucid-kvm-md-32.raw.gz ...
image-lucid7.meta, ubuntu-lucid-kvm-ms-32.raw.gz ...
image-secgtest.meta, ubuntu-lucid-32-secgtest.raw ...
Downloading image file ubuntu-lucid-32-secgtest.raw ...

Now preparing to parallel curl of ubuntu-lucid-32-secgtest.raw

After downloading has finished

calculating checksum of /opt/axsh/wakame-vdc/tmp/images/cassandra.x86_64.openvz.md.raw.tar.gz ...

calculating checksum of /opt/axsh/wakame-vdc/tmp/images/centos-6.3-openvz-md-x86_64-40G.raw.tar.gz ...

calculating checksum of /opt/axsh/wakame-vdc/tmp/images/lb-centos-openvz-md-64-stud.raw.gz ...

calculating checksum of /opt/axsh/wakame-vdc/tmp/images/lbnode-centos-openvz-64.tar.gz ...

demodata_images.sh script crashes at Line #78 with messages:
(Checking local ... line inserted prior to line 78 for my logging)

Checking local checksum with "cat"

cat: /opt/axsh/wakame-vdc/tmp/images/ubuntu-lucid-kvm-32.raw.md5: No such file or directory

Here are the images that downloaded

total 1500396
drwxr-xr-x. 2 root root      4096 Jun 13 13:58 .
drwxr-xr-x. 4 root root      4096 Jun 13 13:41 ..
-rw-r--r--  1 root root 319161822 Jun 13 13:43 cassandra.x86_64.openvz.md.raw.tar.gz
-rw-r--r--  1 root root        33 Jun 13 13:58 cassandra.x86_64.openvz.md.raw.tar.gz.md5
-rw-r--r--  1 root root 197103308 Jun 13 13:45 centos-6.3-openvz-md-x86_64-40G.raw.tar.gz
-rw-r--r--  1 root root        33 Jun 13 13:58 centos-6.3-openvz-md-x86_64-40G.raw.tar.gz.md5
-rw-r--r--  1 root root 230893964 Jun 13 13:51 lb-centos-openvz-md-64-stud.raw.gz
-rw-r--r--  1 root root        33 Jun 13 13:58 lb-centos-openvz-md-64-stud.raw.gz.md5
-rw-r--r--  1 root root 169807684 Jun 13 13:53 lbnode-centos-openvz-64.tar.gz
-rw-r--r--  1 root root        33 Jun 13 13:58 lbnode-centos-openvz-64.tar.gz.md5
-rw-r--r--  1 root root 161422293 Jun 13 13:58 ubuntu-lucid-32-secgtest.raw.gz
-rw-r--r--  1 root root 152659458 Jun 13 13:54 ubuntu-lucid-kvm-32.raw.gz
-rw-r--r--  1 root root 152659010 Jun 13 13:55 ubuntu-lucid-kvm-md-32.raw.gz
-rw-r--r--  1 root root 152658585 Jun 13 13:56 ubuntu-lucid-kvm-ms-32.raw.gz

Another test:
total 1500396
drwxr-xr-x. 2 root root      4096 Jun 13 19:33 .
drwxr-xr-x. 4 root root      4096 Jun 13 19:04 ..
-rw-r--r--  1 root root 319161822 Jun 13 19:09 cassandra.x86_64.openvz.md.raw.tar.gz
-rw-r--r--  1 root root        33 Jun 13 19:33 cassandra.x86_64.openvz.md.raw.tar.gz.md5
-rw-r--r--  1 root root 197103308 Jun 13 19:13 centos-6.3-openvz-md-x86_64-40G.raw.tar.gz
-rw-r--r--  1 root root        33 Jun 13 19:33 centos-6.3-openvz-md-x86_64-40G.raw.tar.gz.md5
-rw-r--r--  1 root root 230893964 Jun 13 19:17 lb-centos-openvz-md-64-stud.raw.gz
-rw-r--r--  1 root root        33 Jun 13 19:33 lb-centos-openvz-md-64-stud.raw.gz.md5
-rw-r--r--  1 root root 169807684 Jun 13 19:20 lbnode-centos-openvz-64.tar.gz
-rw-r--r--  1 root root        33 Jun 13 19:33 lbnode-centos-openvz-64.tar.gz.md5
-rw-r--r--  1 root root 161422293 Jun 13 19:33 ubuntu-lucid-32-secgtest.raw.gz
-rw-r--r--  1 root root 152659458 Jun 13 19:23 ubuntu-lucid-kvm-32.raw.gz
-rw-r--r--  1 root root 152659010 Jun 13 19:26 ubuntu-lucid-kvm-md-32.raw.gz
-rw-r--r--  1 root root 152658585 Jun 13 19:30 ubuntu-lucid-kvm-ms-32.raw.gz

refactor wakame-init

the related issue #184

Should rename the instance state filter parameter from "without_terminated" to "stable_states"

86bf3a9 introduced the new state filter value "without_terminated" but it does not make sense.

the state strings listed in filter function can be said the stable states of instance from the SQL condition.

    def_dataset_method(:without_terminated) do
      filter("state='running' OR state='stopped' OR state='halted'")
    end

Those states should be grouped as a constant and the model class should refer it.

aliasing singleton method

The problem of load balancer API "index"

It appears, when a filter option is specified as load balancer API "index."

./mussel.sh load_balancer index --state=running

---

:error: Dcmgr::Endpoints::Errors::DatabaseError
:message: ! 'Sequel::DatabaseError: Mysql2::Error: Column ''account_id'' in where
clause is ambiguous'
:code: '110'

Change openvz template based on service type

The following pull request made a couple of changes that are specifically necessary for load balancers. Regular instances might not need these.

#129

We should allow openvz configurations to differ based on service type.

GUI: an assignment state of security-groups is not refreshed in instance dialog.

An instance dialog is opened after pushing an "edit" button which is put beside instance list. The control of security-groups shows a state of current assignments. However the state seems to be kept whenever new dialog pops.

After some other menu which is put on left-side is clicked, you can open the instance dialog again and can see that the state was renewed.

Fails in instance generation within hva. (Error exec /etc/vz/conf/7260.mount: Text file busy)

I, [2012-11-27T03:32:12.020413 #56102] INFO -- HvaHandler: Session ID: 5d7b5a82b5127453c3937d7b36f125cb75c1a291: Instance UUID: i-i8kbnkw0: Created config /etc/vz/conf/7260.mount
I, [2012-11-27T03:32:12.020598 #56102] INFO -- HvaHandler: Session ID: 5d7b5a82b5127453c3937d7b36f125cb75c1a291: Instance UUID: i-i8kbnkw0: Executing command: vzctl start i-i8kbnkw0
D, [2012-11-27T03:32:12.029859 #56102] DEBUG -- HvaHandler: Session ID: 5d7b5a82b5127453c3937d7b36f125cb75c1a291: Instance UUID: i-i8kbnkw0: Command Result: fail (exit code=79)
Command PID: 454514

STDOUT=>

Starting container ...

STDERR=>

Warning: distribution not specified in CT config, using defaults from /etc/vz/dists/default
Error exec /etc/vz/conf/7260.mount: Text file busy
Error executing mount script /etc/vz/conf/7260.mount
2012-11-27 03:32:12 JobContext thr=JobWorker[0/1] [ERROR]: Job failed 5d7b5a82b5127453c3937d7b36f125cb75c1a291 [ run_local_store ]: Unexpected exit code=79: vzctl start i-i8kbnkw0
Command PID: 454514

STDOUT=>

Starting container ...

STDERR=>

Warning: distribution not specified in CT config, using defaults from /etc/vz/dists/default
Error exec /etc/vz/conf/7260.mount: Text file busy
Error executing mount script /etc/vz/conf/7260.mount
2012-11-27 03:32:12 JobContext thr=JobWorker[0/1] [ERROR]: Caught Dcmgr::Helpers::CliHelper::ShellRunner::CommandError: Unexpected exit code=79: vzctl start i-i8kbnkw0
Command PID: 454514

STDOUT=>

Starting container ...

STDERR=>

Warning: distribution not specified in CT config, using defaults from /etc/vz/dists/default
Error exec /etc/vz/conf/7260.mount: Text file busy
Error executing mount script /etc/vz/conf/7260.mount
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:180:in block in run!' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:178:intap'
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:178:in run!' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:95:insh'
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/drivers/openvz.rb:272:in run_instance' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/task.rb:151:ininvoke!'
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/task.rb:214:in invoke' /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/rpc/hva_handler.rb:265:inblock in class:HvaHandler'
/opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/1.9.1/gems/isono-0.2.17/lib/isono/runner/rpc_server.rb:69:in `instance_eval'

Rewrite change of ddb9e7e90fac29646818d43a9784b60b9bceded5

The change ddb9e7e is incomplete fix that does not support every cases. Need to get someone to rewrite.

Wrong variable reference in API

10.0.2.2 - - [13/Jun/2014 06:02:23] "GET /api/12.03/instances/i-5gh4x2pt.yml HTTP/1.1" 200 808 0.0320
NameError - undefined local variable or method `v' for #<Dcmgr::Endpoints::V1203::CoreAPI:0x007f662af4fa00>:
/opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/endpoints/12.03/instances.rb:636:in `block (3 levels) in load_namespace'
/opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:181:in `call'
/opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:181:in `block in after_transaction_commit'
/opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:181:in `each'
/opt/axsh/wakame-vdc/dcmgr
/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:181:in `after_transaction_commit'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:293:in `remove_transaction'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:150:in `ensure in _transaction'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:150:in `_transaction'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:102:in `block in transaction'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/connecting.rb:236:in `block in synchronize'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/connecting.rb:236:in `synchronize'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sequel-3.47.0/lib/sequel/database/transactions.rb:95:in `transaction'
    /opt/axsh/wakame-vdc/dcmgr/lib/sinatra/sequel_transaction.rb:55:in `route_eval'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `catch'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:948:in `block in route!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `block in call!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-cors-0.2.9/lib/rack/cors.rb:54:in `call'
    /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/rack/request_logger.rb:28:in `_call'
    /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/rack/request_logger.rb:20:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/lint.rb:49:in `_call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/lint.rb:37:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/showexceptions.rb:24:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/commonlogger.rb:33:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.3/lib/sinatra/base.rb:212:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/chunked.rb:43:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/rack-1.5.2/lib/rack/content_length.rb:14:in `call'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:580:in `process_client'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/oob_gc.rb:60:in `process_client'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:660:in `worker_loop'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:527:in `spawn_missing_workers'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:153:in `start'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>'
    /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'

Re-organize Image data model and Image file generation process

1. Add hypervisor field to images table.
2. Need to aware of OS distribution

LoadBalancer Image

• CentOS 6.3
• Stud, AMQP tool

Test Image

• CentOS6.3, Ubuntu 12.04
  • Hypervisor: KVM, ESXi, LXC, OpenVZ
• Windows Server
  • Hypervisor: KVM, ESXi

Auto build folder structure

• vmapp/

The instance power-on does not work with TGZ image.

@Metallion reported following issue. He run power-on command to OpenVZ instance with tgz image file:

I reviewed but whenever I poweron a halted instance, it gets stuck in "starting" and I see the following error in hva.

Private area already exists in /vz/private/101
Creation of container private area failed
/root/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:180:in block in run!'
/root/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:178:intap'
/root/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:178:in run!'
/root/wakame-vdc/dcmgr/lib/dcmgr/helpers/cli_helper.rb:95:insh'
/root/wakame-vdc/dcmgr/lib/dcmgr/drivers/openvz.rb:140:in run_instance'
/root/wakame-vdc/dcmgr/lib/dcmgr/drivers/openvz.rb:328:inpoweron_instance'

refactor: put comments about inherited method in core/base.rb

"null" hypervisor

in order to call api for test, "null" hypervisor type is one of the solution.

Load balancer spec selection

Currently it is not possible to select a load balancer spec in the GUI. The GUI looks for a load balancer with 1000 max connections. This is hard coded and it will fail if there is no spec for 1000 connections defined in load_balancer_spec.yml

My idea is to display a selection of load balancer spec names. Imagine the following load_balancer_spec.yml file.

---
lb.small:
  description: "haproxy with 1000 mac connections"
  cpu_cores: 2
  memory_size: 1024
  engine: "haproxy"
  max_connection: 1000

lb.large:
  description: "haproxy with 5000 max connections"
  engine: "haproxy"
  max_connection: 5000

The GUI should have a drop down list showing "lb.small" and "lb.large". When one of them is selected, the description should be shown to the user.

Fails backup for local store instance.

• Faiure from test_expect_failure_at_instance_terminate
  Due to the response message change from PUT /instances/i-xxxx/backup, the acceptance test scenario fails to fetch expected backup object ID from the response.
• Failure from test_multiple_backup_tasks_without_confliction

23:53:47 test_multiple_backup_tasks_without_confliction
23:53:51 [2014-06-19T23時53分51秒 #6849] time:1 eval:document_pair? backup_object bo-obk8npkq state available
23:53:54 [2014-06-19T23時53分54秒 #6849] time:2 eval:document_pair? backup_object bo-obk8npkq state available
23:53:57 [2014-06-19T23時53分57秒 #6849] time:3 eval:document_pair? backup_object bo-obk8npkq state available
23:54:00 [2014-06-19T23時54分00秒 #6849] time:4 eval:document_pair? backup_object bo-obk8npkq state available
23:54:03 [2014-06-19T23時54分03秒 #6849] time:5 eval:document_pair? backup_object bo-obk8npkq state available
23:54:06 [2014-06-19T23時54分06秒 #6849] time:6 eval:document_pair? backup_object bo-obk8npkq state available
23:54:09 [2014-06-19T23時54分09秒 #6849] time:7 eval:document_pair? backup_object bo-obk8npkq state available
23:54:12 [2014-06-19T23時54分12秒 #6849] time:8 eval:document_pair? backup_object bo-obk8npkq state available
23:54:15 [2014-06-19T23時54分15秒 #6849] time:9 eval:document_pair? backup_object bo-obk8npkq state available
23:54:18 [2014-06-19T23時54分18秒 #6849] time:10 eval:document_pair? backup_object bo-obk8npkq state available
23:54:21 [2014-06-19T23時54分21秒 #6849] time:11 eval:document_pair? backup_object bo-obk8npkq state available
23:54:24 [2014-06-19T23時54分24秒 #6849] time:12 eval:document_pair? backup_object bo-obk8npkq state available
23:54:27 [2014-06-19T23時54分27秒 #6849] time:13 eval:document_pair? backup_object bo-obk8npkq state available
23:54:30 [2014-06-19T23時54分30秒 #6849] time:14 eval:document_pair? backup_object bo-obk8npkq state available
23:54:33 [2014-06-19T23時54分33秒 #6849] time:15 eval:document_pair? backup_object bo-obk8npkq state available
23:54:36 [2014-06-19T23時54分36秒 #6849] time:16 eval:document_pair? backup_object bo-obk8npkq state available
23:54:39 [2014-06-19T23時54分39秒 #6849] time:17 eval:document_pair? backup_object bo-obk8npkq state available
23:54:42 [2014-06-19T23時54分42秒 #6849] time:18 eval:document_pair? backup_object bo-obk8npkq state available
23:54:45 [2014-06-19T23時54分45秒 #6849] time:19 eval:document_pair? backup_object bo-obk8npkq state available
23:54:48 [2014-06-19T23時54分48秒 #6849] time:20 eval:document_pair? backup_object bo-obk8npkq state available
23:54:51 [2014-06-19T23時54分52秒 #6849] time:21 eval:document_pair? backup_object bo-obk8npkq state available
23:54:55 [2014-06-19T23時54分55秒 #6849] time:22 eval:document_pair? backup_object bo-obk8npkq state available
23:54:58 [2014-06-19T23時54分58秒 #6849] time:23 eval:document_pair? backup_object bo-obk8npkq state available
23:55:01 [2014-06-19T23時55分01秒 #6849] time:24 eval:document_pair? backup_object bo-obk8npkq state available
23:55:04 [2014-06-19T23時55分04秒 #6849] time:25 eval:document_pair? backup_object bo-obk8npkq state available
23:55:07 [2014-06-19T23時55分07秒 #6849] time:26 eval:document_pair? backup_object bo-obk8npkq state available
23:55:10 [2014-06-19T23時55分10秒 #6849] time:27 eval:document_pair? backup_object bo-obk8npkq state available
23:55:13 [2014-06-19T23時55分13秒 #6849] time:28 eval:document_pair? backup_object bo-obk8npkq state available
23:55:16 [2014-06-19T23時55分16秒 #6849] time:29 eval:document_pair? backup_object bo-obk8npkq state available
23:55:19 [2014-06-19T23時55分19秒 #6849] time:30 eval:document_pair? backup_object bo-obk8npkq state available
23:55:22 [2014-06-19T23時55分22秒 #6849] time:31 eval:document_pair? backup_object bo-obk8npkq state available
23:55:25 [2014-06-19T23時55分25秒 #6849] time:32 eval:document_pair? backup_object bo-obk8npkq state available
23:55:28 [2014-06-19T23時55分28秒 #6849] time:33 eval:document_pair? backup_object bo-obk8npkq state available
23:55:31 [2014-06-19T23時55分31秒 #6849] time:34 eval:document_pair? backup_object bo-obk8npkq state available
23:55:34 [2014-06-19T23時55分34秒 #6849] time:35 eval:document_pair? backup_object bo-obk8npkq state available
23:55:37 [2014-06-19T23時55分37秒 #6849] time:36 eval:document_pair? backup_object bo-obk8npkq state available
23:55:40 [2014-06-19T23時55分41秒 #6849] time:37 eval:document_pair? backup_object bo-obk8npkq state available
23:55:44 [2014-06-19T23時55分44秒 #6849] time:38 eval:document_pair? backup_object bo-obk8npkq state available
23:55:47 [2014-06-19T23時55分47秒 #6849] time:39 eval:document_pair? backup_object bo-obk8npkq state available
23:55:50 [2014-06-19T23時55分50秒 #6849] time:40 eval:document_pair? backup_object bo-obk8npkq state available
23:55:53 [2014-06-19T23時55分53秒 #6849] time:41 eval:document_pair? backup_object bo-obk8npkq state available
23:55:56 [2014-06-19T23時55分56秒 #6849] time:42 eval:document_pair? backup_object bo-obk8npkq state available
23:55:59 [2014-06-19T23時55分59秒 #6849] time:43 eval:document_pair? backup_object bo-obk8npkq state available
23:56:02 [2014-06-19T23時56分02秒 #6849] time:44 eval:document_pair? backup_object bo-obk8npkq state available
23:56:05 [2014-06-19T23時56分05秒 #6849] time:45 eval:document_pair? backup_object bo-obk8npkq state available
23:56:08 [2014-06-19T23時56分08秒 #6849] time:46 eval:document_pair? backup_object bo-obk8npkq state available
23:56:11 [2014-06-19T23時56分11秒 #6849] time:47 eval:document_pair? backup_object bo-obk8npkq state available
23:56:14 [2014-06-19T23時56分14秒 #6849] time:48 eval:document_pair? backup_object bo-obk8npkq state available
23:56:17 [2014-06-19T23時56分17秒 #6849] time:49 eval:document_pair? backup_object bo-obk8npkq state available
23:56:20 [2014-06-19T23時56分20秒 #6849] time:50 eval:document_pair? backup_object bo-obk8npkq state available
23:56:23 [2014-06-19T23時56分23秒 #6849] time:51 eval:document_pair? backup_object bo-obk8npkq state available
23:56:26 [2014-06-19T23時56分26秒 #6849] time:52 eval:document_pair? backup_object bo-obk8npkq state available
23:56:29 [2014-06-19T23時56分29秒 #6849] time:53 eval:document_pair? backup_object bo-obk8npkq state available
23:56:32 [2014-06-19T23時56分32秒 #6849] time:54 eval:document_pair? backup_object bo-obk8npkq state available
23:56:35 [2014-06-19T23時56分35秒 #6849] time:55 eval:document_pair? backup_object bo-obk8npkq state available
23:56:38 [2014-06-19T23時56分38秒 #6849] time:56 eval:document_pair? backup_object bo-obk8npkq state available
23:56:42 [2014-06-19T23時56分42秒 #6849] time:57 eval:document_pair? backup_object bo-obk8npkq state available
23:56:45 [2014-06-19T23時56分45秒 #6849] time:58 eval:document_pair? backup_object bo-obk8npkq state available
23:56:48 [2014-06-19T23時56分48秒 #6849] time:59 eval:document_pair? backup_object bo-obk8npkq state available
23:56:51 Retry Failure: Exceed 180 sec: Retried 60 times
23:56:51 ASSERT:expected:<0> but was:<1>
23:56:51 curl: (22) The requested URL returned error: 500 Internal Server Error
23:56:51 ASSERT:expected:<0> but was:<22>
23:56:51 ---
23:56:51 - bo-vexs2p6b
23:56:51 ---
23:56:51 - i-ajd81hem
23:56:54 [2014-06-19T23時56分54秒 #6849] time:1 eval:document_pair? instance i-ajd81hem state terminated
23:56:54 ---
23:56:54 - ssh-q6fraolj
23:56:54 ---
23:56:54 - sg-9e160cbn

Feature IP address assignment retention period

This change aims to improve IP address assignment strategy to avoid to re-use same IP addresses as long as possible.

Assume 192.168.1.0/24 network and .2-254 is given as dynamic range. Then the user will start three instances in the network but second one will be terminated before the third one starts.

Step1.
i-00001 

Step2.
i-00001
i-00002

Step3.
i-00001
i-00002 -> terminate

Step4.
i-00001
i-00003

Following describes how the address is provisioned at this moment:

1.
i-00001: 192.168.1.2

2.
i-00001: 192.168.1.2
i-00002: 192.168.1.3

3.
i-00001: 192.168.1.2

4.
i-00001: 192.168.1.2
i-00003: *192.168.1.3

The descending order assignment is expected normally as below:

1.
i-00001: 192.168.1.2

2.
i-00001: 192.168.1.2
i-00002: 192.168.1.3

3.
i-00001: 192.168.1.2

4.
i-00001: 192.168.1.2
i-00003: *192.168.1.4

** Changes

1. Add retention period field to networks table.
2. Update Models::IpLease#iplease or around where provision IP address

Bug when applying 2 identical netfilter rules

Example

Given there is a security group A with this rule:
tcp:80,80,ip4:0.0.0.0

And there is a security group B that also has this rule:
tcp:80,80,ip4:0.0.0.0

When an instance is started in both group A and group B at the same time, the rule will only be applied once in iptables.

When group A deleted the rule but group B doesn't, the rule will be deleted entirely from iptables.

Cause

In the netfilter task manager, there are several points where Array#uniq is used to delete indentical rules, like so: https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/vnet/netfilter/task_manager.rb#L586

Solution

Remove those and allow identical rules to be applied multiple times. That way one can be removed while the other remains.

Estimated time needed to fix

2 days - 5 days

Quota evaluation missing for request amount.

-> #147

The problem of load balancer API "index"

It appears, when a filter option is specified as load balancer API "index."

./mussel.sh load_balancer index --state=running

---

:error: Dcmgr::Endpoints::Errors::DatabaseError
:message: ! 'Sequel::DatabaseError: Mysql2::Error: Column ''account_id'' in where
clause is ambiguous'
:code: '110'

js-quantities does not support both IE7 and IE8.

Have a problem with the following screen.

• /storage_nodes
• /backups

issue code:

https://github.com/axsh/wakame-vdc/blob/master/frontend/dcmgr_gui/public/javascripts/dcmgr_gui/core.js#L87

q = new Qty(qty + unit);

Popup error on IE:

Object doesn't support this property or method

Device mapping bug when run instance.

Device mapper remain that if an error occur between creating device mapper in kpartx and before mounting device.

It occurs in two placess:

• https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/drivers/openvz.rb#L151-L176
• https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/drivers/openvz.rb#L239-L247

Cause

It need to call detach_loop() when deleting device mapper.

Solution

 def mapping_device(blk)
   begin
     blk.call
   rescue Exception => e
     detach_loop()
   end
 end

  mapping_device do
    sh("kpartx -av %s", [hs.os_devpath])
    # some processing
  end
  sh("mount %s %s", [root_device[0], private_folder])

Appendix

if an error occurs after mounting device, it's no problem.
because calling detach loop() while terminating instance.

https://github.com/axsh/wakame-vdc/blob/master/dcmgr/lib/dcmgr/drivers/openvz.rb#L308-L315

Isono::Util.default_gw_ip dependency

Isono::Util.default_gw_ip causes following issues.

undefined method `split' for nil:NilClass
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/1.9.1/gems/isono-0.2.18/lib/isono/util.rb:44:in `default_gw_ipaddr'

network condigion:

• no default gw

Target files including default-gw_ipaddr

$ git grep default_gw_ipaddr
dcmgr/bin/hva:  node_instance_id "#{Isono::Util.default_gw_ipaddr}"
dcmgr/bin/nsa:  node_instance_id "#{Isono::Util.default_gw_ipaddr}"
dcmgr/bin/sta:  node_instance_id "#{Isono::Util.default_gw_ipaddr}"
dcmgr/config/initializers/isono.rb:        node_instance_id "#{Isono::Util.default_gw_ipaddr}:#{Process.pid}"
dcmgr/lib/dcmgr/vnet/network_modes/security_group.rb:      host_addr = Isono::Util.default_gw_ipaddr

1. dcmgr/bin/* and dcmgr/config/initializers/isono.rb just need unique-id/string,
   • they don't need a kind of ip address.
   • solutions
     • using hostname instead of default_gw_ipaddr
     • etc.
2. dcmgr/lib/dcmgr/vnet/network_modes/security_group.rb
   • host node ip address is required.
   • solutions
     • defining host_addr in /etc/wakame-vdc/hva.conf.
     • etc.

Enable an edit button of "halted" load-balancer.

A edit button on load-balancer list.
It is disable now.

Clean up load balancer vif parameters

Problem

Currently the load balancer API returns a "vifs" parameter similar to the instances. The API should treat the load balancer as a service and not an instance. Also, just by watching the API right now, it is not clear which vif is for management and which is for contact with the instances.

Solution

A proposed cleaner way to write those parameters would be like this:

management_network: nw-xxx
management_ip: x.x.x.x

public_network: nw-yyy
public_ip: y.y.y.y

move plugin array to node_api

IPv4 spoofing bug

IPv4 spoofing is done only for arp packets at the moment. We have to do it for ipv4 too. I don't want to go dig in my old dirty netfilter code any more so I'll fix it in service netfilter v3 when it's merged.

Bug fix for the free resource capacity checking across entire local VDC domain.

Network data model refactoring

1. swap alloc_type field from network_vif_ip_leases to ip_leases.
2. add time period to dhcp_range table that suppresses to re-use IP address.
3. Make IP addresses in networks and network_services table associate from ip_leases.

refactor: way of defining method in core/base.rb

split wakame-vdc-dcmgr rpm package

split dcmgr sub package to at least following packages.

• bksta
• proxy

for each upstart system job configs.

GUI: SSH KEY Pair Create Dialog Box.

Improvement suggestion

The public key field is enabled in default. it should be is disabled because it is optional. if possible, field switch using checkbox field.

Issuing multiple start instance RPC calls to a shared volume instance.

hva receive duplicate RPC calls from sta sometime when the instance has extra volumes. At the event, the hva fails with log messages lbelow:

2014-06-27 13:23:02 JobContext thr=JobWorker[0/1] [INFO]: Job complete 1ac6cce110724b3d352426ab3bcef8ebca8ff742 (Local ID: 7cd0a97bcdaa693ae64989358c2de18c10bc22d1)[ run_vol_store ]: 5.607662015 sec
D, [2014-06-27T13:23:02.966852 #15967] DEBUG -- ServiceNetfilter: event caught: hva.dsv0003/vnic_created: vif-3fkx8ps4
2014-06-27 13:23:02 JobContext thr=JobWorker[0/1] [INFO]: Job start 1ac6cce110724b3d352426ab3bcef8ebca8ff742 (Local ID: f50d9df47b5c33a39f6a0d4ddd5e715d67a52be9)[ run_vol_store ]
I, [2014-06-27T13:23:03.589612 #15967]  INFO -- NetfilterCache: Adding vif-3fkx8ps4 to sg-adlbbcbd
I, [2014-06-27T13:23:04.112026 #15967]  INFO -- HvaHandler: Session ID: 1ac6cce110724b3d352426ab3bcef8ebca8ff742: Instance UUID: i-z3pfq4hk: Booting i-z3pfq4hk
2014-06-27 13:23:04 JobContext thr=JobWorker[0/1] [ERROR]: Job failed 1ac6cce110724b3d352426ab3bcef8ebca8ff742 [ run_vol_store ]: Invalid instance state: running
2014-06-27 13:23:04 JobContext thr=JobWorker[0/1] [ERROR]: Caught RuntimeError: Invalid instance state: running
        /opt/axsh/wakame-vdc/dcmgr/lib/dcmgr/rpc/hva_handler.rb:399:in `block in <class:HvaHandler>'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/runner/rpc_server.rb:69:in `instance_eval'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/runner/rpc_server.rb:69:in `block in job'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/rack/proc.rb:25:in `instance_eval'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/rack/proc.rb:25:in `call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/rack/map.rb:52:in `call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/rack/map.rb:52:in `call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/rack/builder.rb:36:in `call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/rack/job.rb:59:in `block (2 levels) in call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/node_modules/job_worker.rb:67:in `call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/node_modules/job_worker.rb:67:in `block in start'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/thread_pool.rb:32:in `call'
        /opt/axsh/wakame-vdc/dcmgr/vendor/bundle/ruby/2.0.0/gems/isono-0.2.19/lib/isono/thread_pool.rb:32:in `block (2 levels) in initialize'

GUI security groups bug when instance has no vifs

Bug

The edit instance dialog shows the wrong security groups assigned when an instance in started through the GUI with no vifs.

Reproduce steps

• Start an instance through the GUI in security group sg-demofgr without any vifs
• Go to the instances tab in the GUI and click on the new instance's edit button
• sg-demofgr will not show up as assigned.

Cause

Security groups are related to vifs in the database. Since the instance has no vifs, it was impossible to assign security groups to it when it started up.

Long term solution

• Implement GUI features to assign security groups to vifs instead of instances.

Short term solution

• On the launch instance dialog:
  • Move the vif creation panel above the security group selection panel.
  • Disable the security group selection panel until an instance has at least 1 vif.
  • Make it possible to start instances without security groups through the GUI.
• On the edit instance dialog:
  • Disable the security group selection panel if the instance has no vifs.

require plugins in a correct way

rpmbuild downloading images to wrong location

Images should be going into
/var/lib/wakame-vdc/images

Installation crashes when trying to download to:
/opt/axsh/wakame-vdc/tmp/image

Add terminate/delete reason to user resources.

DC resources affected:

• instance
• volume
• backup object
• vif
• network
• ssh key?
• security group?

Add fields to database tables:

Tables having "terminated_at" column:

add_column terminate_code, int
add_column terminate_reason, text

Tables having "deleted_at" column:

add_column delete_code, int
add_column delete_reason, text

TODOs:

• Standardize error codes.

A New Security Group Isolation Mode

This is the proposal for new isolation mode that allows communication between master IP/VIF and slave IPs/VIFs but rejects communications in slave IPs/VIFs.

The expected use case is that #170. The change still has a corner case that all the balanced nodes are allowed to commucate each other although the user does not add the allow rules.

This behavior can be controlled by netfilter layer using both iptables and ebtables.

As you may know, some of home use Wifi/Internet router devices support explained type of packet control. That helps to avoid unwanted local file share between connecting laptops or desktops. But the clients can access to the Internet. The feature is named Privacy Separator.