josin / cloud_gunther Goto Github PK

Instance_launcher and some other components write nice log lines as "Waiting #{WAIT_STEP} sec until instances is ready."
These should be visible to the user.

instance-service.rb, line 39.

Checking if cloud engine reports tasks after 1 minute is superfluous. Task may start after that time.

Task termination code dubious, doesn't take into account why instances terminated.
Needs feedback from instance_launcher and perhaps heartbeat from running instances.

• daemons should send heartbeat that they are running

• application dashboard would show status of all application components e.g.

  outputs worker status: OK
  scheduler status: OK
  instance server: not running

Currently, we are launching exactly as many instances as instances_count, which would be good for parallel computing, but if some don't start, nothing is done about it.

In version 2, we need a machine_count variable, which will be the number of launched instances, while instances_count will be the number of problem instances. This will enable pseudoparallel computations.
The scheduler could also periodically check if there are free resources, as to maximize the number of running instances.
(task.maximize attribute, or machine_count=0)

A true parallel mode will be added where instances_count = machine_count, system will wait until exactly that many instances are running (with the possibility of re-launching failed instances) before starting algorithm.
(task.parallel attribute, or default if machine_count>0)

Version 3 will add better scheduling methods and multi-user load balancing. It will track estimated queue length and offer transfer to other cloud engines if it is too long.

ParamsReader module might be replace with ActiveRecord::Base.store which is introduced in Rails 3.2.

1. Upgrade project to Rails 3.1
2. Upgrade project to Rails 3.2
3. Replace ParamsReader in all model classes to ActiveRecord::Base.store and check views for dependencies

1. If we use :addressing_type => "private", CG must be in the same network as the cloud. Then it won't be able to use multiple cloud engines, unless we use tunnels.
2. Why are we waiting for pending->running transition with exponential back-off and for OS boot with a constant time-out?
3. Why are we connecting with SSH 2 times to each instance?

All those would be solved if first connection went from the instance to CG.

There must be a timeout for that to happen. That will be tracked in scheduler's termination method.
It also must catch errors like direct pending->terminated transitions. (in short: system will track all launched instances)
This kind of errors will cause scheduler to re-launch failed instances. (beware of infinite loops, per-task re-launch quota?)

All instances will be launched at once and instance_service will handle initialization. (no need for threads in instance_launcher)
If the alg_runner reports errors launching algorithm, instance will not re-launch, whole task and all instances may be terminated.