bouthilx / cumulus Goto Github PK

Test with

• Hades (qstat)
• Helios (qstat/showq)
• Graham (squeue)

(Copy of bouthilx/research#49)

Hades seems to have a faulty GPU several times a week. It is very frustrating because it empties the list of queued job by crashing them all on the fautty GPU. If this kind of error could be detected, would could requeue the job rather than marking it as failed. The will still cause a lot of useless operations, because the jobs will keep being thrown to the faulty GPU until a correct one is free.

• Setup continuous integration app
  • circleCI
  • Travis
• Setup code quality app
  • codecov
  • codacy

Later on:

• Drop either CircleCI or Travis based on experience
• Maybe drop codecov or codacy if one is enough

Test with

• Hades (qdel)
• Helios (qdel but with qstat-showq)
• Graham (scancel)

Just as Cumulus is a composite of Clusters, there should be a Project which is a composite of experiments Database

(Related to bouthilx/research#19)

Provide a way for jobs to mark themselves as interrupted. Maybe provide the interruption handler itself, with support for callbacks. The internal machinery would then take care of marking the job as interrupted in the database. The command deploy should look for both queued and interrupted jobs, with an option to force a selection of either of them (cumulus deploy [some options] --restrict-to QUEUED). The database backend will take care of converting the status from QUEUED to whatever the backend should use. Standard status to convert from are defined in AbstractDatabase.

Use db[job_id] to filter monitor (qstat/squeue) and print jobs in the queue (R, Q, C, H) divided by experiments.

cumulus monitor --group-by [experiment|cluster]

• Test with benchmarks

• Hades
• Helios
• Guillimin
• Angel/Monk
• Parallel
• Gravity?

Needs to be tested thoroughly.

Test cases:

• One experiment has no queued jobs
• All experiments have no queued jobs
• One cluster has no free slots
• All clusters have no free slots
• One cluster is down
• What else?

Minimize the number of connections by avoided useless connections on deploy if there is no experiments to run.

(Copy of bouthilx/research#37)

Goals:

Minimize manipulation of code on clusters
Allow experiments to row on different code version simultaneously
Timeline:

Plan
Risks:

Duplicating sacred's tracking of code versioning
Add to much overhead in each experiments
Hard to debug installs

Needs

1. cumulus setup
   None

2. cumulus deploy
   status - To select jobs to submit

3. cumulus watch
   job_id - To combine with squeue maybe?
   status - To print
   cluster - To print
   node - To print

4. cumulus cancel
   status - To update status to CANCELLED
   job_id - To update status to CANCELLED

5. cumulus-particle local
   repos - Copy repositories
   job_id - Set job_id
   cluster - Set cluster
   node - Set node
   status - Set status (or leave it to executed command - like sacred)

Backends

1. Default
2. Sacred

Right now, the connections are done in pool.async_call so self.connected_hostname and self.sshtunnel only exists inside this child process. This means every operations on cluster in subsequent child processes will initiate a connection to the server too. Commands likes deploy and cancel are combinations of remote operations so they will make many useless server connection initiation.

Database is used both for cumulus' logs and for experiments.

There is 2 types of backends

1. Which DB to use; MongoDB, DynamoDB, SQL, etc.
2. Bridges to experiment DBs from experiment manager like Sacred or Sumatra. There will be no support for experiments not store in db.

Could support different backends for different bridges. Ex: cumulus' logs are store in MongoDB but experiments are with SQL.

NOTE: Only support MongoDB for now

• Graham
• Cedar
• Mila
• Khala

(Copy of bouthilx/research#16)
(Related to #19)

File system is horribly slow in the lab. Better to keep stuff on local (H|S)DD. Should write a script to keep synched stuff on /data/lisatmp4 and /Tmp. Could apply to clusters too.