better handling of failed function calls in a big list of jobs about workflow

[bernstei]

failure scenarios:

• abort as soon as one job fails : easy - just don't catch xpr.get_results() exception
• ignore failed jobs, just make sure configs are None or labelled in Atoms.info: easy - catch all xpr.get_results() exceptions, label configs, maybe warn about them
• deal separately with timed out or crashed jobs.
  • loop over all jobs with timeout=0. Report timed out jobs.
  • if any timed out, resubmit those hoping that RemoteInfo has been modified
  • loop over all jobs with desired timeout.

two flags in RemoteInfo

• ignore_job_failures, default False
  • just like skip_failures now, but maybe note reason for failure, or xpr.id, in Atoms.info
• resubmit_killed_jobs, default False.
  • make ExPyRe raise custom expyre.Timeout exception
  • If resubmit_killed_jobs:
    • loop over all jobs with timeout 0, note ones that raise an exception other than expyre.Timeout
    • resubmit jobs that died
    • loop over again

[bernstei]

May be nicer if libAtoms/ExPyRe#23 is done

[bernstei]

@gelzinyte do we want to automatically resubmit (i.e. when setting RemoteInfo.resubmit_killed_jobs = True) only jobs that died without creating a succeeded or failed file (i.e. killed by the queuing system), which then raise an ExPyReJobDiedError, or all jobs that raised any exception, whether it was raised by the remote python process and pickled so it can be re-raised by expyre or an ExPyReJobDiedError raised directly by expire?

The former is easier because it means that I don't have to clean up the job directory.