libatoms / workflow Goto Github PK

Right now if a DFT job fails, recovering is ugly. Running the operation again will note the failure during the result gather stage, but it will (by default at least) exit immediately. It will therefore not notice any other failures, and even if that failure is cleaned up manually, rerunning the workflow will resubmit that job, then get to the gather stage, and will be waiting for that job to finish before it even notices the later failures.

It would be better if the pipeline multiple remote jobs just kept track of all the failures, and reported all of them at the end of the loop. The only downside is that it will not be able to raise the same exception (necessarily), because different failures might have occurred due to different exceptions.

For brevity's and clarity's sake I like using pandas in my codes. I feel like this would be a useful addition to the modify_database scripts, as it would permit easy and straightforward filtering. Another usecase might be the plotting functions, although I have minimal experience with them.

I can post example scripts on Monday, I am out of town at the moment.

Change "dft_code" to more general term, starting here

Call to arbitrary database modify can happen outside of gap_multistage, since only overall rescaling of sigmas for different stages is needed. Remove that functionality, and do the modification from gap rss iter fit before calling gap multistage fit.

Remove call to calculate ref error inside gap multistage fit, and instead call it from gap rss iter fit after calling fit.

PR is being prepared.

Some ideas:

job_chunksize -> items_per_job

chunksize ->

• serial_chunk_size
• serial_chunksize
• num_serial
• num_serial_items
• num_serial_ops
• num_operations_in_serial
• items_in_pool

All modules need to be listed in setup.py, not just packages=["wfl"]. Same goes for UniversalSOAP and ExPyRe. Would packages=setuptools.find_packages(exclude=["test"]) do what you wanted to do in cc4aceb?

Though I've tried that and no modules get listed still (but wfl works):

In [1]: import wfl

In [2]: dir(wfl)
Out[2]:
['__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__']

add extra_info dict in all routines that generate new configs. Decorator?

Collect all arguments for iterable_loop, so these could be more easily passed to functions using iterable_loop and changed without modifying the signature of the calling function.

On some linux machines orca is a system executable that has nothing to do with the DFT code. Maybe all the calculator tests should require that the desired executable be in a particular env var, similar to ASE_{CALCULATOR}_COMMAND

Multistage gap fit writes some additional dicts (for descriptors and fit lines) that are, I think, needed for restarting interrupted runs. These are not staged out in remote jobs (and neither are initial stage fit potentials), and so remote jobs won't restart properly.

The docstring for dict_to_gap_fit_string claims that dicts are turned into : separated strings (e.g. what you'd need for "config_type_sigma"). However, it doesn't seem to actually do this, there's no evidence of code that does this (it's counting on ASE's key_val_dict_to_str, and that doesn't seem to do it), and the simple gap fit pytest doesn't actually test this functionality.

Rewrite interfaces to dft calculators (e.g. castep.evaluate_op) into classes inheriting from ASE calculator. Workflows may then be calculator agnostic by using the generic calculator. And choose different ways of parallelise the calculations, for example generic.run directly or generic.run_op within a function that is then parallelised with iterable_loop. An example is in generate_configs.vib.Vibrations.derive_normal_mode_info()

merge the various CUR implementations, perhaps using an external implementation such as asap/dscribe/librascale

wfl.parallelization has been entirely (I think) supplanted by wfl.pipeline. A few things (maybe only wfl.generate_configs.collision?) still use it, and those should be redone.

Use the upcoming ExPyRe functionality of job start buffering (see libAtoms/ExPyRe#12) to set up all the jobs needed for a set of remote runs and start them in a single call.

Currently autoparallelized functions can only gather results from individual operations if they are Atoms, so they can be saved into a ConfigSet_in. It would be helpful, and probably straightforward, to make it possible to gather arbitrary results, and just pack them into a list (i.e. only in memory, without the ConfigSet disk/ABCD backing store idea, although you could also save and/or pickle them).

separate isolated atoms and dimers?
remove bad default for bond lengths, either no default or a sensible one
rename max_cutoff -> min_box
rename _scale -> _range or vice versa

replace individual types with a lambda based filter, e.g.

filtered_list = simple_filter.select(list_configset, output_spec, lambda at : at.info[prop] < 5.0)

Deprecate bits of code that don't do things the workflow way, or just don't use it. They can be updated to do it the right way (so they can be wrapped by iterable_loop (soon to be autoparallelize), or moved to some user-specific place.

reactions_processing/
plotting/
some things in generate, e.g.
neb, ts, vib, radicals

some things in cli.py

I'm reporting this in regards to doing CASTEP geometry optimisations thorough workflow, and having issues with the output structures. Specifically, it seems the output structures do not fully correspond to the last structures in castep.castep file, as they should. While the forces do correspond, the lattice parameters of the workflow output structures are identical to those of the starting structures, which I suspect are used to hash the structures.

I mention that I am using the workflow on the old repository, but I checked the log here and cannot see that any changes have been made, so I think the problem may still stand. I could follow up once I move to the new workflow.
This may be actually more of an ASE feature that needs implementing, but I thought I'd flag this up just in case.

clean up everywhere

This is a terrible way of finding isolated atom configurations, because it'd match single atom primitive cells that might be there for their E(V) or virial(V). We need to test on config_type, or box size relative to cutoff.

Hi,
While working with wfl, I implemented a simple ase.emt calculator for fast computation, which helped me for developing. The changes include an example python script for an iterative training procedure. In case you're interested, I'd happily create a pull request.

Files affected are:

• wfl/calculators/dft.py

New files are:

• wfl/calculators/emt.py
• examples/EMT_iterative_gap_fit/gap_fit.py
• examples/EMT_iterative_gap_fit/EMT_atoms.xyz
• examples/EMT_iterative_gap_fit/init_md.traj
• examples/EMT_iterative_gap_fit/multistage_gap_params.json

@stenczelt if I understand correctly, the current ORCA calculator is outdated? If so, what's the latest version and would it be possible to merge it into workflow?

When I have been doing single FHI-aims calculations across multiple nodes (AIMS requires a lot of memory), I've been specifying resources in the remoteinfo.json like this:

"resources": { "n" : [4, "nodes"], "max_time": "24h", "ncores_per_task" : 1, "partitions": "highmem" }

Is this the right way to run this calculation? When I do this and manually set OMP_NUM_THREADS and WFL_AUTOPARA_NPOOL to 1 at the start of the job, I get 512 mpi tasks as expected (4 nodes with 128 cores each) and FHI-aims runs correctly. However, by default the code sets WFL_AUTOPARA_NPOOL equal to the number of tasks per node, which is 128. This then ends up parallelising through the chunk size which I do not want. Before I manually set the variables, the FHI-aims output was also telling me that OMP_NUM_THREADS was not 1.

Is this there a right way to do this, using more specification in the resources dictionary? I can't seem to follow how this part of the code passes the information around very easily.

move ace_fit.jl into ACE1pack.

move gap_multistage outside of workflow, probably in QUIP

• move checking for completed work into separate function, to be called explicitly from workflow
• move actual fitting work into separate function
• get rid of input/output file massaging, restart is broken anyway. Just default to output_files=rundir
• get rid of GAP-specific env var setting based on EXPYRE vars, people find it unexpected - make user do it manually in remote_info
• remaining code is essentially independent of fit() - turn into decorator?
• move descriptor_heuristics.py into QUIP as well

same for gap_simple.py, maybe also ace.py

To do

• Populate empty pages
• Revise current content for outdated material
• Collect minimal examples of specific tasks/operations
• Clean up READMEs

@bernstei what else?

by_function? Somehow embed the fact that it returns a bool?

deprecate by energy as part of #74

Some ideas to get started with

Pre docathon

• update function/variable names (especially in comments/docs examples) both workflow and expyre [NB]
• check if @iloop works cleanly with sphinx [NB]
• Fix docstring rendering online [EG]
• Get rid of outdated .yml and split main yml; change trigger for building docs [NB]
• Getting started page-stub [EG]

June 29 docathon

• check that all docstrings render correctly under https://libatoms.github.io/workflow/modules.html
• Fill in sub-sections in "Operations": https://libatoms.github.io/workflow/operations.html
• Examples https://libatoms.github.io/workflow/examples.html
  • What examples will be useful? (complex and simple ones)
    • Selection (esp. via lambda functions)
  • Getting started examples
  • Explain code snippet examples
  • some unit tests that can be converted to code-snippet examples?
  • make the examples automatically testable - somehow extract docs page from same file that's executable, make unit test to run executable
• Expyre
  • add two just-jobs examples (e.g. ORCA/CASTEPS, without Workflow)

For some other time

• test the tutorials/code snippets
• update cli:
  • make sure command line (click) function explanations render correctly
  • remove (clearly) outdated entries

Currently values such as certain default_{property}_sigma values are hardcoded in files such as in gap_rss_set_config_sigmas_from_convex_hull.modify We could allow the users more flexibility by replacing these hardcoded values with *kwarg arguments, with the previously hardcoded values being the default options.

Support nested structure (list of lists) in ConfigSet under a wider range of circumstances.

In memory: already supported

Files: already supported for multi-file with group iterator returning one group per file. Would need to be extended to single file with special tags, num_of_group and num_in_group, probably with additional tag last_in_group.

ABCD: will need tags, like single file suggestion

I have a gap-rss run started on 29th Nov 2021, with code version git https://github.com/gabor1/workflow/commit/968765772e20d9bab4e8ae21633a7a90bdb08e96-dirty.
This is related to the sigma-setting method in wfl/fit/modify_database/gap_rss_set_config_sigmas_from_convex_hull.py, but I noticed this method didn't change since, so the issues should still stand.

In my gap-rss run, I noticed in the training set for the final iteration that there are structures with energy_sigma = 0.001 (and 18 atoms say).

However, when I recently tried to fit a GAP to my own structures, I found reasons to believe that the energy_sigma and virial_sigma values when set per config, are in fact not per atom, but are taken as a total for the atoms in that structure. So for example the energy_sigma = 0.001 for 18 atoms above corresponds to 0.0000(5) per atom.

In the attached picture I show a case where I set the sigmas to around 1meV/atom for each of the structures, but the resulting errors/atom are orders of magnitude smaller; hence the suspicion.

This is corroborated by the bad force component fits from the gap-rss (note if github theme is dark one cannot read the labels):

And the table of errors, where the force data has much larger errors in ratio to their sigma, than energy or virials do.

I spoke with Gabor and he concurs that when set per config, energy_sigma, virial_sigma and hessian_sigma should be scaled by the number of atoms. While force_sigma should not be scaled.

Just making a note that the mechanism of "parallelise Castep single point energy/forces across multiple configs and if one of them fails return no energy/forces results for that config, but don't fail all of them" seems to be broken for Castep. Relevant to @imagdau

Generate perturbed configs for force constant calculations, probably using phonopy and/or phono3py. A version is being written, will be contributed as a PR when ready.

Hi there I am new to wfl (though working with GAP/SOAP for quite some while now).
I was wondering why the per-atom descriptor generated in calc_autopara_wrappable (quippy.py) is not normalized even with normalize=True.
Naturally I included the following in my personal clone:

if local:
...
    if combined_descs.shape[1] > 0:

and then include

        if normalize:
            for k in combined_descs:
                if np.linalg.norm(k) > 0.1:
                    k /= np.linalg.norm(k)

befor then adding the combined descriptor to the ASE.atoms object

        if Zcenter is None:
            use_key = key
        else:
            use_key = f'{key}_Z_{Zcenter}'
        at.new_array(use_key, combined_descs)

Maybe this could/should be included in the main branch as well. Or maybe I do overlook some critical thing here. As mentioned still new on this project.
Best, Jakob

make the convex hull based selection able to operate in a space other than _x and _V.

Lots of docstrings are not formatted according to what sphinx expects. I will start to fix those, at least the ones that give warnings and I can figure out, and push to the docs branch.

cli.py is too long, and also includes some things that probably don't belong in it at all. Refactor it to make the source files more modular, and deprecate or get rid of unneeded bits.

Add functions to wfl.select_configs.by_descriptor to select configs based on per-atom descriptors, rather than global (per-config) ones. Requires version of prep_descs_and_exclude that extracts per-atom descriptors (from Atoms.arrays), and version of CUR_conf_global that figures out indexing from CUR-selected per-atom descriptors to corresponding configs.

Replace cumbersome

def run(inputs, outputs, arg1, ...):
    return iterable_loop(inputs, output, op, arg1, ...)

with decorator (although not using @ syntax) iloop, which based on ideas here

May be able to do

def run(inputs, outputs, arg1, ...):
     # some code
run = functools.partial(iloop, run)

if we can figure out how to do run.__doc__ = run.__doc__.format(iloop_docstring_post=iloop_docstring_post)

• ConfigSet_in -> ConfigSet, ConfigSet_out -> OutputSpec

• pipeline/ -> autoparallelize/, iterable_loop -> autoparallelize

• All low level *_op -> *_autopara_wrappable

• generate_configs/ -> generate/

• select_configs/ -> select/

• move selection_space.py -> select/

• move mpipool_support.py -> autoparallelize

• move calc_descriptors.py -> descriptors/calc.py

closed by #64

Replace prep stage writing files to return strings, workflow can write them if that makes sense.

Maybe remove separate prep stage, if it can be done cheaply enough, and do it "on the fly" during actual running of buildcell.

docs:

• move CUR from generation to selection section

docstrings:

• make flat_histogram docstring better, since it was confusing to explain
• add a docstring to atoms_and_dimers.py::prepare

gap-rss cli can now use a file of extra fitting configs, but it should also be possible to re-evaluate those (using the current reference calculator) in addition to using them as-is.

make all code that depends on atom "sizes" use a uniform data structure, maybe a class (so that it can be switched from Z1-Z2 using arithmetic mean to some other expression of r_Z1 and r_Z2).

E.g. start from length_scales.yaml dict, but add functionality for per-pair things.

Code that interrupts using debug_interrupt_stage and debug_interrupt_committee was removed, so test that claims to check interrupted runs doesn't actually do so.

Test of pwscf via Espresso calculator of an actual converged calculation is broken. The default parameters fail to converge, and pwscf returns an error status after 100 iterations. The new Espresso detects this as a convergence failure. Loosening the convergence tolerance allows it to stop without apparent failure, but then the hard-coded values are wrong.

I suggest finding some system/parameters where convergence is achieved, so the return values can be meaningfully tested.

Make all selection routines be able to return results in a few ways

• list of configs matching criteria
• mark a specified Atoms.info with bool indicating match
• list of indices (flattened, or tuples for nested ConfigSet?) for matching configs.

Give all select routines an exclude argument that can make them ignore certain configs (by info field? by lambda? by index?)