shirtsgroup / physical_validation

Physical validation of molecular simulations

physical_validation: A Python package to assess the physical validity of molecular simulation results

physical_validation is a package testing results obtained by molecular simulations for their physical validity.

Please cite

Merz PT, Shirts MR (2018) Testing for physical validity in molecular simulations. PLoS ONE 13(9): e0202764. https://doi.org/10.1371/journal.pone.0202764
Merz et al., (2022). physical_validation: A Python package to assess the physical validity of molecular simulation results. Journal of Open Source Software, 7(69), 3981, https://doi.org/10.21105/joss.03981

physical_validation incorporates the functionality of checkensemble.

This software is developed in the Shirts group at University of Colorado in Boulder.

Documentation

Please check https://physical-validation.readthedocs.io for the full reference.

physical_validation's Issues

Add parser for AMBER

Coordinate with SiliconTherapeutics - they have a working draft of this.

Implement code features requested for submission to Journal of Open Source Software

https://joss.theoj.org/
https://joss.readthedocs.io/en/latest/submitting.html

Investigate physical validation of anisotropic pressure coupling

We want to be able to test the physical validity of anisotropic barostats. The theory of this is not completely worked out yet.

Obsolete Github pages deployment

We are still deploying a dummy webpage to https://shirtsgroup.github.io/physical_validation/ redirecting to readthedocs. For a short time (a few months in 2017, probably), we used the github pages website as the official documentation. We could probably get rid of that redirect by now...

Think about pymbar dependency

physical_validation has a dependency on pymbar. This is only needed to access the timeseries submodule of pymbar. It seems a bit of an overkill to require the full pymbar package just to decorrelate a few timeseries...

Set up CI trough Github actions

We should also remove previous TravisCI hooks.

Streamline use of Units throughout code

The UnitData, part of every SimulationData object, is not consistently used throughout the code. This is a reminder to check the use cases, and if possible simplify the UnitData object.

Make the variables in the axes consistent

Right now, in the graphs, it prints "Energy" in the distribution no matter what the variable is (enthalpy, energy, pressure); this should be improved (could borrow code from checkensemble)

Adding grand, semigrand, and Gibbs ensemble

grand and semigrand are implemented in checkensemble, so it would be mostly porting over.

Plot of sampled distribution and p-value from kinetic energy validation

Hi,

I am using the Maxwell-Boltzmann ensemble validation function to check a system of mine. When executed, my simulation result fits the Boltzmann distribution quite nicely, but the p-value is low (3.95e-08) and thus the null hypothesis is rejected. This seems to be contradictory. I am attaching the Gromacs files I have read in for this function, as well as PDF of the plot produced. Any insight on this would be appreciated.

Thanks,
Ray Matsumoto

Gromacs files: pyrrol_valid.zip

Plot: KE_val.pdf

Make package installable (via pip? conda?)

Migrated from shirtsgroup/checkensemble#3

• pip
• conda

Use fancy repo badges

Once we have our CI up and running, we'll want to have a lot of fancy repo badges 😎

Discuss pull request requirements / branch protection

We should come up with a policy on who is allowed to push to which branches, what needs to be fulfilled to allow PRs to be merged, who is allowed to approve, etc.

Review code-level documentation

This concerns the documentation of the code itself, not accompanying tutorials / manual. Check for completeness, correctness, and accessibility to new developers.

TopologyData name adjustment

Perhaps we should change TopologyData to MolecularData? Topology data suggests something much more complicated than the relatively simple data that is needed for the tests.

Update/tweak the tutorials

Located in https://github.com/shirtsgroup/physical_validation_workshop. Place specific issues to fix that there.

Add example for usage with OpenMM

Usage with OpenMM should work by feeding physical_validation python data structures. We should make sure this works and have examples in the documentation.

Parsing molecular masses from [default][atomtypes]

physical_validation/physical_validation/util/gromacs_interface.py

Lines 316 to 322 in c8a5a23

	masses = []
	for atom in topology[molecule]['atoms']:
	if len(atom.split()) >= 8:
	masses.append(float(atom.split()[7]))
	else:
	code = atom.split()[1]
	masses.append(float(topology['defaults']['atomtypes'][code].split()[3]))

Hello everyone,

I ran into problems with the kinetic_energy.equipartition() method and traced it back to parsing the wrong masses from my topology. It seems like the masses are parsed from the [moleculetype][atoms] if possible but the fallback is to take it from the [defaults][atomtypes] section. When falling back to the else statement the fourth value in each line of [defaults][atomtypes] is parsed as molecular masses, however, at least for my topology this is the charge and I actually want to parse the third value ([...].split()[2]).

As I'm using an in-house FF and all of the topology is written by me or colleagues this might be specific to our syntax. However, I looked at some of the ffnonbonded.itp of the amber FF and they also should have run into the same problem (from amber94.ff/ffnonbonded.itp):

; name at.num mass charge ptype sigma epsilon
Br 35 79.90 0.0000 A 3.95559e-01 1.33888e+00 ; Converted from parm99.dat
C 6 12.01 0.0000 A 3.39967e-01 3.59824e-01
CA 6 12.01 0.0000 A 3.39967e-01 3.59824e-01

Python issue that occurs when importing library

When I import the physical validation package, i.e.,

import physical_validation as pv

I get the error:

Traceback (most recent call last):
  File "ana_flatfile.py", line 2, in <module>
    import physical_validation as pv
  File "/Users/cri/anaconda3/lib/python2.7/site-packages/physical_validation/__init__.py", line 43, in <module>
    from . import kinetic_energy
  File "/Users/cri/anaconda3/lib/python2.7/site-packages/physical_validation/kinetic_energy.py", line 36, in <module>
    from .util import kinetic_energy as util_kin
  File "/Users/cri/anaconda3/lib/python2.7/site-packages/physical_validation/util/__init__.py", line 31, in <module>
    from . import ensemble
  File "/Users/cri/anaconda3/lib/python2.7/site-packages/physical_validation/util/ensemble.py", line 774
    '[0/{:d}]'.format(bs_repetitions), end='')
                                          ^
SyntaxError: invalid syntax

Note, on this machine I am running: Python 2.7.15 :: Anaconda, Inc.

Have webpage on separate repository

Add more real-world examples to documentation

Examples (to be extended):

• Replica exchange
• Expanded ensemble

Figure out a way to take scaling of systems into account in checking the kinetic energy

For large enough N, then any errors in the distribution will eventually show up (i.e. for the same algorithm, the p-value should go down as N increases). It would be important to understand how significant this is, and if possible, find a way to make sure validity checks are relatively insensitive to the size of the system.

Make sure that documentation is beginner-friendly

Ideally, the documentation should allow users having moderate knowledge of molecular simulations to run tests on their systems within a few minutes. It should then have additional information for users interested in the details of theory or implementation. It would be very helpful if group members that have little or no knowledge of the physical validation package could verify if the current documentation is beginner-friendly, and suggest improvements!

Brainstorming ideas/tasks for making physical validation a group project

• Adding continuous integration testing framework (from MolSSI cookie cutter)
• Adding pytest tests to the CI
• adding different nontrivial examples
  • checking replica exchange simulations
  • Checking a vapor phase simulation?
  • What else?
• Improving the documentation at read the docs
• Adding support for different molecular simulation tools
  • AMBER (just needs to be tested once we are given it by SiliconTherapeutics)
  • LAMMPS
  • HOOMD (can add angular kinetic energy validation)
• perform pylint/black reformatting
• Finish python3 conversion.
• List the things from JOSS needed.
• Make it conda-installable?

conda-forge to-do list

The PR is open at conda-forge/staged-recipes#11370 but we should do a few things, and make a new release, before anticipating that PR be merged. This issue is to document those things

• General cleanup - largely handled by #41
• Tests - #59
• License - #62 not a blocker but if it is going to be changed, it may be easier to have only the new license on forge
• New release (including new license file packaged)
• Add conda instructions on "Installation" page.
• Update release docs https://github.com/shirtsgroup/physical_validation/wiki
• Update license in conda recipe - we moved to MIT
• Tests have additional dependencies, see devtools/conda-envs/test_env.yaml. Do we need to include them in the conda recipe? Can we reuse that .yaml file to reduce the number of places that list dependencies?

This is all that comes to mind - any other suggestions?

integrator convergence example

Seems like we should probably go out three steps instead of two to make the patterns clear.

Add support for grand and semigrand ensembles

Already exists in checkensemble need to port over.

Add developer guide to documentation

Should include explanation of workflow, requirements to contributions, style guide, how tos for maintenance of the project, other useful topics.

Related to #45.
#42 #43 #44 should probably be made a part of this documentation.

naming conventions for better readbility?

I wonder if it would be better to make the function names more readable, for example

physical_validation instead of physicalvalidation
kinetic_energy instead of kineticenergy

Add parser for LAMMPS results

Add parser for HOOMD

Generating machine-readable results

One nice suggestion was to generate machine-readable results for the output of physical_validation. The steps would be

1. Determine a JSON schema for the results
2. The functions fill in the schema.
3. the logfile is actually the output of running the JSON internal representation through an interpreter.

This is lower priority, but a good idea overall - would make it easier to have GROMACS take return values, or to have other programs incorporate results in an automated way.

Evaluate whether to use units packages instead of own units.

For example, Pint https://pint.readthedocs.io/en/0.7.2/

Retire py2 support

Python 2 is finally out of support 🥳

There are workarounds in different places of the code to make it compatible with both py2 and py3. We should find these and remove any hacks not needed anymore.

Catch the NaN error that occurs when histograms are not well-populated

Migrated from shirtsgroup/checkensemble#2

mrshirts commented on Jun 20, 2016:

When distributions are chosen poorly, it's very likely the ratio of the histograms will be undefined, as the denominator will be zero. This needs to be caught with a proper warning.

mrshirts commented on Mar 6, 2017:

The issues show up as warnings:

checkensemble/checkensemble/checkensemble.py:910: RuntimeWarning: divide by zero encountered in log
ratio = numpy.log(hlist[1]/hlist[0]) # this should have the proper exponential distribution
checkensemble/checkensemble/checkensemble.py:911: RuntimeWarning: invalid value encountered in divide
dratio = numpy.sqrt((dhlist[0]/hlist[0])**2 + (dhlist[1]/hlist[1])**2)
checkensemble/checkensemble/checkensemble.py:910: RuntimeWarning: divide by zero encountered in divide
ratio = numpy.log(hlist[1]/hlist[0]) # this should have the proper exponential distribution

Finish the automated graphic capability.

Modernize checkensemble.py

To date, checkensemble.py has hardly been changed from the original version in https://github.com/shirtsgroup/checkensemble. Eventually, we should clean it up.

License discussion

From #40:

SB: was there a particular reason to go with LGPL rather than something like MIT?

PM: Part of the code is currently in the GROMACS repo - the package was shipped with GROMACS before it became a stand-alone package and has not yet been removed. (When we have a stable release, I'll make sure that GROMACS gets the package from pip rather than shipping an outdated version.) At the time, we chose the same license as GROMACS to keep things simple.

MS: Agreed [that licensing can certainly be rediscussed should probably be a separate issue where we can discuss and come to a conclusion]

so here's that issue!

physical_validation needs testing!

Test coverage right now is close to zero. This is a catch-it-all issue to discuss setting up testing.

Kolmogorov-Smirnov test result: p = 0.000000

In the documentation, adjust it so it uses the updated format to report p, rather than
"Kolmogorov-Smirnov test result: p = 0.000000"

Make the repository public

Seems ready to launch!

Broken docs link

On the GitHub landing page next to your tagline the website link is https://shirtsgroup.github.io/physical-validation/ which points to a redirect.

Evaluate use of other packages reading molecular simulation result files

Connected to #51 #52 #53 #54 - is having our own parsers the best way to move forward? Could MDTraj / MDAnalysis / pytraj / other packages make our lives easier? Note that physical_validation has some very specific requirements of what information is needed, which were not fulfilled by any other package last we checked.

Related to do: Make list of information physical_validation needs to read from simulation results files.

Handle non-cuboid boxes in GROMACS parser

Document how documentation push to readthedocs works

Better labeling for equipartition molecule parts

Some of the abbreviations ('rni' in particular) in the output are vague. It would be better to print out the full label, i.e. "rotational and internal'

Extensively test GROMACS topologies with #include / #ifdef statements

Starting from commit 6211d2a, the GromacsParser interprets preprocessor commands instead of simply ignoring them. This issue is a reminder to test that on a larger set of topologies.

Document how to release package

• pip
• conda

Document how to set up CI

Check / prove that integrator test is valid for temperature / pressure coupling

The integrator test analyzes the fluctuations of the constant of motion of a simulation. Theoretically, this is expected to be proportional to dt^2 (since the integrator preserves the constant of motion of the shadow Hamiltonian, but we are observing the constant of motion of the real Hamiltonian...).

We are only advertising to use this in NVE simulations, since we never cleanly proved that it would work with constants of motion of extended systems using temperature or pressure coupling. It shouldn't be too hard to check this & update the documentation accordingly.