imclab / stereopipelinetest Goto Github PK

StereoPipelineTest is a comprehensive, distributed, and fully
automated test suite for the Ames Stereo Pipeline (ASP). It attempts
to cover most, if not all, of the ways in which ASP can be used, and
the test suite should be updated regularly as more functionality is
added to ASP.

Usage: bin/run_tests.pl settings.conf

After the tests finish, a report is written to 'report.txt', and the
report is also emailed to the user.

A sample settings file is provided, named 'pfe.conf'. The settings
file has:

1. The tests to run (wildcard expressions are accepted).
2. The machines to distribute the runs across (they must be accessible
   via ssh and share disk storage).
3. How many processes to use on each machine (each process in turn uses 
   multiple threads).
4. If to do strict validation (that is, not allow, vs. allow, a small
   discrepancy between current and reference runs, more below).
5. Environmental variables, such as the path to the ASP executables.

Each test needs to be in its own directory. A test is executed by
running the script 'run.sh' in that directory, which should create an
output directory named 'run'. A 'gold' directory must be present,
which has the reference run. At the conclusion of the run, the result in
the 'run' directory is compared to the reference in 'gold'. The test
will fail if the produced result is different or absent.

If a new test is added, the name of the test directory must be listed
in the settings file or match the wildcard pattern already present
there. Each test must have a 'run.sh' file, and a validation script,
named 'validate.sh'. The 'validate.sh' script must return exit status
0 on successful validation, and non-zero otherwise.

When tests fail, which is inevitable when something changes, and the
new results are deemed acceptable, the 'gold' reference directory
needs to be updated by copying the output from the 'run' directory. 

This test suite runs on multiple machines and architectures. The most
comprehensive tests are done on pfe (Pleiades supercomputer head
nodes). Results on other machines may differ from the results on the
pfe machines. It is then possible to specify that a tolerance for the
maximum allowed discrepancy be accepted, by setting strictValidation =
0 in the configuration file for the desired machine. Those tolerances
can be put there as well, one per line, in the format: "testName
tolerance". Such a list is auto-generated by running the command:

bin/print_allowed_error.pl report.txt

where 'report.txt' is the latest run on the current machine. On pfe
machines no tolerances should be used, as we'd like to know when
results on those machines change even by tiniest bit.