e3sm-project / cmdv-testing

Examples from the CMDV-software project on functional testing

Quick start:

1. Testing instructions for the cmdv-test-runner

Documentation

Release

Preparing 0.9 beta

Jenkins

Start Jenkins server and slave, server UI at localhost:8080:

• docker-compose -f Docker/jenkins-compose.yaml up

User name and password for Jenkins admin:

• cmdv:cmdv

cmdv-testing's Issues

The baseCommand does not work under "deploy"

The "baseCommand" is not executed under "deploy".

failing nightly tests of cmdv-test-runner due to test output getting written to error.log instead of screen

The nightly cmdv-test-runner CDash tests I set up are failing due to changes to cmdv-test-runner that made the output be written to error.log instead of the screen:

https://my.cdash.org/testDetails.php?test=46129640&build=1625679

I am attaching a sample log file. One can see that all the tests passed from the output:

Ran 4 tests in 0.001s
OK

I can change my testing scripts to look in error.log instead of the console output for the string indicating the self-testing of cmdv-test-runner was successful, but it seems to not make sense for output from a successful run to go in an error log file.
error.log

stdout not getting piped to file

cmdvVersion: v1.0
class: TestWorkflow
label: Hello World Example
steps:
  deploy: none
  run:
    label: hello world
    run:
      stdout: hello_world.out
      stderr: hello_world.err
      baseCommand: ['echo Hello World!']
      out:
        log:
          type: File
          glob: hello_world.out

No hello_world.out file is written

Improve wiki documentation

Remove gitpython dependency

We are not using the functionality provided by the module. Remove the module.

Add unit tests

Incorrect running/building path in cmdv-test-runner

The test case of water uptake process in MAM has been added to cmdv-test-runner. However, the sources codes are compiled at a different path from the running path. So a long relative path is required in the .yaml file.

Issue when running cmdv-test-runner with python3

I recently switched my workstation to use python3.7 by default, and noticed I get the following error when running the cmdv-test-runner self-test:

1: Running UnitTest/Verification suite :
1:
1: Traceback (most recent call last):
1:   File "../scripts/cmdv-test-runner", line 26, in <module>
1:     from Workflows.CMDV import Workflow
1:   File "/home/ikalash/CMDV/nightlyCDash/build/Build/lib/python/Workflows/CMDV.py", line 2, in <module>
1:     from Workflow import Workflow as Parent
1:   File "/home/ikalash/CMDV/nightlyCDash/build/Build/lib/python/Workflow.py", line 212
1:     except UnicodeDecodeError, e:
1:                              ^
1: SyntaxError: invalid syntax

I think the except UnicodeDecodeError syntax is not recognized with python3. Could you please have a look at this? It would be good to be able to run the code with python2 as well as python3.

Extend cmdv-test-runner to work with python 3

Issues with python 3 have been reported (https://acme-climate.atlassian.net/wiki/spaces/CMDV/pages/907543103/Test+runner+for+water+uptake+process?focusedCommentId=907510479#comment-907510479), e.g.,

Traceback (most recent call last):
  File "//Users/sunj695/Downloads/CMDV-testing/scripts/cmdv-test-runner", line 26, in <module>
    from Workflows.CMDV import Workflow
  File "/Users/sunj695/Downloads/CMDV-testing/lib/python/Workflows/CMDV.py", line 2, in <module>
    from Workflow import Workflow as Parent
  File "/Users/sunj695/Downloads/CMDV-testing/lib/python/Workflow.py", line 212
    except UnicodeDecodeError, e:
                             ^
SyntaxError: invalid syntax

Test for step output

Test if step is producing output and check if valid

Change directory structure and clean up code