buildtesters / buildtest Goto Github PK

provide an easy installation method by letting pip install buildtest and its components.

Seems to be a bug when $BUILDTEST_TESTDIR/ebapp is not created in advance.

(buildtest-0.5.0) [siddis14@adwnode1 buildtest-framework]$ _buildtest build --system acl
Traceback (most recent call last):
  File "/home/siddis14/.conda/envs/buildtest-0.5.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/siddis14/.conda/envs/buildtest-0.5.0/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/siddis14/github/buildtest-framework/buildtest/main.py", line 43, in <module>
    from buildtest.tools.menu import buildtest_menu
  File "/home/siddis14/github/buildtest-framework/buildtest/tools/menu.py", line 49, in <module>
    class buildtest_menu():
  File "/home/siddis14/github/buildtest-framework/buildtest/tools/menu.py", line 67, in buildtest_menu
    app_choices = run_app_choices()
  File "/home/siddis14/github/buildtest-framework/buildtest/test/run/app.py", line 41, in run_app_choices
    app_name_list = [ f.path for f in os.scandir(app_root_testdir) if f.is_dir()]
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/buildtest-tests/ebapp'

Add support for csh, tcsh, zsh, ksh.

Any file extension with .rb should be processed by ruby wrapper

Add flag --scan-Python-packages, --scan-R-packages, and --scan-Perl-packages to report list of tests found in buildtest against the packages installed in Python, Perl and R.

This will be useful to provide users information on what tests can be generated from buildtest repos for R, Python, and Perl. Currently all tests are generated and behavior should be done on per package basis if found in the system.

Permutations for each build opts to see which set of operations result in the best mechanism for building software.

Add support for command history support similar to yum history command in linux. This will only show history of buildtest commands that may be useful to see track of all buildtest commands. The following options could be as follows

buildtest history --build and short option buildtest history -b for showing history of builds
buildtest history --run and short option buildtest history -r for showing history of runs
buildtest history --all and short option buildtest history -a for showing history of all buildtest commands

buildtest can write all buildtest command to file (i.e /tmp/buildtest/history/build.txt, /tmp/buildtest/history/run.txt, /tmp/buildtest/history/all.txt). This can be controlled by configuration variable BUILDTEST_HISTORY_DIR or environment variable that is set to /tmp/buildtest/history.

Whenever any buildtest command is executed it will write to /tmp/buildtest/history/all.txt and it may run write to build.txt or run.txt if buildtest builds or runs a test

This file will append any new command to this file and it can be viewed by reading the file or running the buildtest history command.

The output format could look something like this for /tmp/buildtest/history/all.txt

ID  | COMMAND
----|------------------------
 1  |  buildtest -V 
 2  |  buildtest show -c
 3  |  buildtest --help
 4  |  buildtest build -S compilers 

buildtest could leverage the history concept to rerun commands something like buildtest history --id <id> --type <all | build | run> where --id is the numerical index and --type is the different files. To rerun the first command it could be buildtest history --id 1 --type all. There could be a short option -i and -t for these commands

Python yaml config and code directory needs to be structured by python package. Create a python package directory where the code and yaml file will reside.

Directory structure for numpy
Python/code/numpy/
Python/config/numpy

The test scripts generated by buildtest should be in format Python/pyver/toolchain-name/toolchain-ver/numpy/example.py

Create a CMakeLists for each python package

(buildtest) [siddis14@adwnode1 buildtest-framework]$ _buildtest  --logdir=/tmp  build -s GCCcore/6.4.0
Detecting Software:  GCCcore/6.4.0
--------------------------------------------
[STAGE 1]: Building Binary Tests
--------------------------------------------
Detecting Test Type: Software
Processing Binary YAML configuration:  /home/siddis14/buildtest-configs/buildtest/ebapps/gcccore/6.4.0/command.yaml
Generating  10  binary tests
Binary Tests are written in  /tmp/buildtest-tests/ebapp/GCCcore/6.4.0/
--------------------------------------------
[STAGE 2]: Building Source Tests
--------------------------------------------
Processing all YAML files in directory:  /home/siddis14/buildtest-configs/buildtest/ebapps/gcccore/config
Generating 22  Source Tests and writing at  /tmp/buildtest-tests/ebapp/GCCcore/6.4.0/
Writing Log file:  /tmp/buildtest/GCCcore/6.4.0/buildtest_16_46_03_10_2018.log

(buildtest) [siddis14@adwnode1 buildtest-framework]$ _buildtest --show
         buildtest configuration summary
         (C): Configuration File,  (E): Environment Variable
BUILDTEST_ROOT                                     (E) = /home/siddis14/github/buildtest-framework
BUILDTEST_CLEAN_BUILD                              (C) = False
BUILDTEST_CONFIGS_REPO                             (C) = /home/siddis14/buildtest-configs
BUILDTEST_CONFIGS_REPO_SOFTWARE                    (C) = /home/siddis14/buildtest-configs/buildtest/ebapps
BUILDTEST_CONFIGS_REPO_SYSTEM                      (C) = /home/siddis14/buildtest-configs/buildtest/system
BUILDTEST_ENABLE_JOB                               (C) = False
BUILDTEST_IGNORE_EASYBUILD                         (C) = False
BUILDTEST_JOB_TEMPLATE                             (E) = /home/siddis14/github/buildtest-framework/template/job.slurm
BUILDTEST_LOGDIR                                   (C) = /tmp/buildtest
BUILDTEST_MODULE_NAMING_SCHEME                     (C) = FNS
BUILDTEST_MODULE_ROOT                              (C) = /clust/app/easybuild/2018/Broadwell/redhat/7.3/modules/all
BUILDTEST_PERL_REPO                                (C) = /home/siddis14/github/Perl-buildtest-config
BUILDTEST_PERL_TESTDIR                             (C) = /home/siddis14/github/Perl-buildtest-config/buildtest/perl/code
BUILDTEST_PYTHON_REPO                              (C) = /home/siddis14/github/Python-buildtest-config
BUILDTEST_PYTHON_TESTDIR                           (C) = /home/siddis14/github/Python-buildtest-config/buildtest/python/code
BUILDTEST_RUBY_REPO                                (C) = /home/siddis14/github/Ruby-buildtest-config
BUILDTEST_RUBY_TESTDIR                             (C) = /home/siddis14/github/Ruby-buildtest-config/buildtest/ruby/code
BUILDTEST_R_REPO                                   (C) = /home/siddis14/github/R-buildtest-config
BUILDTEST_R_TESTDIR                                (C) = /home/siddis14/github/R-buildtest-config/buildtest/R/code
BUILDTEST_SHELL                                    (C) = sh
BUILDTEST_TESTDIR                                  (C) = /tmp/buildtest-tests

--logdir not overriding path. Need to fix this

Create a submenu hpl under benchmark that will be able to run the HPL benchmark.

_buildtest benchmark hpl

Create a YAML framework for HPL benchmark that create a test script to run HPL. Preferably this test should be run via job scheduler

buildtest should detect .R files and set compiler to either R CMD BATCH or Rscript

any .lua extension should be set processed by lua wrapper

Some tests are expected to fail (try/catch, Exception) to test functionality so need a way to provide expected exit status to see if test succeeded for these types of tests.

name: example
source: example.c
exit_code: 1

The --testset option can be replaced by the following options
Python: --python-package
R: --r-package
Perl: --perl-package
Ruby: --ruby-package

The command below will attempt to build tests for all python packages present in software and found in Python-buildtest-config repo

buildtest -s Python/X.Y.Z --python-package

buildtest can auto-populate choice selection for option --python-package so that user may do the following

buildtest -s Python/X.Y.Z --python-package numpy,scipy,tensorflow

This same concept can be extended for other options.

Each job run by buildtest should be able to control process through an option controlled by buildtest or inside the test script.

If option is set in test script, user would be interested testing same code with different jobslot configuration so an option to dynamically create tests with different job slot would be nice.

Create a submenu hpcg under benchmark that will run HPCG benchmark.

_buildtest benchmark hpcg

Create a YAML framework for hpcg benchmark that create a test script to run hpcg . Preferably this test should be run via job scheduler

buildtest variants could enhance test scripts by writing a different test for each permutation initial support can start out by doing flag variants to build test with different compiler flags

testblock: singlesource
test:
    source: hello.cpp
    compiler: gnu
    variants:
      flags:
        - O3
        - O2
        - O1 
    module:
      - gcc

A test variant for compilers may look as follows for a OpenMP example

testblock: singlesource
test:
    source: omp_mm.c
    vars:
        OMP_NUM_THREADS: 4
    variants:
      compiler: gnu
         ldflags: "-fopenmp"
         module: 
           - gcc
       compilers: intel
         ldflags: "-qopenmp"
         module:
           - intel
       compiler: pgi
         ldflags: "-mp"
         module: 
            - pgi

To compile a openmp example with gcc, intel and pgi it will require different flags.

gnu: gcc -fopenmp omp_mm.c -o omp_mm
intel: icc -qopenmp omp_mm.c -o omp_mm
pgi: pgi -fopenmp omp_mm.c -o omp_mm

buildtest can create one test script per variant the naming scheme could be as simple as
omp_mm.c.yml.variant_<first 15 digits of sha256>.sh where omp_mm.c.yml is the yml file https://github.com/HPC-buildtest/buildtest-configs/blob/master/buildtest/suite/openmp/matrixmultiplication/omp_mm.c.yml to ensure unique names for test scripts

Need to present buildtest logging in a cleaner method for troubleshooting. Currently too much text is crammed together. Need to pick and chose what should be logged and to what level.

https://www.python.org/dev/peps/pep-0008/
https://www.python.org/dev/peps/pep-0257/
https://www.python.org/dev/peps/pep-3101/

For more details on string formatting check out https://realpython.com/python-f-strings/ and https://pyformat.info/

Need to implement a method to detect OS to allow buildtest to create system package test for any linux distribution. Currently it supports Fedora distribution only

buildtest should present its test cases in CDASH (a testing framework dashboard) see https://www.cdash.org/

This would be helpful to present users with a visual overview of all the tests.

See build https://readthedocs.org/projects/buildtest/builds/8669213/

The issue is importing buildtest modules in readthedocs due to environment setup

Getting errors like

WARNING: autodoc: failed to import module 'benchmark' from module 'buildtest.benchmark'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/latest/lib/python3.7/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/src/buildtest/benchmark/benchmark.py", line 24, in <module>
    from buildtest.benchmark.osu import run_osu_microbenchmark, list_osu_tests, osu_info
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/src/buildtest/benchmark/osu.py", line 31, in <module>
    from buildtest.tools.config import config_opts
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/src/buildtest/tools/config.py", line 51, in <module>
    DEFAULT_CONFIG_FILE = os.path.join(os.getenv("BUILDTEST_ROOT"),"settings.yml")
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/latest/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

This causes the documentation for https://buildtest.readthedocs.io/en/latest/command_reference.html to not show any buildtest methods.

When building documentation locally the command reference page is generated properly.

By default buildtest will clean the root of software test directory to avoid duplicate test names added in CMakeList.txt.

buildtest can mimic this same behavior with option --rebuild which will clean the test directory before writing test and the default behavior will be to not delete the root of software test directory. This may be useful if user is running multiple buildtest commands to generate test for Python packages or rebuild test with different shell.

buildtest should provide feature to run HPC benchmarks using buildtest --benchmark which can be used for generating test-scripts that can be later run via CTest or job scheduler. Initially will focus on MPI benchmarks to get performance metrics with various processor configuration.

buildtest should automatically detect # of cores in system and determine permutation and generate all the test scripts.

Tests can be generated in $BUILDTEST_TESTDIR/benchmarks/$benchmark-name/

Test scripts should encode processor configuration to distinguish test cases with different node/ppn parameters. Test scripts can follow the following naming format:

$TESTNAME_$PPNx$NODE.sh

For instance the osu_allreduce from OSU benchmark can be named as follows

PPN = 1 Node = 1 osu_allreduce_1x1.sh
PPN = 2 Node = 1 osu_allreduce_2x1.sh
PPN =2 Node = 3 osu_allreduce_2x3.sh

To run OSU benchmark it can be done by running the following

buildtest --benchmark=osu

To run STREAM do the following

buildtest --benchmark=stream

Options to --benchmark will be name of benchmark name that should have separate python module to initialize the environment necessary to build the benchmark, whether it is precompiled binary or needs to be build from source.

buildtest currently supports Hierarchical Module Naming Scheme, need to have option in buildtest to process modules based on module naming scheme to generate header information in test script correctly.

The testblock: python2 should be a class that will parse this yaml configuration and return a test script to run Python 2.x code.

A similar testblock can be implemented for python3 where pip, virtualenv will be compatible for python3 and assuming python script that is to be run is python 3 compatible.

The configuration below could have support for virtual environment as yaml keys to allow user to configure the virtual environment before running the python code.

testblock: python2
source: random.py
pyver: 2.6, 2.7, 2.7.10, 2.7.15
virtualenv: yes
virtualenvopts:
  name: venv1
  download: yes
  no-site-packages: yes
  python: /usr/bin/python

pythonpackages:
  - numpy
  - scipy

testblock:
  name:  random
  

The script will be named random.sh with the following content

#!/bin/sh
virtualenv --download --no-site-packages -p /usr/bin/python venv1
source $HOME/venv1/bin/activate
pip install numpy scipy
python random.py

This feature will report all the tests that can be run based on the available software found in the system and all the YAML configs found in buildtest.

The user will get a report for all the packages and # of tests for each package. This will include system package test by checking all YAML configs in system package and installed system package found in the system by using rpm -q

buildtest should create test scripts for system packages to automate test. Create a flag --systempkg to generate test for systempkg.

ability to create YAML config for binary test for system & easybuild packages. This way tests configs can be generated very quickly.

buildtest should have an option for generating tests based on cpu affinity for all applications. This would be particularly useful for users in NUMA architecture to fine-tune their tests based on cpu topology

Allow users to submit test their conda environment

buildtest -s <Anaconda module> --conda <CONDA-ENV-PATH>

buildtest will build test for python and any additional python packages that are installed in the conda environment.
For instance if the conda environment is the following $HOME/.conda/env/example this can be passed to option --conda as follows

buildtest -s Anaconda2/4.2.0 --conda $HOME/.conda/env/example

The test directory can be $BUILDTEST_TESTDIR/conda/$CONDA_ENV_PATH to ensure unique directory path when testing multiple conda environments

Each test will need to can be the following

#!/bin/sh
module purge
module load Anaconda2/4.2.0
source activate $HOME/.conda/env/example
<COMMAND>

Add option --show-keys to display description of yaml keys via command line. This would display information currently captured in http://buildtestdocs.readthedocs.io/en/latest/Writing_Test_In_YAML.html#yaml-key-description

buildtest should detect .pl extension and set run command to use perl

buildtest is creating a empty binary list which should not be written to file but rather be skipped.

description: Binary test for OpenBLAS/0.2.20
binaries: []

Ideas from other testing frameworks implementation:

• ref. https://bitbucket.org/michilu/shell-doctest
  • Labels (inside docstrings, nice)
  • Test per host case (think GROMACS against a 1 or 2-socket affinity, 4-gpu server)
• ref. https://github.com/brodie/cram
  • supporting different shells and contexts
  • Initialize variables
  • interactive training of tests

This could be a simple hello world program in YAML file that user may want to compile against different compilers

source: hello.c
testblock:
  test:
    name: hello_c_gcc
    compiler: gcc
    buildopts: -O2
    module:
    - gcc
    name: hello_c_gcc_system
    compiler: gcc
    buildopts: -O2
    module:
    - system
  test:
    name: hello_c_intel
    compiler: intel
    buildopts: -O3
    module:
    - intel
  test:
    name: hello_c_pgi
    compiler: pgi
    buildopts: -O1 
    module: 
    - pgi

The YAML file should create 4 different scripts that may look as follows

Script: hello_c_gcc

#!/bin/sh
module purge
module load gcc
gcc -O2 hello.c -o hello_c_gcc
./hello_c_gcc

Script: hello_c_gcc_system this will use system gcc and no modules will be loaded

#!/bin/sh
module purge
gcc -O2 hello.c -o hello_c_gcc_system
./hello_c_gcc_system

Script: hello_c_intel this will use intel module

#!/bin/sh
module purge
module load intel
icc -O3 hello.c -o hello_c_intel
./hello_c_intel

Script: hello_c_pgi this will use pgi module

#!/bin/sh
module purge
module load pgi
pgcc -O1 hello.c -o hello_c_pgi
./hello_c_pgi

A key used for setting up environment for a test script. This would be used for defining environment variables based on your shell.

Add support for email integration. This can be set in config.yaml as follows

BUILDTEST_EMAIL: True
EMAIL:
   - SENDER: [email protected]
   - FROM: [email protected]
   - SUBJECT: BUILDTEST ALERT

If BUILDTESDT_EMAIL: True is set buildtest will send email for test runs that fall below threshold (BUILDTEST_SUCCESS_THRESHOLD).

This command should work when running tests (_buildtest run)

Add support in buildtest to run STREAM benchmark see http://www.cs.virginia.edu/stream/ref.html

The STREAM benchmark can be run with buildtest by creating stream subparser in benchmark.

_buildtest benchmark stream

Instead of creating test scripts (.sh) create job scripts for LSF

Add a flag and provide a job template to alter job parameters.

Run a sanity check on each file in host system against for files provided by system package to ensure all files exist in system.