buildtesters / buildtest Goto Github PK
View Code? Open in Web Editor NEWHPC System and Software Testing Framework
Home Page: https://buildtest.rtfd.io
License: MIT License
HPC System and Software Testing Framework
Home Page: https://buildtest.rtfd.io
License: MIT License
provide an easy installation method by letting pip install buildtest and its components.
Seems to be a bug when $BUILDTEST_TESTDIR/ebapp is not created in advance.
(buildtest-0.5.0) [siddis14@adwnode1 buildtest-framework]$ _buildtest build --system acl
Traceback (most recent call last):
File "/home/siddis14/.conda/envs/buildtest-0.5.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/siddis14/.conda/envs/buildtest-0.5.0/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/siddis14/github/buildtest-framework/buildtest/main.py", line 43, in <module>
from buildtest.tools.menu import buildtest_menu
File "/home/siddis14/github/buildtest-framework/buildtest/tools/menu.py", line 49, in <module>
class buildtest_menu():
File "/home/siddis14/github/buildtest-framework/buildtest/tools/menu.py", line 67, in buildtest_menu
app_choices = run_app_choices()
File "/home/siddis14/github/buildtest-framework/buildtest/test/run/app.py", line 41, in run_app_choices
app_name_list = [ f.path for f in os.scandir(app_root_testdir) if f.is_dir()]
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/buildtest-tests/ebapp'
Add support for csh, tcsh, zsh, ksh.
Any file extension with .rb should be processed by ruby wrapper
Add flag --scan-Python-packages, --scan-R-packages, and --scan-Perl-packages to report list of tests found in buildtest against the packages installed in Python, Perl and R.
This will be useful to provide users information on what tests can be generated from buildtest repos for R, Python, and Perl. Currently all tests are generated and behavior should be done on per package basis if found in the system.
Permutations for each build opts to see which set of operations result in the best mechanism for building software.
Add support for command history support similar to yum history command in linux. This will only show history of buildtest commands that may be useful to see track of all buildtest commands. The following options could be as follows
buildtest history --build
and short option buildtest history -b
for showing history of builds
buildtest history --run
and short option buildtest history -r
for showing history of runs
buildtest history --all
and short option buildtest history -a
for showing history of all buildtest commands
buildtest can write all buildtest command to file (i.e /tmp/buildtest/history/build.txt
, /tmp/buildtest/history/run.txt
, /tmp/buildtest/history/all.txt
). This can be controlled by configuration variable BUILDTEST_HISTORY_DIR
or environment variable that is set to /tmp/buildtest/history
.
Whenever any buildtest command is executed it will write to /tmp/buildtest/history/all.txt
and it may run write to build.txt
or run.txt
if buildtest builds or runs a test
This file will append any new command to this file and it can be viewed by reading the file or running the buildtest history
command.
The output format could look something like this for /tmp/buildtest/history/all.txt
ID | COMMAND
----|------------------------
1 | buildtest -V
2 | buildtest show -c
3 | buildtest --help
4 | buildtest build -S compilers
buildtest could leverage the history concept to rerun commands something like buildtest history --id <id> --type <all | build | run>
where --id is the numerical index and --type is the different files. To rerun the first command it could be buildtest history --id 1 --type all
. There could be a short option -i
and -t
for these commands
Python yaml config and code directory needs to be structured by python package. Create a python package directory where the code and yaml file will reside.
Directory structure for numpy
Python/code/numpy/
Python/config/numpy
The test scripts generated by buildtest should be in format Python/pyver/toolchain-name/toolchain-ver/numpy/example.py
Create a CMakeLists for each python package
(buildtest) [siddis14@adwnode1 buildtest-framework]$ _buildtest --logdir=/tmp build -s GCCcore/6.4.0
Detecting Software: GCCcore/6.4.0
--------------------------------------------
[STAGE 1]: Building Binary Tests
--------------------------------------------
Detecting Test Type: Software
Processing Binary YAML configuration: /home/siddis14/buildtest-configs/buildtest/ebapps/gcccore/6.4.0/command.yaml
Generating 10 binary tests
Binary Tests are written in /tmp/buildtest-tests/ebapp/GCCcore/6.4.0/
--------------------------------------------
[STAGE 2]: Building Source Tests
--------------------------------------------
Processing all YAML files in directory: /home/siddis14/buildtest-configs/buildtest/ebapps/gcccore/config
Generating 22 Source Tests and writing at /tmp/buildtest-tests/ebapp/GCCcore/6.4.0/
Writing Log file: /tmp/buildtest/GCCcore/6.4.0/buildtest_16_46_03_10_2018.log
(buildtest) [siddis14@adwnode1 buildtest-framework]$ _buildtest --show
buildtest configuration summary
(C): Configuration File, (E): Environment Variable
BUILDTEST_ROOT (E) = /home/siddis14/github/buildtest-framework
BUILDTEST_CLEAN_BUILD (C) = False
BUILDTEST_CONFIGS_REPO (C) = /home/siddis14/buildtest-configs
BUILDTEST_CONFIGS_REPO_SOFTWARE (C) = /home/siddis14/buildtest-configs/buildtest/ebapps
BUILDTEST_CONFIGS_REPO_SYSTEM (C) = /home/siddis14/buildtest-configs/buildtest/system
BUILDTEST_ENABLE_JOB (C) = False
BUILDTEST_IGNORE_EASYBUILD (C) = False
BUILDTEST_JOB_TEMPLATE (E) = /home/siddis14/github/buildtest-framework/template/job.slurm
BUILDTEST_LOGDIR (C) = /tmp/buildtest
BUILDTEST_MODULE_NAMING_SCHEME (C) = FNS
BUILDTEST_MODULE_ROOT (C) = /clust/app/easybuild/2018/Broadwell/redhat/7.3/modules/all
BUILDTEST_PERL_REPO (C) = /home/siddis14/github/Perl-buildtest-config
BUILDTEST_PERL_TESTDIR (C) = /home/siddis14/github/Perl-buildtest-config/buildtest/perl/code
BUILDTEST_PYTHON_REPO (C) = /home/siddis14/github/Python-buildtest-config
BUILDTEST_PYTHON_TESTDIR (C) = /home/siddis14/github/Python-buildtest-config/buildtest/python/code
BUILDTEST_RUBY_REPO (C) = /home/siddis14/github/Ruby-buildtest-config
BUILDTEST_RUBY_TESTDIR (C) = /home/siddis14/github/Ruby-buildtest-config/buildtest/ruby/code
BUILDTEST_R_REPO (C) = /home/siddis14/github/R-buildtest-config
BUILDTEST_R_TESTDIR (C) = /home/siddis14/github/R-buildtest-config/buildtest/R/code
BUILDTEST_SHELL (C) = sh
BUILDTEST_TESTDIR (C) = /tmp/buildtest-tests
--logdir not overriding path. Need to fix this
[siddis14@adwnode1 ~]$ pipenv install buildtest-framework
Installing buildtest-framework...
Collecting buildtest-framework
Using cached https://files.pythonhosted.org/packages/8c/23/1b082de48a413bd39c651c159d35b602158a25b8a7db4065517875d2e1fe/buildtest-framework-0.4.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-sdnh3zof/buildtest-framework/setup.py", line 29, in <module>
from buildtest.tools.config import BUILDTEST_VERSION
File "/tmp/pip-install-sdnh3zof/buildtest-framework/buildtest/tools/config.py", line 34, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
----------------------------------------
Error: An error occurred while installing buildtest-framework!
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-sdnh3zof/buildtest-framework/
This is likely caused by a bug in buildtest-framework. Report this to its maintainers.
Create a submenu hpl under benchmark that will be able to run the HPL benchmark.
_buildtest benchmark hpl
Create a YAML framework for HPL benchmark that create a test script to run HPL. Preferably this test should be run via job scheduler
buildtest should detect .R files and set compiler to either R CMD BATCH or Rscript
any .lua extension should be set processed by lua wrapper
Some tests are expected to fail (try/catch, Exception) to test functionality so need a way to provide expected exit status to see if test succeeded for these types of tests.
name: example
source: example.c
exit_code: 1
The --testset
option can be replaced by the following options
Python: --python-package
R: --r-package
Perl: --perl-package
Ruby: --ruby-package
The command below will attempt to build tests for all python packages present in software and found in Python-buildtest-config repo
buildtest -s Python/X.Y.Z --python-package
buildtest can auto-populate choice selection for option --python-package
so that user may do the following
buildtest -s Python/X.Y.Z --python-package numpy,scipy,tensorflow
This same concept can be extended for other options.
Each job run by buildtest should be able to control process through an option controlled by buildtest or inside the test script.
If option is set in test script, user would be interested testing same code with different jobslot configuration so an option to dynamically create tests with different job slot would be nice.
Create a submenu hpcg under benchmark that will run HPCG benchmark.
_buildtest benchmark hpcg
Create a YAML framework for hpcg benchmark that create a test script to run hpcg . Preferably this test should be run via job scheduler
buildtest variants could enhance test scripts by writing a different test for each permutation initial support can start out by doing flag variants to build test with different compiler flags
testblock: singlesource
test:
source: hello.cpp
compiler: gnu
variants:
flags:
- O3
- O2
- O1
module:
- gcc
A test variant for compilers may look as follows for a OpenMP example
testblock: singlesource
test:
source: omp_mm.c
vars:
OMP_NUM_THREADS: 4
variants:
compiler: gnu
ldflags: "-fopenmp"
module:
- gcc
compilers: intel
ldflags: "-qopenmp"
module:
- intel
compiler: pgi
ldflags: "-mp"
module:
- pgi
To compile a openmp example with gcc, intel and pgi it will require different flags.
gnu
: gcc -fopenmp omp_mm.c -o omp_mm
intel
: icc -qopenmp omp_mm.c -o omp_mm
pgi
: pgi -fopenmp omp_mm.c -o omp_mm
buildtest can create one test script per variant the naming scheme could be as simple as
omp_mm.c.yml.variant_<first 15 digits of sha256>.sh
where omp_mm.c.yml is the yml file https://github.com/HPC-buildtest/buildtest-configs/blob/master/buildtest/suite/openmp/matrixmultiplication/omp_mm.c.yml to ensure unique names for test scripts
Need to present buildtest logging in a cleaner method for troubleshooting. Currently too much text is crammed together. Need to pick and chose what should be logged and to what level.
https://www.python.org/dev/peps/pep-0008/
https://www.python.org/dev/peps/pep-0257/
https://www.python.org/dev/peps/pep-3101/
For more details on string formatting check out https://realpython.com/python-f-strings/ and https://pyformat.info/
Need to implement a method to detect OS to allow buildtest to create system package test for any linux distribution. Currently it supports Fedora distribution only
buildtest should present its test cases in CDASH (a testing framework dashboard) see https://www.cdash.org/
This would be helpful to present users with a visual overview of all the tests.
(buildtest-0.5.0) [siddis14@adwnode1 buildtest-framework]$ _buildtest --sysyaml apr
Traceback (most recent call last):
File "/home/siddis14/.conda/envs/buildtest-0.5.0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/siddis14/.conda/envs/buildtest-0.5.0/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/siddis14/github/buildtest-framework/buildtest/main.py", line 135, in <module>
main()
File "/home/siddis14/github/buildtest-framework/buildtest/main.py", line 125, in main
create_system_yaml(bt_opts.sysyaml)
File "/home/siddis14/github/buildtest-framework/buildtest/tools/generate_yaml.py", line 57, in create_system_yaml
binary = get_binaries_from_systempackage(name)
File "/home/siddis14/github/buildtest-framework/buildtest/tools/system.py", line 92, in get_binaries_from_systempackage
print ("There are no binaries found in package: %s", name)
NameError: name 'name' is not defined
See build https://readthedocs.org/projects/buildtest/builds/8669213/
The issue is importing buildtest modules in readthedocs due to environment setup
Getting errors like
WARNING: autodoc: failed to import module 'benchmark' from module 'buildtest.benchmark'; the following exception was raised:
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/latest/lib/python3.7/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
__import__(modname)
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/src/buildtest/benchmark/benchmark.py", line 24, in <module>
from buildtest.benchmark.osu import run_osu_microbenchmark, list_osu_tests, osu_info
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/src/buildtest/benchmark/osu.py", line 31, in <module>
from buildtest.tools.config import config_opts
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/src/buildtest/tools/config.py", line 51, in <module>
DEFAULT_CONFIG_FILE = os.path.join(os.getenv("BUILDTEST_ROOT"),"settings.yml")
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/latest/lib/python3.7/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
This causes the documentation for https://buildtest.readthedocs.io/en/latest/command_reference.html to not show any buildtest methods.
When building documentation locally the command reference page is generated properly.
By default buildtest will clean the root of software test directory to avoid duplicate test names added in CMakeList.txt
.
buildtest can mimic this same behavior with option --rebuild
which will clean the test directory before writing test and the default behavior will be to not delete the root of software test directory. This may be useful if user is running multiple buildtest commands to generate test for Python packages or rebuild test with different shell.
buildtest should provide feature to run HPC benchmarks using buildtest --benchmark which can be used for generating test-scripts that can be later run via CTest or job scheduler. Initially will focus on MPI benchmarks to get performance metrics with various processor configuration.
buildtest should automatically detect # of cores in system and determine permutation and generate all the test scripts.
Tests can be generated in $BUILDTEST_TESTDIR/benchmarks/$benchmark-name/
Test scripts should encode processor configuration to distinguish test cases with different node/ppn parameters. Test scripts can follow the following naming format:
$TESTNAME_$PPNx$NODE.sh
For instance the osu_allreduce from OSU benchmark can be named as follows
PPN = 1 Node = 1 osu_allreduce_1x1.sh
PPN = 2 Node = 1 osu_allreduce_2x1.sh
PPN =2 Node = 3 osu_allreduce_2x3.sh
To run OSU benchmark it can be done by running the following
buildtest --benchmark=osu
To run STREAM do the following
buildtest --benchmark=stream
Options to --benchmark will be name of benchmark name that should have separate python module to initialize the environment necessary to build the benchmark, whether it is precompiled binary or needs to be build from source.
buildtest currently supports Hierarchical Module Naming Scheme, need to have option in buildtest to process modules based on module naming scheme to generate header information in test script correctly.
The testblock: python2
should be a class that will parse this yaml configuration and return a test script to run Python 2.x code.
A similar testblock can be implemented for python3 where pip, virtualenv will be compatible for python3 and assuming python script that is to be run is python 3 compatible.
The configuration below could have support for virtual environment as yaml keys to allow user to configure the virtual environment before running the python code.
testblock: python2
source: random.py
pyver: 2.6, 2.7, 2.7.10, 2.7.15
virtualenv: yes
virtualenvopts:
name: venv1
download: yes
no-site-packages: yes
python: /usr/bin/python
pythonpackages:
- numpy
- scipy
testblock:
name: random
The script will be named random.sh
with the following content
#!/bin/sh
virtualenv --download --no-site-packages -p /usr/bin/python venv1
source $HOME/venv1/bin/activate
pip install numpy scipy
python random.py
This feature will report all the tests that can be run based on the available software found in the system and all the YAML configs found in buildtest.
The user will get a report for all the packages and # of tests for each package. This will include system package test by checking all YAML configs in system package and installed system package found in the system by using rpm -q
buildtest should create test scripts for system packages to automate test. Create a flag --systempkg to generate test for systempkg.
ability to create YAML config for binary test for system & easybuild packages. This way tests configs can be generated very quickly.
buildtest should have an option for generating tests based on cpu affinity for all applications. This would be particularly useful for users in NUMA architecture to fine-tune their tests based on cpu topology
Allow users to submit test their conda environment
buildtest -s <Anaconda module> --conda <CONDA-ENV-PATH>
buildtest will build test for python and any additional python packages that are installed in the conda environment.
For instance if the conda environment is the following $HOME/.conda/env/example
this can be passed to option --conda
as follows
buildtest -s Anaconda2/4.2.0 --conda $HOME/.conda/env/example
The test directory can be $BUILDTEST_TESTDIR/conda/$CONDA_ENV_PATH
to ensure unique directory path when testing multiple conda environments
Each test will need to can be the following
#!/bin/sh
module purge
module load Anaconda2/4.2.0
source activate $HOME/.conda/env/example
<COMMAND>
Add option --show-keys
to display description of yaml keys via command line. This would display information currently captured in http://buildtestdocs.readthedocs.io/en/latest/Writing_Test_In_YAML.html#yaml-key-description
buildtest should detect .pl extension and set run command to use perl
buildtest is creating a empty binary list which should not be written to file but rather be skipped.
description: Binary test for OpenBLAS/0.2.20
binaries: []
Ideas from other testing frameworks implementation:
This could be a simple hello world program in YAML file that user may want to compile against different compilers
source: hello.c
testblock:
test:
name: hello_c_gcc
compiler: gcc
buildopts: -O2
module:
- gcc
name: hello_c_gcc_system
compiler: gcc
buildopts: -O2
module:
- system
test:
name: hello_c_intel
compiler: intel
buildopts: -O3
module:
- intel
test:
name: hello_c_pgi
compiler: pgi
buildopts: -O1
module:
- pgi
The YAML file should create 4 different scripts that may look as follows
Script: hello_c_gcc
#!/bin/sh
module purge
module load gcc
gcc -O2 hello.c -o hello_c_gcc
./hello_c_gcc
Script: hello_c_gcc_system
this will use system gcc and no modules will be loaded
#!/bin/sh
module purge
gcc -O2 hello.c -o hello_c_gcc_system
./hello_c_gcc_system
Script: hello_c_intel
this will use intel module
#!/bin/sh
module purge
module load intel
icc -O3 hello.c -o hello_c_intel
./hello_c_intel
Script: hello_c_pgi
this will use pgi module
#!/bin/sh
module purge
module load pgi
pgcc -O1 hello.c -o hello_c_pgi
./hello_c_pgi
A key used for setting up environment for a test script. This would be used for defining environment variables based on your shell.
Add support for email integration. This can be set in config.yaml as follows
BUILDTEST_EMAIL: True
EMAIL:
- SENDER: [email protected]
- FROM: [email protected]
- SUBJECT: BUILDTEST ALERT
If BUILDTESDT_EMAIL: True
is set buildtest will send email for test runs that fall below threshold (BUILDTEST_SUCCESS_THRESHOLD
).
This command should work when running tests (_buildtest run
)
Add support in buildtest to run STREAM benchmark see http://www.cs.virginia.edu/stream/ref.html
The STREAM benchmark can be run with buildtest by creating stream subparser in benchmark.
_buildtest benchmark stream
Instead of creating test scripts (.sh) create job scripts for LSF
Add a flag and provide a job template to alter job parameters.
Run a sanity check on each file in host system against for files provided by system package to ensure all files exist in system.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.