solvcon / devenv

Common development environment for solvcon

SOLVCON implements conservation-law solvers that use the space-time Conservation Element and Solution Element (CESE) method.

Install

Clone from https://github.com/solvcon/solvcon:

$ git clone https://github.com/solvcon/solvcon

SOLVCON needs the following packages: A C/C++ compiler supporting C++11, cmake 3.7+, pybind11 Git master, Python 3.6+, Cython 0.16+, Numpy 1.5+, LAPACK, NetCDF 4+, SCOTCH 6.0+, Nose 1.0+, Paramiko 1.14+, boto 2.29.1+, and gmsh 3+. Support for VTK is to be enabled for conda environment.

To install the dependency, run the scripts contrib/conda.sh and contrib/build-pybind11-in-conda.sh (they use Anaconda).

The development version of SOLVCON only supports local build:

$ make; python setup.py build_ext --inplace

To build SOLVCON from source code and install it to your system:

$ make; make install

Test the build:

$ nosetests --with-doctest
$ nosetests ftests/gasplus/*

Building document requires Sphinx 1.3.1+, pstake 0.3.4+, and graphviz 2.28+. Use the following command:

$ make -C doc html

The document will be available at doc/build/html/.

devenv's Issues

Support `devenv launch <application>`

Following by #23 and #22 (comment) , let's trace the feature enhancement with this ticket.

CI takes long time (5+ minutes)

The github action takes 9 minutes to finish which seems to be a bit too long. We may want to try to reduce it later.

Originally posted by @yungyuc in #1 (comment)

Support customized flavor directory path

If devevn supports customized flavor directory path, we could use this feature to package devenv as some more deliverable or conventional package for user to consume, e.g. debian package.

This idea is inspired by the online sprint preparation meeting. For more context, please refer to https://docs.google.com/document/d/1HrBPlR9nGoo3RPg7kZHaOn0azDtTZQ3qc9ZhIN-OV4c/edit?pli=1#

numpy build failed

• numpy version: f589407
• error:

  RuntimeError: Missing `SVML` submodule! Run `git submodule update --init` to fix this.
  

Upgrade mac CI to use version newer than 10.15

It seems github actions do not have sufficient runners for mac 10.15 any more.

Install zlib using tarball

A while back I had a discussion (probably with @j8xixo12) for whether to use tarball for installing zlib. I did not remember what was the conclusion, but for now we still pull down the source code from git:

devenv/scripts/build.d/zlib

Line 9 in fb0e494

syncgit https://github.com/madler ${pkgname} ${pkgbranch}

I think using tarball is a more efficient and stable approach for building it and we should make it the default for the package.

[sprint][210712] add unittest for scripts/env.d/devenv_impl:namemunge()

Add GNU system make to build source

Since all the build source using the GNU make command.

When devenv running on clean system, the following message will show up.

log file: devenv/flavors/demo/src/openssl-master/configure.log
run command: make -j 6
devenv/scripts/func.d/build_utils: line 3: make: command not found

Instead of #25, I decide to build GNU make from source.

`devenv list` returns incorrect return code

If there is any existing flavor, devenv list should return 0 instead of non-0.

┌─[^_^]─[~/work-my-projects/solvcon/devenv]
  tai271828@syakaro: 6 file 48Kb
└──╼ devenv list
prime
┌─[O_o]─[~/work-my-projects/solvcon/devenv]
  tai271828@syakaro: 6 file 48Kb
└──╼ echo $?
87

[sprint][210712] add unittest for scripts/func.d/bash_utils:get_list()

Build script for silver searcher (ag)

Add a build script for ag

Use github template

Github provides a very useful feature to build a default issue and pull request template by committing the template into the source repository under .github. It's very useful. Please thumbs up if you would like to use the feature. If yes, I am glad to create the template.

Or, we may defer the usage until the project and community grow, and issues need to be managed in such way.

[solvcon] CI fails (for numpy version, PEP400, and build failure of scotch git branch)

We introduce #97 for #92 , and I believe the pull request fails SOLVCON built by devenv in the Action runs manually triggered by me again today:

• https://github.com/solvcon/devenv/runs/4594808783?check_suite_focus=true
• https://github.com/solvcon/solvcon/runs/4594813592?check_suite_focus=true

Move the script files outside the root directory

I don't think we should leave script files in the root directory.

Also, we would need a simple document to tell users how to use the tool.

Build script for vim (only text-mode is needed)

Need a build script for a new version of vim.

Upgrade github action test runner to newer OS

Currently the github action runners are using Ubuntu 20.04 and macos 10.5 which are old. We should determine what versions to upgrade to and whether or not to keep the older versions.

Check environment effectiveness after enabling

I assume after enable environment, we can achieve some isolate build environment,
but in README usage we cannot know the effectiveness of this environment.

(e.g. poetry run python --version can check Python version difference between system python & poetry shell python).

We should add a PoC after enable environment make new user know what happen.

This repo should not depend on the SOLVCON repo

Currently SOLVCON repo is cloned in prepare-solvcon-dev.sh line #19 to make the script working. So the SOLVCON repo circularly depends on this devenv.

We should not clone the SOLVCON repo for installing packages.

Enable ubuntu-20.04 and macos-11 .0 in CI

The current CI OS matrix (https://github.com/solvcon/devenv/blob/main/.github/workflows/solvcon_install.yml#L10) includes ubuntu-18.04 and macos-latest (which is an alias to macos-10.15). I would like it to include two more versions: ubuntu-20.04 and macos-11.0, which are beyond the latest aliases. The current macos-latest should be renamed to macos-10.15.

See the github hosted runners for more information.

Add an application for building modmesh on Taiwania1

The directory applications/ houses scripts for building development environment for specific repository and farms. Create a new sub-directory modmesh/ under it and include the build scripts for building modmesh on Taiwania1.

how to execute init?

(base) -virtual-machine:/devenv-main/scripts$ source init
bash: source: /usr/sbin/init: cannot execute binary file
(base) -virtual-machine:/devenv-main/scripts$

No specific error message when deleting non-existing environment

It confused me at the first time.

$ devenv list
modmesh
$ devenv del modesh
removing 'modesh' dev environment

Bump CI from Ubuntu 18.04 to 20.04 for application SOLVCON

[sprint][210712] add unittest for scripts/env.d/devenv_impl:nameremove()

[app][solvcon] build-pybind11-in-conda.sh may be redundant

Heads-up from #86 (review) . We will use this issue as the story tracker.

Support download and extract tarball

There are some situations that the computer or server not allow to access internet, user only can upload file by themself, or just easy for install prebuilt binary.
If devenv can support not only download by git but also tarball, and automatic check whether the package has been downloaded before, if yes, then just untar it, it would be helpful!!

python build failed

• gcc version: 12.2.1 20230201
• python version: 3.11.1
• error message:

  ...
  gcc -L/home/chestercheng/.devenv/flavors/tmp/usr/lib -Wl,-rpath=/home/chestercheng/.devenv/flavors/tmp/usr/lib,--no-as-needed -L/home/chestercheng/.devenv/flavors/tmp/usr/lib -Wl,-rpath=/home/chestercheng/.devenv/flavors/tmp/usr/lib,--no-as-needed -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate -o Programs/_freeze_module Programs/_freeze_module.o Modules/getpath_noop.o Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o     Modules/config.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre/sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o -ldl                         -lm
  lto1: internal compiler error: original not compressed with zstd
  0x175af38 internal_error(char const*, ...)
          ???:0
  0x9c849e lto_end_uncompression(lto_compression_stream*, lto_compression)
          ???:0
  0x9c2bca lto_get_section_data(lto_file_decl_data*, lto_section_type, char const*, int, unsigned long*, bool)
          ???:0
  0x6e9c2d cgraph_node::get_untransformed_body()
          ???:0
  0x6f03f3 cgraph_node::expand()
          ???:0
  0x654cf6 lto_main()
          ???:0
  Please submit a full bug report, with preprocessed source (by using -freport-bug).
  Please include the complete backtrace with any bug report.
  See <https://bugs.archlinux.org/> for instructions.
  lto-wrapper: fatal error: gcc returned 1 exit status
  compilation terminated.
  /usr/bin/ld: error: lto-wrapper failed
  collect2: error: ld returned 1 exit status
  make: *** [Makefile:1196: Programs/_freeze_module] Error 1
  ...
  

`launch` does not fully use pre-define `$DEVENVFLAVOR`

Description

devenv launch solvcon does not use $DEVENVFLAVOR and use the hard-coded flavor name solvcon-dev.

Steps to Reproduce

1. devenv launch solvcon. devenv will complain $DEVENVFLAVOR not defined.
2. devenv add flavor1
3. devenv launch solvcon
4. devenv show

Expected Result

Only flavor1 is listed.

Actual Result

Both flavor1 and solvcon-devenv are listed.

Enhancement of `devenv show`

This issue tracks the follow-up enhancement of #35 . They could be:

1. autocomplete in scripts/init/_devenv
2. bin/devenv for the usage of a new command

[solvcon] find a way to resolve numpy support of extra_c_compile_args in build_ext.py

When working on #98 , we realized that the latest numpy seems to change its distribution approach so
numpy/distutils/command/build_ext.py does not support
extra_c_compile_argsin the way we expected to cause the following error:

    /usr/lib/python3.8/site-packages/numpy/distutils/command/build_ext.py", line 380,
    in build_extension extra_cflags = ext.extra_c_compile_args or [] AttributeError:
    'Extension' object has no attribute 'extra_c_compile_args'
    make: *** [Makefile:84: legacy] Error 1                                           

We need to find a way to resolve this issue so the whole SOLVCON build stack could enjoy the latest numpy in the end.

[macos][solvcon] libbz2.so.1.0.8: No such file or directory and download the tarball by default

I was trying to build SOLVCON with the application feature of devenv[1]. The build looks good in Ubuntu, but failed to find the pre-built bz2 lib which was also built by devenv. For example, see this CI result https://github.com/tai271828/solvcon/runs/4092547398?check_suite_focus=true

I don't have a macOS, but I believe this issue could be reproduced by devenv launch solvcon, or devenv build bzip2 and then check the library names under the devenv flavor folder.

[1] mainly for testing this landed pull request as well #91

`devenv show` is missing usage descripion

See #38 (comment)

devenv cannot be initiated under zsh on Intel macbook

Enviroment: macOS Big Sur 11.2.3

Shell: zsh

Terminal output:
zhxie@Zong-HandeMacBook-Pro .devenv % pwd
/Users/zhxie/.devenv
zhxie@Zong-HandeMacBook-Pro .devenv % source scripts/init
scripts/init:.:3: no such file or directory: /Users/zhxie/scripts/env.d/vars
scripts/init:64: command not found: complete

Add a command to amalgamate one or multiple build scripts

The build script files under scripts/build.d/ cannot run without the full source tree of devenv. Therefore we cannot build the dependency packages by simply copying the build scripts.

I would like to propose to add an enhancement feature to amalgamate a build script with the shared bash code. By amalgamation, it is like how rapid yaml creates a single header file or how sqlite creates the single C file. But for devenv the amalgamation creates a single bash script rather than a source file for compiler.

With the amalgamation, we may replace the hand-crafted build script with a devenv-generated one.

Build script for tmux

Need a build script for tmux. It should use tarball.

[SOLVCON] Consolidate the CI code

Placeholder of solvcon/solvcon#250

[sprint][210712] add unittest for scripts/func.d/bash_utils:display()

Enable CI with mac-10.15 instance

context #52 (comment)

[Feature request] devenv launch modmesh

• Ubuntu Jammy has shipped with Qt6 needed for building modmesh
• Ubuntu Focal could build modmesh smoothly by installing third-party Qt libraries from Qt's website

I think it is time to make the feature of devenv launch modmesh.

macOS fails to build scotch with git branch but succeeds to build with tarball

When working on #98 , which is raised by multiple causes. I realized one of the failure root causes is macOS fail to build scotch with its git branch when succeeding to build with scotch tarball.

You may refer to the result of the CI job run with my branch https://github.com/tai271828/devenv/actions/runs/1766863747

CI may fail

The CI of my personal forked repository failed https://github.com/tai271828/devenv/runs/6630300980?check_suite_focus=true so I am aware that the scheduled CI of devenv did not run for a while because Github automatically disabled it for no activity for a while in this repository.

I checked the failure of the CI job https://github.com/tai271828/devenv/runs/6630300980?check_suite_focus=true and I believe devenv's official CI should fail as well. I have re-enabled the Github action for devenv again now. Let's see how it will go.

FWIW: the failure seems some dependency version failure, maybe cmake.

python 3.8 built on macOS does not function correctly

When building SOLVCON with python 3.8, the corresponding CI works well on Ubuntu, but not macOS.

The error messages shows

/Users/runner/work/solvcon/solvcon/devenv/scripts/build.d/python: line 72: /Users/runner/work/solvcon/solvcon/devenv/flavors/prime/usr/bin/python3: No such file or directory

Step to Reproduce

I don't have a mac so I did not try practically, but I believe this could be reproduce by simply executing devenv build python:

1. Create a devevn flavor from scratch as usual. For example, prime by devenv add prime and then devenv use prime
2. devenv build python
3. which python3

Expected Result

The real path of python3 is shown and it's a copy of, or a symbolic link to python 3.8 binary built by devenv.

Actual Result

This SOLVCON CI job shows python3 is unavailable.

gmsh bin is not found when using macos

I am recently aware of this issue via some SOLVCON CI runs, for example: https://github.com/tai271828/solvcon/runs/3440179913?check_suite_focus=true

Steps to Reproduce

1. Create a devevn flavor from scratch as usual. For example, prime by devenv add prime and then devenv use prime
2. VERSION=gmsh_3_0_6 devenv build gmsh
3. which gmsh

Expected Result

The real path of gmsh binary which will be used is shown.

Actual Result

Nothing is found (return none)

Fail to run the test driver of devenv

The output message are listed below:

*** test file: test_init_var.sh
test_devenvroot
test_devenvflavor
test_devenvflavorroot
test_devenvcurrentroot
test_devenvlibrary_path_backup
test_devenvpath_backup
test_devenvprefix
test_devenvdlroot

Ran 8 tests.

OK
*** test file: test_bash_utils.sh
test_display_type
ASSERT:expected: but was:<>
shunit2:ERROR test_display_type() returned non-zero return code.
test_display
test_display_e
devenv shunit2 unittest
test_get_cmd_list
test_get_build_list
test_get_application_list
test_get_flavor_list

Ran 7 tests.

FAILED (failures=2)
*** test file: test_devenv_impl.sh
test_namemunge
test_nameremove

Ran 2 tests.

OK

---

There is a conflict between the function 'display()' of devenv and the shell command /usr/bin/display.
The display is a command provided by the package 'imagemagick'.
Possibly we should change the function name of 'display()' of devenv.

Build script for git

Need a build script for git when the host does not provide latest git installation.

Discuss the use of miniconda for installing dependencies

This script uses miniconda for most of the dependencies so I think it's probably good to highlight it in the filename. We may do it with this PR or act after more discussions.

Originally posted by @yungyuc in #1 (comment)

Preparing slide for devenv build experience

Non-English based system languages cause the failure of running the test driver

My current system default language is traditional Chinese.
By running the unit test of devenv, I would have the following output messages:

---

*** test file: test_init_var.sh
test_devenvroot
test_devenvflavor
test_devenvflavorroot
test_devenvcurrentroot
test_devenvlibrary_path_backup
test_devenvpath_backup
test_devenvprefix
test_devenvdlroot

Ran 8 tests.

OK
*** test file: test_bash_utils.sh
test_display_type
ASSERT:expected:<denenv_display is a function> but was:<>
shunit2:ERROR test_display_type() returned non-zero return code.
test_display
test_display_e
devenv shunit2 unittest
test_get_cmd_list
test_get_build_list
test_get_application_list
test_get_flavor_list

Ran 7 tests.

FAILED (failures=2)
*** test file: test_devenv_impl.sh
test_namemunge
test_nameremove

Ran 2 tests.

OK

---

By further checking the error message, we could know the failure was from test function 'test_display_type()' in tests/test_bash_utils.sh

---

test_display_type() {
expected=display is a function"
actual=$(type display | grep "is a")
assertEquals "${expected}" "${actual}"
}

---

This function tried to check the output of the following command:
$ type display

The expected output message requested by the unit test should be 'display is a function'

Since my system default language was traditional Chinese, the output message would be 'display 是一個函數'

So the unit test would be failed due to non-English system language.