fcmaes.optimizer.de_cma throws SystemError and TypeError about fast-cma-es

[dietmarwo]

It is probably OS related. Which OS do you use? If it is linux which gcc version is installed? Do you see errors when you execute pytest after cloning the github project:

git clone https://github.com/dietmarwo/fast-cma-es.git
cd fast-cma-es
pytest

Tested under Windows 10 and Ubuntu and cannot reproduce these errors on these OS.
Is your fitness function in Python?

I googled "SystemError: null argument to internal routine" and indeed it is related to some error at the C++ side. But this could be
related to some incompatible system libraries. On linux you could try to compile the C++ code yourself to make sure everything fits.

[dietmarwo]

Got reports about dll recursive dependency issues on windows. The existing dlls are compiled using mingw64. All required dlls to load these are in the anaconda distribution for windows at C:\Users\uname\anaconda3\Library\mingw-w64\bin so I didn't notice the dependency issue. In version 1.2.21 these mingw64 dlls are included now. Try a 'pip install --upgrade' in case you observe a dll loading issue. Unfortunately the "SystemError: null argument to internal routine is still not reproducible.

[timjim333]

Hi @dietmarwo sorry for being slow getting back to you - it's going to be pretty busy for 1.5 weeks, so I won't have a chance to debug properly until then, but in answer to your previous questions:

OS: CentOS 8
gcc: 8.3.1 20191121 (Red Hat 8.3.1-5)

N.B. glibc shipped with this system was too old, so I had to recompile the files in fast-cma-es/_fcmaescpp. Incidentally, I couldn't figure out the libgtop dependency, so I didn't compile that one (though it only looked like it was needed for astro.py). I then copied _fcmaescpp/libacmalib.so to fcmaes/lib.

I'm not really sure where to start looking to debug the SystemError - I tried placing print lines at various places in the code but I couldn't figure out what was throwing the error!

I tried running pytest and it returned a lot of output plus this failure summary at the end:

============================================================================================== FAILURES ===============================================================================================
________________________________________________________________________________________ test_eggholder_retry _________________________________________________________________________________________

    def test_eggholder_retry():
        dim = 2
        testfun = Eggholder()
    
        limit = -956
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = retry.minimize(wrapper.eval, testfun.bounds,
                                 num_retries=100)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
>       assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
E       assert 508093 == 609009
E        +  where 508093 =      fun: -959.640662720851\n    nfev: 508093\n success: True\n       x: [512.0, 404.23180522114546].nfev
E        +  and   609009 = <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7fbf469c5790>>()
E        +    where <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7fbf469c5790>> = <fcmaes.testfun.Wrapper object at 0x7fbf469c5790>.get_count

fcmaes/test_cma.py:304: AssertionError
____________________________________________________________________________________ test_eggholder_advanced_retry ____________________________________________________________________________________

    def test_eggholder_advanced_retry():
        dim = 2
        testfun = Eggholder()
    
        limit = -956
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = advretry.minimize(wrapper.eval, testfun.bounds,
                                    num_retries=300)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
>       assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
E       assert 819087 == 835557
E        +  where 819087 =      fun: -959.640662720851\n    nfev: 819087\n success: True\n       x: [512.0, 404.2318051133138].nfev
E        +  and   835557 = <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7fbf46ab1b90>>()
E        +    where <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7fbf46ab1b90>> = <fcmaes.testfun.Wrapper object at 0x7fbf46ab1b90>.get_count

fcmaes/test_cma.py:321: AssertionError
======================================================================================= short test summary info =======================================================================================
FAILED fcmaes/test_cma.py::test_eggholder_retry - assert 508093 == 609009
FAILED fcmaes/test_cma.py::test_eggholder_advanced_retry - assert 819087 == 835557
============================================================================== 2 failed, 12 passed in 132.52s (0:02:12) ===============================================================================

[dietmarwo]

Hi @timjim333 , sorry I didn't monitor this issue for some time, I am too busy with other stuff
( https://kelvins.esa.int/space-debris-the-origin/leaderboard/ ) and adding multi-objective / constraints to this library.
You guessed correctly, libgtop is only required for testing, I didn't include the source because of licensing issues.
Something like maven / pip would be nice for C++, so that we just have to define references to dependencies.

At least you have 12 tests passed. The "retry.minimize" is concerning since parallel retry is an essential feature.
I tested with gcc-7 and gcc-9 - had to switch because of compilation issues with biteopt on gcc-7. Tested on Ubuntu/Mint and Windows 10. I tried gcc-8 only on Windows, there it is the actual mingw version. Could you retry with the newest fcmaes version? Is it possible to update your glibc version so that you can check the original shared library without recompiling?
If not I probably have to install CentOS in VMware to reproduce your issue. Which Python version do you use? I test with Python 3.7 and 3.8. Could be a Python multiprocessing issue.

[dietmarwo]

For the version 1.3.4 I added all dependent shared libraries both for windows and linux in the fcmaes/lib folder.
Should help avoiding issues with wrong versions of the dependent libraries.

[timjim333]

Nice work on kelvins.esa.int!

I tried again pulling the latest version of fcmaes but on pytest, I still have the issue with glibc. Unfortunately, CentOS doesn't ship with a newer one:

/opt/fcmaes/fast-cma-es$ pytest
================================================= test session starts ==================================================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /opt/fcmaes/fast-cma-es
collected 0 items / 1 error                                                                                            

======================================================== ERRORS ========================================================
_________________________________________ ERROR collecting fcmaes/test_cma.py __________________________________________
fcmaes/test_cma.py:11: in <module>
    from fcmaes import cmaes, de, decpp, cmaescpp, gcldecpp, retry, advretry
fcmaes/decpp.py:163: in <module>
    libcmalib = ct.cdll.LoadLibrary(basepath + '/lib/libacmalib.so')
/home/tjim/anaconda3/envs/sound4/lib/python3.7/ctypes/__init__.py:442: in LoadLibrary
    return self._dlltype(name)
/home/tjim/anaconda3/envs/sound4/lib/python3.7/ctypes/__init__.py:364: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/fcmaes/fast-cma-es/fcmaes/lib/libacmalib.so)
=============================================== short test summary info ================================================
ERROR fcmaes/test_cma.py - OSError: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/fcmaes/fast-cm...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================== 1 error in 0.38s ===================================================

So I tried recompiling again to my system glibc and I get:

/opt/fcmaes/fast-cma-es$ pytest 
================================================= test session starts ==================================================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /opt/fcmaes/fast-cma-es
collected 15 items                                                                                                     

fcmaes/test_cma.py ...........F.FFatal Python error: Segmentation fault

Current thread 0x00007f620ee5a740 (most recent call first):
  File "/opt/fcmaes/fast-cma-es/fcmaes/cmaescpp.py", line 115 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/optimizer.py", line 275 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/optimizer.py", line 121 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 445 in _retry_loop
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/process.py", line 99 in run
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/process.py", line 297 in _bootstrap
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/popen_fork.py", line 74 in _launch
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/popen_fork.py", line 20 in __init__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/context.py", line 277 in _Popen
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/context.py", line 223 in _Popen
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/process.py", line 112 in start
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 124 in <listcomp>
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 124 in retry
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 116 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/test_cma.py", line 350 in test_eggholder_advanced_retry
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163 in main
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/tjim/anaconda3/envs/sound4/bin/pytest", line 11 in <module>
Fatal Python error: Segmentation fault

Current thread 0x00007f620ee5a740 (most recent call first):
  File "/opt/fcmaes/fast-cma-es/fcmaes/cmaescpp.py", line 115 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/optimizer.py", line 275 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/optimizer.py", line 121 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 445 in _retry_loop
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/process.py", line 99 in run
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/process.py", line 297 in _bootstrap
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/popen_fork.py", line 74 in _launch
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/popen_fork.py", line 20 in __init__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/context.py", line 277 in _Popen
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/context.py", line 223 in _Popen
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/multiprocessing/process.py", line 112 in start
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 124 in <listcomp>
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 124 in retry
  File "/opt/fcmaes/fast-cma-es/fcmaes/advretry.py", line 116 in minimize
  File "/opt/fcmaes/fast-cma-es/fcmaes/test_cma.py", line 350 in test_eggholder_advanced_retry
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163 in main
  File "/home/tjim/anaconda3/envs/sound4/lib/python3.7/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/tjim/anaconda3/envs/sound4/bin/pytest", line 11 in <module>
F                                                                               [100%]

======================================================= FAILURES =======================================================
______________________________________________ test_rosen_decpp_parallel _______________________________________________

    def test_rosen_decpp_parallel():
        popsize = 8
        dim = 2
        testfun = Rosen(dim)
        max_eval = 10000
        limit = 0.01
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = decpp.minimize(wrapper.eval, dim, testfun.bounds,
                           max_evaluations = max_eval,
                           popsize=popsize, workers = popsize)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
        assert(max_eval + popsize >= ret.nfev) # too much function calls
        assert(max_eval // popsize + 2 > ret.nit) # too much iterations
        assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
>       assert(almost_equal(ret.x, wrapper.get_best_x(), eps = 1E-2)) # wrong best X returned
E       assert False
E        +  where False = almost_equal(array([1.0065809 , 1.01367687]), [1.0017504095915344, 1.0035544851050473], eps=0.01)
E        +    where array([1.0065809 , 1.01367687]) =      fun: 3.3199898595244694e-06\n    nfev: 10007\n     nit: 1008\n  status: 0\n success: True\n       x: array([1.0065809 , 1.01367687]).x
E        +    and   [1.0017504095915344, 1.0035544851050473] = <bound method Wrapper.get_best_x of <fcmaes.testfun.Wrapper object at 0x7f608d84be50>>()
E        +      where <bound method Wrapper.get_best_x of <fcmaes.testfun.Wrapper object at 0x7f608d84be50>> = <fcmaes.testfun.Wrapper object at 0x7f608d84be50>.get_best_x

fcmaes/test_cma.py:300: AssertionError
_________________________________________________ test_eggholder_retry _________________________________________________

    def test_eggholder_retry():
        dim = 2
        testfun = Eggholder()
    
        limit = -956
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = retry.minimize(wrapper.eval, testfun.bounds,
                                 num_retries=100)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
>       assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
E       assert 1036951 == 1621526
E        +  where 1036951 =      fun: -959.640662720851\n    nfev: 1036951\n success: True\n       x: [512.0, 404.2318050386627].nfev
E        +  and   1621526 = <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7f608dde9690>>()
E        +    where <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7f608dde9690>> = <fcmaes.testfun.Wrapper object at 0x7f608dde9690>.get_count

fcmaes/test_cma.py:338: AssertionError
____________________________________________ test_eggholder_advanced_retry _____________________________________________

    def test_eggholder_advanced_retry():
        dim = 2
        testfun = Eggholder()
    
        limit = -956
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = advretry.minimize(wrapper.eval, testfun.bounds,
                                    num_retries=300)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
>       assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
E       assert 663334 == 681059
E        +  where 663334 =      fun: -959.640662720851\n    nfev: 663334\n success: True\n       x: [512.0, 404.23180528869597].nfev
E        +  and   681059 = <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7f608d85af90>>()
E        +    where <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7f608d85af90>> = <fcmaes.testfun.Wrapper object at 0x7f608d85af90>.get_count

fcmaes/test_cma.py:355: AssertionError
=============================================== short test summary info ================================================
FAILED fcmaes/test_cma.py::test_rosen_decpp_parallel - assert False
FAILED fcmaes/test_cma.py::test_eggholder_retry - assert 1036951 == 1621526
FAILED fcmaes/test_cma.py::test_eggholder_advanced_retry - assert 663334 == 681059
======================================= 3 failed, 12 passed in 163.85s (0:02:43) =======================================

Do you have any ideas?

How does bitopt behave? Have you had a chance to compare it to your fcmaes? Would it work as a drop-in replacement, in terms of performance? I might consider checking it out and try to compile it, if so! I'd like to get fcmaes working if possible though!

Thanks for the help.

[timjim333]

I notice in the biteopt notes, it mentions:

Rogue optimums may not be found by this method. A rogue optimum is an optimum that has a very small, almost undetectable area of descent and is placed apart from other competing minima. The method favors minimum with a larger area of descent. The Damavandi test function is a perfect example of the limitation of this method (this test function is solved by this method, but requires a lot of iterations). In practice, however, rogue optimums can be considered as undesired outliers that rely on unstable parameter values (if such parameters are used in real-world system that has a certain parameter value precision, a system may leave the "rogue" optimal regime easily).

In the case of the EHVI method we are using, as we zone in on the optimal solutions, I guess it begins to match the description of 'rogue optimums' in that the area of descent becomes more and more needle-like. I guess it it worth a shot though, it it can outperform scipy's optimize routines.

[dietmarwo]

Hi @timjim333 ,
scipy's optimize routines should be easy to outperform. And the "rogue"-like problem can be mitigated by parallel
retry using different random seeds. See https://github.com/dietmarwo/fast-cma-es/blob/master/examples/examples.py
how to apply BiteOpt (Bite_cpp(max_evals)) using parallel retry. Because of your compilation problems you could try to compile
BiteOpt from its source, but I had problems using gcc-7, switched to gcc-9 for this reason. Seems in an old system indirect library dependencies are still a problem. Next thing we could try is installing gcc-9 on your OS as additional compiler:

https://stackoverflow.com/questions/61590926/how-to-install-gcc-g-9-on-centos-8-docker-centoslatest

dnf -y install gcc-toolset-9-gcc gcc-toolset-9-gcc-c++
source /opt/rh/gcc-toolset-9/enable

Newer runtime libs should be installed this way. The best single objective algorithms from fcmaes are BiteOpt and DE, where for DE there is a native Python version. CMAES is mainly for control/space flight optimizations. For multi-objective/constraint problems now (version 1.3.8) there is MoDe both as native Python and as C++ implementation. Outperforms traditional algorithms like NSGA-II by some margin.

[timjim333]

Hi @dietmarwo thanks for the insight! Ok, I will give installing the latest toolset a go and get back to you. It's fantastic that you have managed to incorporate the method as part of your parallel retry framework - I'm looking forward to getting it running, as I will be tackling a higher-dimensional problem soon.

Where you say 'CMAES is mainly for control/space flight optimizations' - do you mean you believe that it may perform poorly for a design task? Or is it just better now to chain BiteOpt and DE? Thanks for your thoughts!

[dietmarwo]

I was quite involved in the testing of the last version of BiteOpt and closely worked together with Aleksey. Mainly because I use it myself, for instance for solving the solar orbiter optimization problem esa/pykep#127 which is really challenging. Some algorithms are optimized to work standalone but scale bad in a retry scenario. BiteOpt (and fcmaes-DE) don't try to solve the "rogue optimum" problem standalone, but should be used with many, sometimes thousands parallel retries. Modern many core CPUs make this strategy much more attractive. On a 64 core CPU you get 128 parallel retries "for free", so modern algorithms should be adapted and optimized for this scenario. Am Fr., 23. Juli 2021 um 18:11 Uhr schrieb Tim Jim ***@***.***

…

: Hi @dietmarwo <https://github.com/dietmarwo> thanks for the insight! Ok, I will give installing the latest toolset a go and get back to you. It's fantastic that you have managed to incorporate the method as part of your parallel retry framework - I'm looking forward to getting it running, as I will be tackling a higher-dimensional problem soon. Where you say 'CMAES is mainly for control/space flight optimizations' - do you mean you believe that it may perform poorly for a design task? Or is it just better now to chain BiteOpt and DE? Thanks for your thoughts! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOIZMKXHJK3QVNW2IMOD7BLTZGIDFANCNFSM43RM5SGA> .

[timjim333]

Hi @dietmarwo I just tried recompiling with the gcc toolset 9 enabled, and on tunning pytest, I now only get 2 errors instead of 3! Unfortunately, they are still related to the advanced rety:

======================================================= FAILURES =======================================================
_________________________________________________ test_eggholder_retry _________________________________________________

    def test_eggholder_retry():
        dim = 2
        testfun = Eggholder()
    
        limit = -956
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = retry.minimize(wrapper.eval, testfun.bounds,
                                 num_retries=100)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
>       assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
E       assert 977940 == 1515650
E        +  where 977940 =      fun: -959.640662720851\n    nfev: 977940\n success: True\n       x: [512.0, 404.2318052180589].nfev
E        +  and   1515650 = <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7fab49e78910>>()
E        +    where <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7fab49e78910>> = <fcmaes.testfun.Wrapper object at 0x7fab49e78910>.get_count

fcmaes/test_cma.py:338: AssertionError
____________________________________________ test_eggholder_advanced_retry _____________________________________________

    def test_eggholder_advanced_retry():
        dim = 2
        testfun = Eggholder()
    
        limit = -956
        for _ in range(5):
            wrapper = Wrapper(testfun.fun, dim)
            ret = advretry.minimize(wrapper.eval, testfun.bounds,
                                    num_retries=300)
            if limit > ret.fun:
                break
    
        assert(limit > ret.fun) # optimization target not reached
>       assert(ret.nfev == wrapper.get_count()) # wrong number of function calls returned
E       assert 750878 == 774491
E        +  where 750878 =      fun: -959.640662720851\n    nfev: 750878\n success: True\n       x: [512.0, 404.2318050919475].nfev
E        +  and   774491 = <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7facafcb49d0>>()
E        +    where <bound method Wrapper.get_count of <fcmaes.testfun.Wrapper object at 0x7facafcb49d0>> = <fcmaes.testfun.Wrapper object at 0x7facafcb49d0>.get_count

fcmaes/test_cma.py:355: AssertionError
=============================================== short test summary info ================================================
FAILED fcmaes/test_cma.py::test_eggholder_retry - assert 977940 == 1515650
FAILED fcmaes/test_cma.py::test_eggholder_advanced_retry - assert 750878 == 774491
======================================= 2 failed, 13 passed in 131.55s (0:02:11) =======================================

[dietmarwo]

Could not reproduce but anyway tried to make the test more robust. Since the issue is only regarding the number of function calls it is nothing related to the functionality of the library. Will close the issue.