sdu-cfei / modest-py

FMI-compliant Model Estimation in Python

FMI-compliant Model Estimation in Python

Description

ModestPy facilitates parameter estimation in models compliant with Functional Mock-up Interface.

Features:

• combination of global and local search methods (genetic algorithm, pattern search, truncated Newton method, L-BFGS-B, sequential least squares),
• suitable also for non-continuous and non-differentiable models,
• scalable to multiple cores (genetic algorithm from modestga),
• Python 3.

Due to time constraints, ModestPy is no longer actively developed. The last system known to work well was Ubuntu 18.04. Unit tests in GitHub Actions are run on Ubuntu 18.04 and Python 3.6/3.7. It does not mean it will not work on other systems, but it is not guaranteed. Use Docker (as described below) if you want to run ModestPy on a tested platform.

Installation with pip (recommended)

It is now possible to install ModestPy with a single command:

pip install modestpy

Alternatively:

pip install https://github.com/sdu-cfei/modest-py/archive/master.zip

Installation with conda

Conda is installation is less frequently tested, but should work:

conda config --add channels conda-forge
conda install modestpy

Docker

Due to time constraints, Modestpy is no longer actively developed. The last system known to work well was Ubuntu 18.04. If you encounter any issues with running ModestPy on your system (e.g. some libs missing), try using Docker.

I prepared a Dockerfile and some initial make commands:

• make build - build an image with ModestPy, based on Ubuntu 18.04 (tag = modestpy)
• make run - run the container (name = modestpy_container)
• make test - run unit tests in the running container and print output to terminal
• make bash - run Bash in the running container

Most likely you will like to modify Dockerfile and Makefile to your needs, e.g. by adding bind volumes with your FMUs.

Test your installation

The unit tests will work only if you installed ModestPy with conda or cloned the project from GitHub. To run tests:

>>> from modestpy.test import run
>>> run.tests()

or

cd <project_directory>
python ./bin/test.py

Usage

Users are supposed to call only the high level API included in modestpy.Estimation. The API is fully discussed in the docs. You can also check out this simple example. The basic usage is as follows:

from modestpy import Estimation

if __name__ == "__main__":
    session = Estimation(workdir, fmu_path, inp, known, est, ideal)
    estimates = session.estimate()
    err, res = session.validate()

More control is possible via optional arguments, as discussed in the documentation.

The if __name__ == "__main__": wrapper is needed on Windows, because modestpy relies on multiprocessing. You can find more explanation on why this is needed here.

modestpy automatically saves results in the working directory including csv files with estimates and some useful plots, e.g.:

1. Error evolution in combined GA+PS estimation (dots represent switch from GA to PS):
2. Visualization of GA evolution:
3. Scatter matrix plot for interdependencies between parameters:

Cite

To cite ModestPy, please use:

K. Arendt, M. Jradi, M. Wetter, C.T. Veje, ModestPy: An Open-Source Python Tool for Parameter Estimation in Functional Mock-up Units, Proceedings of the American Modelica Conference 2018, Cambridge, MA, USA, October 9-10, 2018.

The preprint version of the conference paper presenting ModestPy is available here. The paper was based on v.0.0.8.

License

Copyright (c) 2017-2019, University of Southern Denmark. All rights reserved.

This code is licensed under BSD 2-clause license. See LICENSE file in the project root for license terms.

modest-py's Issues

lp_len has to be explicitly defined if lp_frame is shorter than provided data

When instantiating Estimation users should not have to explicitly specify lp_len when lp_frame is provided. In case of missing lp_len assume that entire lp_frame is to be used.

Add requirements.txt

Add requirements.txt with all required dependencies.

Cost function: choose between RMSE, NRMSE etc.

Users should be free to choose the cost function.

Add FMPy support

Add FMPy support: https://github.com/CATIA-Systems/FMPy/tree/master/fmpy

Simplify API

Currently the module tree is too complex and the number of methods required to set up an estimation session is too large. Ideally, it should look similar to this:

import modest

session = modest.Estimation(workdir, fmu_path, inp, known, est, ideal,
                            lp_n, lp_length, lp_bounds,
                            vp)  # Sets up estimation and validation
estimates = session.estimate()   # Estimates and validates

Add pip installation

Add test suite

modestga zero division error

When running the example simple_1param.py we get a zero division error from modestga:

(venv) (base) D:\code\modest-py>python examples\simple\simple_1param.py
Traceback (most recent call last):
  File "examples\simple\simple_1param.py", line 70, in <module>
    estimates = session.estimate()
  File "d:\code\modest-py\modestpy\estimation.py", line 299, in estimate
    m_estimates = m_inst.estimate()
  File "d:\code\modest-py\modestpy\estim\ga_parallel\ga_parallel.py", line 309, in estimate
    out = minimize(
  File "D:\code\modest-py\venv\lib\site-packages\modestga\ga.py", line 199, in minimize
    mut_rate /= 1 - 1 / len(bounds)                 # Mutate more often
ZeroDivisionError: float division by zero

It happens when running modestga with default parameters.

Unrecognized argument passed to SciPy solver

$ python bin/test.py 
.../home/krza/code/modest-py/modestpy/estim/scipy/scipy.py:177: OptimizeWarning: Unknown solver options: full_output                                                          options=self.options) 

PEP8, flake8, autopep8

For Python code there is the comprehensive style guide PEP8:
https://www.python.org/dev/peps/pep-0008/

flake8 can be used to check the code,
autopep8 can be used to fix most of the issues.
Both are installable from PyPI:

python -m pip install -U flake8 autopep8

Then, cd to the directory and run the following commands to get some harmless whitespace changes:

cd modest-py
flake8 --count --max-line-length=200 --statistics -qq
autopep8 --in-place --recursive --max-line-length=200 --select="E101,W1,W2" .

After that, check the git diff (with and without ignoring whitespace changes) and commit.
If more cleaning is desired, run

flake8 --count --max-line-length=200 --statistics -qq
autopep8 --in-place --recursive --max-line-length=200 --select="E101,E3,W1,W2,W3" .
autopep8 --in-place --recursive --max-line-length=200 .
autopep8 --in-place --recursive --max-line-length=200 --aggressive --aggressive --aggressive .

Or select single rules and check the git diff after each rule to see what changed.

Bug: Plots cannot be created for single parameter optimization

Plotting does not work with one free parameter.

Catch an exception from FMU if solver failed and continue estimation

Currently, when the FMU solver fails the estimation stops. FMU solver sometimes fails only for certain sets of parameters . The estimation could discard these parameters and continue the estimation.

For this to work, we would have to add Exception handling to the optimization algorithms, e.g. https://github.com/krzysztofarendt/modestga

In addtion, this feature would depend on the correct elevation of FMU exceptions to Python in the FMU backend (currently FMPy).

Python 3

I suggest changing the header of EVERY file to the following:

# -*- coding: utf-8 -*-

"""
Copyright (c) 2017, University of Southern Denmark
All rights reserved.
This code is licensed under BSD 2-clause license.
See LICENSE file in the project root for license terms.
"""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

# original script below

As discussed e.g. here:
http://python-future.org/quickstart.html
https://www.digitalocean.com/community/tutorials/how-to-port-python-2-code-to-python-3

Accuracy criterion should be accessed from Estimation API

Currently, the Estimation API gives access only to maximum number of iterations, but not to other stopping criteria affecting the accuracy. These parameters should be exposed as optional parameters.

Does not work on Ubuntu 20.04

When I run python modestga/test/run.py on Ubuntu 20.04, tests fail on:

Failed to load shared library /tmp/tmpi0sgvqkt/binaries/linux64/Simple2R1C.so. libgfortran.so.3: cannot open shared object file: No such file or directory

I have not time to actively maintain this package anymore, but I will add a Dockerfile with a setup working with Modestpy.

FMU created with OpenModelica

When using an FMU created with OpenModelica the identification process result is a crash.

Environment:

• Ubuntu 16.04
• OpenModelica compiled from last source OMEdit v1.13.0-dev.214+gb23d97a
• FMU type tried with both co-simulation and model-exchange FMU version2 and both dynamic lib and static lib
• Modest-py from latest git

Test:

• I have substituted the Simple2R1C_ic_linux64.fmu in example/simple/resource with the same compiled with OpenModelica.
• I have run the simple.py test
• The result is a crash
• I have put the original file Simple2R1C_ic_linux64.fmu back to example/simple/resource, the identification process gives me a result

Files in attach:

• simple.log: the modest-py log when the failure occours
• screenlog.log: the screen log dumped to file when the failure occours
• Simple2R1C_ic_linux64.fmu.zip: the FMU compiled with OpenModelica

simple.log
Simple2R1C_ic_linux64.fmu.zip
screenlog.log

Add win64 FMUs in examples and unit tests

Currently there are only FMUs for lin64 and win32. Generation of other is possible if pymodelica is installed, but most users don't use pymodelica.

User-defined cost function

A user needs to have a possibility to add his own cost function.

Test model with only 1 valid set of parameters

Currently the parameters in the test models are interdependent, leading to often non-convex cost function. Add model with only 1 valid solution.

ValueError when time is floating point in _select_lp

Hi, when I gave floating point time instances, random.randint() gives Value error. Is there a particular reason for using random.randint() instead of random.uniform() ? [I locally replaced it with random.uniform(), and it worked with floating time also].

modest-py/modestpy/estimation.py

Line 612 in d539a94

new_t0 = random.randint(t0, tend - lp_len)

Note that I did not give lp_frame explicitly, so it took the time from "ideal". (If the choice is related to performance, maybe rounding can be made?)

Thanks in advance

Refactor code with black and isort

Refactor code with black and isort

Update front page and documentation

Front page and docs are outdated.

run.tests() fails on Linux (WSL2, Ubuntu 18.04)

When running run.tests() on Ubuntu 18.04 (WSL2) I get the following output from the second simulation:

.[Model] Simulation count = 1
Simulation interval    : 0.0 - 215940.0 seconds.
Elapsed simulation time: 0.1823941000002378 seconds.
[Model] Simulation count = 2
Simulation interval    : 0.0 - 215940.0 seconds.
Elapsed simulation time: 0.17961690000083763 seconds.
[Model] Simulation count = 3
Simulation interval    : 0.0 - 215940.0 seconds.
Elapsed simulation time: 0.17289809999783756 seconds.
.[Model] Simulation count = 1
Simulation interval    : 0.0 - 215940.0 seconds.
Elapsed simulation time: 0.18132080000214046 seconds.
[Model] Simulation count = 2
Segmentation fault

Environment installed as follows:

conda create --name modestpy python=3.8
conda activate modestpy
conda install scipy pandas numpy matplotlib
conda install -c conda-forge pyfmi pydoe

The same environment works fine on Windows.
It might be an issue with the FMU Linux binaries.

Add unit test and example for Estimation class

Unit test fails on Debian 9

Fresh install on Debian 9, Python 3 (Anaconda), installed via conda:

======================================================================
FAIL: test_estimation_rmse (modestpy.test.test_estimation.TestEstimation)

Traceback (most recent call last):
File "/home/krza/anaconda3/envs/modestpy/lib/python3.5/site-packages/modestpy/test/test_estimation.py", line 119, in test_estimation_rmse
self.assertLess(err['tot'], 1.48)
AssertionError: 2.754943309237994 not less than 1.48

---

Ran 10 tests in 81.882s

FAILED (failures=1)

Initialize GA with LHS

There should be a possibility to initialize GA with LHS samples.

Replace the GA implementation

The current implementation of the genetic algorithm requires some refactoring. A new version was released as a separate package: https://github.com/krzysztofarendt/modestga.

modestga has a very similar interface to scipy.optimize.minimize, so it should be very easy to add it.

modestga: cannot start a new process

Happens when running test_estimation_basic_parallel() on Windows 10 with FMPy.

ERROR: test_estimation_basic_parallel (modestpy.test.test_estimation.TestEstimation)
Will use default methods ('MODESTGA', 'PS')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "d:\code\modest-py\modestpy\test\test_estimation.py", line 86, in test_estimation_basic_parallel
    estimates = session.estimate()
  File "d:\code\modest-py\modestpy\estimation.py", line 299, in estimate
    m_estimates = m_inst.estimate()
  File "d:\code\modest-py\modestpy\estim\ga_parallel\ga_parallel.py", line 309, in estimate
    out = minimize(
  File "D:\code\modest-py\venv\lib\site-packages\modestga\ga.py", line 166, in minimize
    p.start()
  File "c:\users\krzys\appdata\local\programs\python\python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "c:\users\krzys\appdata\local\programs\python\python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "c:\users\krzys\appdata\local\programs\python\python38\lib\multiprocessing\context.py", line 326, in _Popen
    return Popen(process_obj)
  File "c:\users\krzys\appdata\local\programs\python\python38\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "c:\users\krzys\appdata\local\programs\python\python38\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "c:\users\krzys\appdata\local\programs\python\python38\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

__init__.py does not load Estimation class

Currently to load the estimation class one needs to write:
from modestpy.estimation import Estimation

instead of:
from modestpy import Estimation

FMUs stored in Temp folder

When I run Modestpy it appears that my temporary folder (C:\Users\USER\AppData\Local\Temp) gets filled with the different FMUs in folders with names such as tmp9zz77pwi. Recently it caused the program to stop, since the temporary folder was full. I would expect FMPy to remove the folder after simulation, but I don't know if this is controlled by FMPy or Modestpy?

Add simple GUI

ModestPy was aimed to serve as a supplementary package for more advanced applications (e.g. Model Predictive Control). However, some users might be interested in using it for offline parameter estimation of standalone models. A simple GUI would be helpful in such cases.

Upload v. 0.0.8 to PyPi

Version 0.0.8 is the one used in the ModestPy paper.

The main change with respect to 0.0.7 is the addition of the SciPy algorithms.

Tests should use temporary directories

Tests should use temporary directories and delete the results after testing completion.

CVode_options key error

Simulations of FMUs shipped with non-CVode solver fail due to KeyError:

File "modestpy/fmi/model.py", line 136, in simulate
    opts["CVode_options"]["atol"] = 1e-6
KeyError: 'CVode_options'

Setting 'atol' should be solver-independent and, preferably, user-accessible, not fixed.

Generate documentation

Bug: Cannot set ps_iter to 0

The Estimation class does not work with ps_iter = 0.

Validate Hooke-Jeeves

Validate Hooke-Jeeve, check the implementation of adaptive steps, make sure the error always decreases...

Bug: Matplotlib error

Sometimes after estimation the following warning is printed in the terminal:

In [5]: can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"

It does not affect the results and there is no exception raised. The problem is likely linked with the way the plots are stored, saved and closed (all at once).

Logging should not interfere with the logger of the parent application

• Remove logging.basicConfig from log_init.py
• Move all loggers to constructors

Pseudo-random population generator

There should be a possibility to generate a pseudo-random population in GA. That would allow to implement better unit tests.

Add SLSQP as an alternative method

SLSQP could be used as an alternative method to GA+PS.

Step-by-step instructions to get examples up and running

Hi,

I'm having some difficulty getting the example code up and running. I cloned the github repo on a new Ubuntu linux VM and ran pip install .

I then ran python3 ./examples/simple/simple.py

The output looks like:

[Errno 2] No such file or directory: 'examples/simple/resources/Simple2R1C_ic_linux64.fmu'
[Errno 2] No such file or directory: 'examples/simple/resources/Simple2R1C_ic_linux64.fmu'
[Errno 2] No such file or directory: 'examples/simple/resources/Simple2R1C_ic_linux64.fmu'
[Errno 2] No such file or directory: 'examples/simple/resources/Simple2R1C_ic_linux64.fmu'

Examples to be run from any directory

A user should be able to run examples also from the example subdirectory.

Save numerical results with optimization trajectory in addition to plots

Plots generated by ModestPy have fixed format, sometimes overlapping labels etc. It would be useful to save numerical results with estimation trajectories to allow a user to reproduce all plots.

Axis labels: 'RMSE' vs. 'NRMSE'

Currently axis labels on plots always refer to RMSE, even if NRMSE is used.

Python 3 support

PyFMI, Assimulo and other dependencies are already working with Python 3 and can be easily installed through conda (together with the required binary packages).

Plot size too small

Output plots are too small and the fonts are overlapping.

Switch to pytest

Switch to pytest. It has a much nicier API.

Generate test FMUs for Windows

Add examples

Add usage examples of modestpy.