FMI Import cross-check rules - what are suitable "reference values"? about fmi-cross-check

[t-sommer]

The validation against the reference results was introduced to reject complete nonsense results (e.g. all-zero), missing variables etc. of which we had a lot in the old SVN repo.
Passing the validation (with its very generous delta) cannot be used as a proof of compliance. If we decide to introduce "Reference FMUs" (a working group with the same name has been set up at the Design Meeting in Renningen), we might introduce stricter rules for these FMUs in the next Cross-Check Rules.

[ghorwin]

Would it be meaningful then to distinguish between "basic functionality" (i.e. FMI importing tool is able to read, analyse and run the FMU without generating nonsense) and "correct calculation behavior"?

If so, an FMI importing tool may get a two-level compliance certificate ("one or two stars"), but that would make it even more difficult for users to differentiate between tools. Ideally, each "compliant" tool would be able to generate correct results, so that engineers can rely on the results.

However, for the "basic functionality" check, I agree that the current rules are sufficient.