Tracker for Coverage over CellML Model Repository about cellmltoolkit.jl

[shahriariravanian]

The remaining issues are:

1. Some CellML XML files are defective (missing some initial values). Currently, CellMLToolkit throws an error for these. However, the plan is to return a list of uninitiated variables for the user to provide the values.
2. Some models have more than one iv (in fact, some use partial_diff tag). This is uncommon in CellML models but is supported in the specs.
3. The main remaining active tissue is to implement imports completely. Currently, we have an incomplete implementation. Full import is rather complicated, as CellML XML files can recursively import and rename components and connections (links between variables from different components) from other files. Because of the connections, we may need to import some components implicitly.
4. The ODEProblms which were not solved are not a big problem, as we used a fixed solver (TRBDF2) with some default parameters.
5. Large models (XML size > 500K) can take a long time to generate an ODESystem. I'm going to profile and see where the main problem is, but we may need to change the strategy in how to use structural_simplfy for the very large models.

[anandijain]

2.1.0 is giving ~178/940

[anandijain]

2.2.0 is giving ~477/940

[anandijain]

It is a known issue that some files in the CellML Model Repository have bad XML or do not fit the specification of CellML we use. (aside @shahriariravanian which version of CellML are we guaranteeing should work?)

removing Goldbeeter_2006 from my data folder we now get. The problem is caused in EzXML, where if it hits an error in parsing, it pushes to a global error stack that prevents further usage. why they do this, I have no idea...

530/940

[anandijain]

861 CellML models
718 successfully converted to ODESystem
635 successfully converted to ODEProblem
595 successfully solved

we get 940 from the curls, but cloning the git repos returns 861, so that's where that discrepancy comes from
595/861 is quite good IMO. as a lot of the models are truly defective

[ChrisRackauckas]

What are the issues you see?

[anandijain]

this data is from @shahriariravanian. could you shed some light on chris' question?

[anandijain]

Great, could you name a model with ? I'd like to look into that. Similarly for a model with missing vars and components.

Also, if you end up doing some profiling, I think it'd be good to add benchmarking to our testing of the model repo. I'm happy to add this too with BenchmarkTools.

This may help pin down inefficiencies, ie "is it dependent on parameter count, state count, etc... ?".

[shahriariravanian]

This is the results of the latest run:

#	outcome
867	CellML models
6	too large (>500K, excluded)
744	successfully converted to ODESystem
650	successfully converted to ODEProblem
608	successfully solved

[shahriariravanian]

Here is the result file as a CSV file. The res col codes are:

0 -> fail to generate ODESystem
1 -> fail to generate ODEProblem
2 -> fail to solve ODEProblem
3 -> success!
9 -> too large a file, ignored

cellml_results.txt

[ChrisRackauckas]

Try setting the runner to a lower tolerance. That should help the domain error cases. If not, generate sqrt -> sqrt(abs so step rejects don't error out but instead reject.

[shahriariravanian]

These are the latest tracking results using ver 2.4.1 (to be pushed soon):

#	outcome
867	CellML models
6	too large (>500K, excluded)
775	successfully converted to ODESystem
688	successfully converted to ODEProblem
643	successfully solved

cellml_results_8.txt