SBML Test-Suite Results Tracker

[anandijain]

SBML test suite report

SBMLtoolkit.jl is regularly tested against the SBML test suite. The last test was run on October 24, 2021 on cases 00001 to 01664. Detailed results can be found in the csv below. Out of the 1664 cases 1219 expectedly failed, as

• they contain SBML constraints, rules or events, or
• do not contain any reactions, or
• cannot be converted to SBML Version 3 Level 2, or
• contain kineticLaws in reversible reactions with ambiguous forward and reverse part, or
• are flux balance models.

Of the remaining 445 cases

• 422 can be converted to a ModelingToolkit.ReactionSystem, of which
• 416 simulate without returning an error, of which
• 416 yield the correct simulation results.

CSV

test_suite_report.csv

[matthiaskoenig]

You should submit the results on the official webpage. This allows people to easily evaluate what can be simulated.
See here for the database: http://raterule.caltech.edu/Facilities/Database
And here for an example: http://raterule.caltech.edu/Facilities/Database/Submission/Details/281

edit: fixed links

[anandijain]

@matthiaskoenig both of those 404

[matthiaskoenig]

@anandijain SBML moved the webpage. The new addresses are available via
https://sbml.org/software/sbml-test-suite/

and at the moment
http://raterule.caltech.edu/Facilities/Database
http://raterule.caltech.edu/Facilities/Database/Submission/Details/281

[matthiaskoenig]

Hi all,
I want to use SBMLToolkit but would need some way to see what is currently supported/failing to see if it makes sense to use the library or it is simpler to use own converters of mine to get the models in Julia.

How can I run the SBMLTestSuite? Where is the code which generates the test_suite_report.csv so I can run it on the latest release/develop branch? I would be especially interested in support of rules/events/delays. If most of what I need is already supported I would like to add support for missing pieces.

Best Matthias

[paulflang]

Hi Matthias, here is a link to a recent SBMLToolkitTestSuite run and the code that generates it. I wanted to upload these results to http://raterule.caltech.edu/Facilities/Database a long time ago, but the website is still down. Do you know if the database has been moved elsewhere?

Rules should be supported, events only partially (i.e. they always fire regardless of direction of the trigger pass - the user is warned of this behaviour), and delays not at all. I also have to admit, I haven't had time to properly look at the test suite cases since three years and we haven't set up a system to check for regression (beyond of what is implemented in the SBMLToolkit repo tests itself). I am sure you are familiar of using SBMLImporter as an alternative.

I haven't looked into delays, but for a long time, event directionality was not supported by Catalyst and therefore not by SBMLToolkit. @TorkelE I forgot where we stand on this issue atm. Once implemented in Catalyst, it is probably just this line that would have to be modified in SBMLToolkit.

[TorkelE]

So I think directional events should now be supported within MTK, so we could be able to use them in Catalyst. But I have since then forgotten what the progress was and if it was supported in principle or it was fully ready. Sebastian probably knows.

[ChrisRackauckas]

They are in MTK.