Slow compile times about delaydiffeq.jl

[ChrisRackauckas]

Is it mostly Vern9?

[devmotion]

The problem is not new. With

• DelayDiffEq 5.4.1
• OrdinaryDiffEq 5.8.1
• DiffEqBase 5.10.3
• DiffEqDevTools 2.9.0
• DiffEqMonteCarlo 0.14.0

(roughly 20 days ago) I get

160.435743 seconds (221.02 M allocations: 81.873 GiB, 8.79% gc time)
  0.156057 seconds (1.32 M allocations: 24.280 MiB, 9.76% gc time)

Multiple runs produce slightly different numbers (sometimes also slower than on master), but the same issue seems to be present in this configuration as well.

Is it mostly Vern9?

I only measure BS3 in this script, Vern9 is only used to compute an approximate solution.

[ChrisRackauckas]

Was it the introduction of @..?

[devmotion]

No, does not seem to be the case. Using

• DelayDiffEq 5.2.0
• OrdinaryDiffEq 5.6.1
• DiffEqBase 5.8.0
• DiffEqDevTools 2.8.0
• DiffEqProblemLibrary 4.1.0
• DiffEqMonteCarlo 0.14.0

(with calculate_residuals moved to DiffEqBase which uses @..) gives

 33.210539 seconds (204.77 M allocations: 20.490 GiB, 12.65% gc time)
 12.239943 seconds (152.93 M allocations: 2.808 GiB, 15.72% gc time)

[devmotion]

A much stronger impact has the removal of BenchmarkTools from DiffEqDevTools. With

• DelayDiffEq 5.2.0
• OrdinaryDiffEq 5.6.0
• DiffEqBase 5.7.0
• DiffEqDevTools 2.9.0
• DiffEqProblemLibrary 4.1.0
• DiffEqMonteCarlo 0.14.0

I get

 55.918532 seconds (134.95 M allocations: 47.677 GiB, 9.38% gc time)
  0.153114 seconds (1.32 M allocations: 24.550 MiB, 15.23% gc time)

This is still better than what I observe on the current master branches, but a huge change compared with DiffEqDevTools 2.8.0.

[ChrisRackauckas]

DiffEqDevTools wasn't computing correctly with BenchmarkTools because of the fancy compiler tricks it was doing, so I'm not sure that's a real effect.

[devmotion]

I think we're talking about different things, the times are not the benchmark results but (on purpose) include the compile time. When I ran the Mackey and Glass benchmarks yesterday, the results looked very similar to the ones that are available online (I haven't compared them in detail due to SciML/DiffEqDevTools.jl#42, but that's fixed now). However, it took a lot of time to run those benchmarks just due to the initial compile time, and together with what we observed in #124 (comment), I was worried that we might have introduced some severe regressions regarding compilation times.

However, it seems that already the update from DiffEqDevTools 2.8.0 to 2.9.0 (including the removal of BenchmarkTools) led to an increased initial compile time and the current state is similar to how it was almost a month ago, before I started working on DelayDiffEq.

[devmotion]

My assumption is that the long compilation times that we observe could also be caused by the complicated setup of the cache using generated functions...

What makes this whole setup a bit complicated is that for some algorithms the type of the cache depends on f (all those with a non-linear solver, e.g.).

However, in principle the solution with the dense history and the ODE integrator would need only a very reduced cache with the tableaus to be able to perform all interpolations, it seems. Executing addsteps! before interpolating/extrapolating the history and the ODE integrator does not change the stages since they are already complete all the time. We have to ensure this because otherwise it can happen that the computation of an additional stage requires the evaluation of the history function at a time point in the same integration step, which in turn then calls the same addsteps! method, and so on... That actually happened quite commonly when using the Verner methods in the beginning and took me some time to figure out.

So, both the dense history and the ODE integrator use the cache only for dispatching to the correct interpolation and to access the correct tableau - they never call f or change any stages. So I'm wondering whether we could somehow call into the interpolation machinery in OrdinaryDiffEq by just using the supertype of the cache and the tableau, hoping that we can get rid of all dependencies on f and all unneeded types and objects.

[devmotion]

I asked for help regarding those extreme compilation times: https://discourse.julialang.org/t/extremely-long-compilation-time-2000-sec-on-win32/26496/1

[devmotion]

Avoided for now by making Zygote an optional dependence of SparseDiffTools.