Importing DataInterpolations.jl takes more than 8 seconds about datainterpolations.jl

[ChrisRackauckas]

We can make it a weak dependency with v1.9.

[GlenHertz]

Added the weak dependency on 1.9 (while still being backwards compat with 1.8) with the help of @oscardssmith at #131

[ChrisRackauckas]

Can you do @time_imports using DataInterpolations? That will show what exactly is taking all of the time.

[nrontsis]

Sure:

julia> @time_imports using DataInterpolations
      0.4 ms  Compat
     72.0 ms  ChainRulesCore
      3.7 ms  DocStringExtensions 57.11% compilation time
      0.2 ms  SnoopPrecompile
     57.0 ms  RecipesBase
      3.7 ms  StaticArraysCore
      3.9 ms  ArrayInterfaceCore
     36.3 ms  MacroTools
      0.3 ms  ZygoteRules
      0.4 ms  Adapt
      0.5 ms  ArrayInterfaceStaticArraysCore
    165.5 ms  FillArrays
      0.2 ms  DataValueInterfaces
      1.5 ms  DataAPI
      0.2 ms  IteratorInterfaceExtensions
      0.1 ms  TableTraits
     16.5 ms  Tables
      4.1 ms  GPUArraysCore
     35.2 ms  RecursiveArrayTools
      0.2 ms  Reexport
    668.6 ms  StaticArrays
      0.3 ms  Requires
     20.1 ms  FiniteDiff 54.03% compilation time (14% recompilation)
     13.9 ms  IrrationalConstants
      1.1 ms  DiffRules
      3.3 ms  DiffResults
     30.2 ms  Preferences
      2.2 ms  OpenLibm_jll
      0.5 ms  NaNMath
      0.8 ms  ChangesOfVariables
      1.5 ms  InverseFunctions
      1.0 ms  LogExpFunctions
      0.4 ms  JLLWrappers
     46.7 ms  CompilerSupportLibraries_jll
     48.1 ms  OpenSpecFun_jll 5.09% compilation time
     23.1 ms  SpecialFunctions
      0.4 ms  CommonSubexpressions
    148.6 ms  ForwardDiff
      6.1 ms  NLSolversBase
      1.1 ms  PositiveFactorizations
      0.2 ms  UnPack
      0.5 ms  Parameters
      7.1 ms  LineSearches
     92.1 ms  DataStructures
      1.1 ms  SortingAlgorithms
     11.3 ms  Missings
      0.4 ms  StatsAPI
     33.3 ms  StatsBase
     27.7 ms  Optim
      0.3 ms  TermInterface
      0.4 ms  ExprTools
    185.7 ms  TimerOutputs
      0.3 ms  AutoHashEquals
      0.8 ms  ConstructionBase
     44.4 ms  Setfield
     11.5 ms  SplittablesBase
      0.6 ms  ArgCheck
     41.2 ms  InitialValues
    741.1 ms  BangBang 93.70% compilation time (60% recompilation)
     11.4 ms  MicroCollections
      3.6 ms  Referenceables
     93.2 ms  Baselet
      0.2 ms  CompositionsBase
      0.1 ms  DefineSingletons
     86.2 ms  Transducers 38.60% compilation time
     33.8 ms  ThreadsX
    265.2 ms  Metatheory 88.79% compilation time
     35.9 ms  IntervalSets
      2.4 ms  CompositeTypes
    149.2 ms  DomainSets
     25.7 ms  AbstractTrees
      0.1 ms  IfElse
      8.1 ms  Combinatorics
    393.0 ms  MutableArithmetics
     92.1 ms  MultivariatePolynomials
     36.6 ms  DynamicPolynomials
      1.8 ms  Bijections
      1.4 ms  PreallocationTools
     32.2 ms  LabelledArrays
    311.3 ms  SymbolicUtils
      1.3 ms  RuntimeGeneratedFunctions
      0.2 ms  CommonSolve
     17.9 ms  FunctionWrappers
      1.3 ms  FunctionWrappersWrappers
      0.4 ms  EnumX
    251.1 ms  SciMLBase
      0.2 ms  TreeViews
     31.5 ms  RandomExtensions
      6.8 ms  GroupsCore
    424.4 ms  AbstractAlgebra
      0.2 ms  IntegerMathUtils
     13.8 ms  Primes
      5.7 ms  Groebner
     30.0 ms  PDMats
    299.5 ms  Rmath_jll 91.47% compilation time (97% recompilation)
     97.8 ms  Rmath 89.93% compilation time
      3.4 ms  Calculus
    123.2 ms  DualNumbers
      1.6 ms  HypergeometricFunctions
      9.2 ms  StatsFuns
      4.6 ms  QuadGK
      2.2 ms  DensityInterface
    293.5 ms  Distributions
      1.2 ms  LaTeXStrings
      1.0 ms  Formatting
     58.1 ms  Latexify 31.86% compilation time (100% recompilation)
     42.9 ms  LambertW 40.29% compilation time
    913.7 ms  Symbolics 0.64% compilation time
    290.6 ms  MLStyle
      0.2 ms  Memoize
      0.2 ms  Underscores
     27.7 ms  LeastSquaresOptim
     28.8 ms  Lazy
      2.5 ms  RegularizationTools
    274.9 ms  DataInterpolations

[ChrisRackauckas]

Looks like it's mostly the symbolics stack. @YingboMa @shashi so you're aware.

[GlenHertz]

In my project the using MyProject time went from 12 seconds to 6 seconds when I removed Symbolics (these lines) from DataInterpolations:

https://github.com/PumasAI/DataInterpolations.jl/blob/d31cb1cd1e23cd7df4849d2cf60d774aa752d3b6/src/DataInterpolations.jl#L39-L41

Does DataInterpolations need to depend on Symbolics?

[GlenHertz]

I was running SnoopCompile to find invalidations in my project and most of them were to do with isequal definitions in Symbolics.

[ChrisRackauckas]

Open an issue and share it?

[GlenHertz]

Continued the invalidations issue in Symbolics here:
JuliaSymbolics/SymbolicUtils.jl#496

[baggepinnen]

The deps that arise from (I believe) the CurveFit interpolation are quite heavy

In particular MLStyle which, if Symbolics is already loaded, contributes the majority of the loading time

julia> @time_imports using DataInterpolations
    436.9 ms  MLStyle
      0.2 ms  Memoize
      0.2 ms  Underscores
     21.2 ms  LeastSquaresOptim
     27.3 ms  Lazy
      2.0 ms  RegularizationTools
    432.6 ms  DataInterpolations

[nrontsis]

Do you think we can close this now?

In the v4.0.0 release I get:

$ docker run --rm -it julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> add https://github.com/PumasAI/DataInterpolations.jl

julia> @time_imports using DataInterpolations
      9.7 ms  DocStringExtensions 30.09% compilation time
     30.1 ms  Preferences
      0.2 ms  SnoopPrecompile
     53.7 ms  RecipesBase
      4.3 ms  StaticArraysCore
      0.5 ms  Requires
    134.6 ms  ArrayInterface 78.50% compilation time (2% recompilation)
      1.1 ms  SymbolicIndexingInterface
      0.9 ms  Compat
     79.0 ms  ChainRulesCore
     41.1 ms  MacroTools
      0.4 ms  ZygoteRules
     17.1 ms  Adapt 59.95% compilation time
    197.6 ms  FillArrays
      0.2 ms  DataValueInterfaces
      1.8 ms  DataAPI
      0.2 ms  IteratorInterfaceExtensions
      0.2 ms  TableTraits
      8.8 ms  OrderedCollections
     33.4 ms  Tables
      4.7 ms  GPUArraysCore
     65.4 ms  RecursiveArrayTools 16.59% compilation time
     66.3 ms  DataInterpolations 67.38% compilation time

Either way, thanks so much for all the awesome work 😍

[ChrisRackauckas]

I wonder what's pulling in FillArrays... but it's not us so that's a problem for another time. Cheers.

[nrontsis]

I wonder what's pulling in FillArrays.

I think it's RecursiveArrayTools

julia> PkgDependency.tree("DataInterpolations")
 DataInterpolations v4.0.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━
            │
            ├── RecipesBase v1.3.3
            │   └── SnoopPrecompile v1.0.3
            │       └── Preferences v1.3.0
            │           └── TOML v1.0.0
            ├── RecursiveArrayTools v2.38.0
            │   ├── StaticArraysCore v1.4.0
            │   ├── GPUArraysCore v0.1.4
            │   │   └── Adapt v3.6.1
            │   │       └── Requires v1.3.0
            │   ├── IteratorInterfaceExtensions v1.0.0
            │   ├── ZygoteRules v0.2.3
            │   │   ├── ChainRulesCore v1.15.7
            │   │   │   └── Compat v4.6.1
            │   │   └── MacroTools v0.5.10
            │   ├── Tables v1.10.1
            │   │   ├── DataAPI v1.14.0
            │   │   ├── OrderedCollections v1.4.1
            │   │   ├── IteratorInterfaceExtensions v1.0.0 (*)
            │   │   ├── DataValueInterfaces v1.0.0
            │   │   └── TableTraits v1.0.1
            │   │       └── IteratorInterfaceExtensions v1.0.0 (*)
            │   ├── DocStringExtensions v0.9.3
            │   ├── ArrayInterface v7.2.1
            │   │   ├── Requires v1.3.0 (*)
            │   │   ├── SnoopPrecompile v1.0.3 (*)
            │   │   └── Adapt v3.6.1 (*)
            │   ├── SymbolicIndexingInterface v0.2.2
            │   │   └── DocStringExtensions v0.9.3 (*)
            │   ├── RecipesBase v1.3.3 (*)
            │   ├── ChainRulesCore v1.15.7 (*)
            │   ├── Requires v1.3.0 (*)
            │   ├── FillArrays v0.13.9
            │   └── Adapt v3.6.1 (*)
            ├── Requires v1.3.0 (*)
            └── Reexport v1.2.2

[ChrisRackauckas]

FillArrays is addressed in SciML/RecursiveArrayTools.jl#257