juliasparse / mklsparse.jl

Make available to Julia the sparse functionality in MKL

MKLSparse.jl

MKLSparse.jl is a Julia package to seamlessly use the sparse functionality in MKL to speed up operations on sparse arrays in Julia. In order to use MKLSparse.jl you do not need to install Intel's MKL library nor build Julia with MKL. MKLSparse.jl will automatically download and use the MKL library for you when installed.

Matrix multiplications

Loading MKLSparse.jl will make sparse-dense matrix operations be computed using MKL.

Solving linear systems

Solving linear systems with triangular sparse matrices is supported. These matrices should be wrapped in their corresponding type, for example LowerTriangular for lower triangular matrices.

For solving general sparse linear systems using MKL we refer to Pardiso.jl.

Misc

• The integer type that should be used in order for MKL to be called is the same as used by the Julia BLAS library, see Base.USE_BLAS64.

Possible TODO's

• Wrap BLAS1 (SparseVector)
• Wrap DSS
• Wrap Incomplete LU preconditioners

mklsparse.jl's Issues

Error tagging new release

The tag name "v0.1" is not of the appropriate SemVer form (vX.Y.Z).
cc: @KristofferC

Contributing

Hello, I am interested in contributing to this.

I have previously done some work that integrates the Pardiso interface in MKL here https://github.com/JuliaSparse/Pardiso.jl. Is that something you plan on adding to this package? In that case I could probably help a bit with that.

Since this package is MKL specific maybe it is better to use the more simple DSS interface in MKL https://software.intel.com/en-us/node/470302 instead of the Pardiso interface?

Remove type piracy

The matrix multiplication here should likely be implemented with an MKLSparse.matmul or something instead of type piracying the Base matrix multiplication with more specific versions.

Can't install MKLSparse on Windows

I installed MKLSparse on my Windows machine, but when using MKLSparse returns this error:

INFO: Precompiling module MKLSparse.
ERROR: LoadError: error compiling __init__: could not load library "libmkl_rt"
The specified module could not be found.

I've already installed MKL for Windows. I'd appreciate if you could elaborate on how to use the package.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Bug when using MKLSparse following use of MKL

When I use MKLSparse after a previous use MKL, a bug manifests when multiplying a real-valued SparseMatrixCSC{ComplexF64, Int64} matrix times a ComplexF64 or Float64 vector. The following output exhibits the bug:

julia> include("reproducer.jl")
typeof.((QAAQ, fsrc, b)) = (SparseMatrixCSC{ComplexF64, Int64}, Vector{ComplexF64}, Vector{ComplexF64})
size.((QAAQ, fsrc, b)) = ((429567, 429567), (429567,), (429567,))
nnz(QAAQ) = 11532
maximum(abs ∘ imag, QAAQ) = 0.0
maximum(abs, QAAQ - real(QAAQ)) = 0.0
maximum(abs, b - real(QAAQ) * fsrc) = 0.0
0.0

julia> using MKL  # This does not introduce the bug

julia> include("reproducer.jl")
typeof.((QAAQ, fsrc, b)) = (SparseMatrixCSC{ComplexF64, Int64}, Vector{ComplexF64}, Vector{ComplexF64})
size.((QAAQ, fsrc, b)) = ((429567, 429567), (429567,), (429567,))
nnz(QAAQ) = 11532
maximum(abs ∘ imag, QAAQ) = 0.0
maximum(abs, QAAQ - real(QAAQ)) = 0.0
maximum(abs, b - real(QAAQ) * fsrc) = 0.0
0.0

julia> using MKLSparse  # this will introduce the bug

julia> include("reproducer.jl")
typeof.((QAAQ, fsrc, b)) = (SparseMatrixCSC{ComplexF64, Int64}, Vector{ComplexF64}, Vector{ComplexF64})
size.((QAAQ, fsrc, b)) = ((429567, 429567), (429567,), (429567,))
nnz(QAAQ) = 11532
maximum(abs ∘ imag, QAAQ) = 0.0
maximum(abs, QAAQ - real(QAAQ)) = 0.0
maximum(abs, b - real(QAAQ) * fsrc) = 119.4196587781747
119.4196587781747

The final output should be zero since b was computed as QAAQ * fsrc and QAAQ has zero imaginary part for each element. Note that the bug does not manifest if I use the two packages in the other order: MKLSparse first, followed by MKL.

The four files necessary for reproducing this are in this gist. Here is my version info:

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65e (2023-01-08 06:45 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 8

and my package status:

(error_reproducer) pkg> st
Status `D:\peter\Documents\julia\book_codes\RumpfFDFD\error_reproducer\Project.toml`
  [33e6dc65] MKL v0.6.0
  [0c723cd3] MKLSparse v1.1.0

Migrate to new API

See #15 (comment)

The speed ratio of matrix multiplication using MKLSparse stop to grow while number of of threads >= 8.

In my practice, the speed ratio of sparse matrix multiplication using MKLSparse stop to grow while number of of threads >= 8.
Here is my result

System info

Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, cascadelake)

Different type matrices does not use multi-thread

Seems that matrices must have the same typed elements, real(S) * D reverts back to mul! from LinearAlgebra.jl

ENV["OMP_NUM_THREADS"] = Threads.nthreads()÷2
import BenchmarkTools: @btime, @benchmark
using SparseArrays, LinearAlgebra
using MKLSparse

n=2^8
D=rand(ComplexF64,n,n)
S=sprand(ComplexF64,n,n,0.01)
@btime $S * $D # multi thread
@btime $real(S) * $D # single thread
@btime $S * $real(D) # single thread

error caused by isreadable(::String)

I get this error message with Julia 0.6. I think isreadable(::String) vanished with Julia 0.5.

$ julia regression.jl all
ERROR: LoadError: MethodError: no method matching isreadable(::String)
Closest candidates are:
  isreadable(!Matched::Base.Mmap.Anonymous) at mmap.jl:16
  isreadable(!Matched::TCPSocket) at socket.jl:320
  isreadable(!Matched::BufferStream) at stream.jl:1132
  ...
Stacktrace:
 [1] include_from_node1(::String) at ./loading.jl:539
 [2] include(::String) at ./sysimg.jl:14
 [3] process_options(::Base.JLOptions) at ./client.jl:305
 [4] _start() at ./client.jl:371
while loading /usr/people/sseung/Sparso/test/correctness/regression.jl, in expression starting on line 1271

Error on macOS

I cannot use MKLSparse on macOS, I receive the error below:

ERROR: LoadError: Failed to precompile MKLSparse [0c723cd3-b8cd-5d40-b370-ba682dde9aae] to "/Users/eschnett/.julia/compiled/v1.11/MKLSparse/jl_WvNVir".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64})
    @ Base ./loading.jl:3002
  [3] (::Base.var"#1074#1075"{Base.PkgId})()
    @ Base ./loading.jl:2389
  [4] mkpidlock(f::Base.var"#1074#1075"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.0-beta1+0.x64.apple.darwin14/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
  [5] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.11.0-beta1+0.x64.apple.darwin14/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.0-beta1+0.x64.apple.darwin14/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
  [7] #invokelatest#2
    @ ./essentials.jl:1032 [inlined]
  [8] invokelatest
    @ ./essentials.jl:1027 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1074#1075"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:3525
 [10] maybe_cachefile_lock
    @ ./loading.jl:3522 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2385
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2217
 [13] #invoke_in_world#3
    @ ./essentials.jl:1064 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1061 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2208
 [16] macro expansion
    @ ./loading.jl:2148 [inlined]
 [17] macro expansion
    @ ./lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2105
 [19] #invoke_in_world#3
    @ ./essentials.jl:1064 [inlined]
 [20] invoke_in_world
    @ ./essentials.jl:1061 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2098
 [22] include(mod::Module, _path::String)
    @ Base ./Base.jl:558
 [23] include(x::String)
    @ DashSVD ~/.julia/packages/DashSVD/WnFVH/src/DashSVD.jl:1
 [24] top-level scope
    @ ~/.julia/packages/DashSVD/WnFVH/src/DashSVD.jl:5
 [25] include
    @ ./Base.jl:558 [inlined]
 [26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2721
 [27] top-level scope
    @ stdin:4

The reason might be that older versions of MKL (2023.x) supported macOS, but newer version (2024.x) do not support macOS any more. Julia might then try to load an older version. Maybe libmkl_rt doesn't exist there?

Are sparse matrix-matrix multiplications implemented?

Running sparse matrix-dense-vector calculations I can see multi threads being used. I do not see this for sparse matrix-matrix multiplcations. Are they implemented? @code_lowered returns the SparseArray mul! function

julia> @code_lowered A*A
CodeInfo(
1 ─ %1 = SparseArrays.spmatmul(A, B)
└──      return %1
)

Tests don't pass

Hey, the tests don't run, because Base.LinAlg.A_ldiv_B doesn't exist, therefore the tests associated with them would fail.
You import them in dss.jl in the tests folder.

Add two-stage algorithms

Motivated in part by https://discourse.julialang.org/t/wraping-a-mkl-handle-in-julia/48241 I looked in more detail at the two-stage algorithms for recent MKL versions, They are of interest to me, specifically some of the multiplication and solutions. I could create a PR on this repository or create a separate Julia package to wrap these.

Because the two-stage algorithms go beyond some of the BLAS functions I am leaning toward creating a separate package. However, I can be convinced that they would fit in with this package. Any opinions?

5-argument mul! has different signature than in stdlib

In MKLSparse, the 5-argument mul! has a signature mul!(α, A, B, β, C) while in LinearAlgebra (and SparseArrays) it is mul!(C, A, B, α, β) (since Julia v1.3). MKLSparse should adopt the same LinearAlgebra API in order to allow third-party code to work with one or other library without changes.

CC: @fernandopenaranda, who noticed this

matrix-dense-vector multiplication faster for symmetric/hermitian matrices using adjoint

As Julia is for now exclusively using CSC storage format, the matrix-dense-vector multiplication is faster for symmetric/hermitian matrix A if one computes A' * v instead of A * v. A simple example script hinting this is the following:

julia> let
       for D in [10^3, 10^4, 10^5, 10^6]
           A = sprand(D,D,1/D)
           A = A+A'
           v = rand(D)
           w = similar(v)
           @btime mul!($w, $A, $v)
           @btime mul!($w,$A',$v)
       end
       end
  3.500 μs (0 allocations: 0 bytes)
  1.540 μs (0 allocations: 0 bytes)
  23.100 μs (0 allocations: 0 bytes)
  14.100 μs (0 allocations: 0 bytes)
  305.400 μs (0 allocations: 0 bytes)
  206.300 μs (0 allocations: 0 bytes)
  13.283 ms (0 allocations: 0 bytes)
  4.167 ms (0 allocations: 0 bytes)

On

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

and MKLSparse v1.1.0.

Should MKLSparse detect this with ishermitian ?

Add dependency on MKL_jll.jl

That would make the package work automatically and make it easy to enable CI. MKL.jl overwrites OpenBLAS which currently can cause some issues when using other binary dependencies which require BLAS, e.g. Arpack so we might want to wait a bit. MKL.jl also causes a regression in REPL compile latency but hopefully, I'll be able to fix that soon.

We could probably add MKL.jl as a test dependency here as soon as it has been registered.

Julia 1.0 Support?

Hi, are there any plans to support Julia 1.0?

How to control number of threads?

Probably naive question, but I installed this package and got a nice speed boost for sparse matrix * dense vector multiplication, but I still only observe 8 of 64 cores active during the computation. Is there a way to increase the usage further? Not sure if its supposed to matter, but I do already have OMP_NUM_THREADS=64.