Remove the need for SuiteSparse_wrapper.c about sparsearrays.jl

[ettersi]

Pinging @ettersi to see this discussion as well.

Happy to help, but I still feel like I don't understand the build process and the various constraints well enough to get very far on my own. So for the time being, I think the best I can do is to just follow along in the various conversations until things start to click.

[ViralBShah]

The next step is to take the output in the repo posted above (https://github.com/Gnimuc/SuiteSparseGenerator) and update the cholmod wrappers (do they even need updating?) to use them. I don't think any binary building updates are necessary.

@Gnimuc initially we could just manually run clang.jl and check in the results instead of automating.

[ettersi]

Probably better in the long run, but requires a complete overhaul of the CHOLMOD module. So I'd say if we do this, then we should be sure that this is truly what we want and take some time to really get it right. In particular, I guess it would make sense to not only update CHOLMOD, but also the UMFPACK and SPQR wrappers?

[ViralBShah]

It is not easy to separate out SuiteSparse.jl into a separate repo - because it is a bit tightly coupled with SparseArrays, which is a bit tightly coupled with LinearAlgebra. So, there's some non-trivial work to separate these out, as I understand it. @KristofferC has perhaps tried it and can say more.

[ViralBShah]

The actual work will have to be carried out in the JuliaPackaging/Yggdrasil repo, since that's where the wrapper lives nowadays.

[ViralBShah]

As @simonbyrne pointed out on slack, the 32/64 bit stuff is in module init, not the wrapper.

[Gnimuc]

With JuliaInterop/Clang.jl#289, we can do something like https://github.com/Gnimuc/SuiteSparseGenerator. Like _JLL packages, it just generates one binding per platform.

Any thoughts? @ViralBShah, @simonbyrne

[ViralBShah]

I'd love to try it out. We can make it part of the SuiteSparse_jll build.

[giordano]

We can make it part of the SuiteSparse_jll build.

It isn't so easy, we need to run some Julia code at the end of the build which isn't part of the regular BinaryBuilder pipeline.

I find the approach suggested by Jameson much more viable, it should probably work right now, but someone needs to take the time to do it.

[ViralBShah]

If I understand what @Gnimuc has proposed, there is no Julia code that needs to be run. Can't we check this stuff into the SuiteSparse stdlib (I realize it doesn't belong in SuiteSparse_jll), the files in gen/ and config/ both? They can just be regenerated when we bump SuiteSparse. Basically the toml files here include the information that @vtjnash shows how to get programatically

[giordano]

Oh, ok, I thought we'd need to run Clang.jl at some stage, as it was suggested at some point in JuliaPackaging/BinaryBuilder.jl#673

[ViralBShah]

Pinging @ettersi to see this discussion as well.

[Gnimuc]

To clarify, there is no need to run Clang.jl in the BB environment but we still need to run Clang.jl to generate those bindings on someone's machine(or certain CI environment).

I don't know how those xxx_jll.jl repos are generated, but I believe they're generated by running some Julia code. I guess we can follow the same way to trigger and run the scripts with Clang.jl.

stdlibs like LibCURL.jl and Statistics.jl are developed independently out of the Julia source tree in separated repos. When building Julia, the only thing we need is just downloading. Is this possible to let SuiteSparse do the same?

[ViralBShah]

We do want to separate out suitesparse (and all of linear algebra) into separate repos. See JuliaLang/julia#22698.

[ettersi]

update the cholmod wrappers (...) to use them

Functions and types, or just the types? The types are essentially the same, only difference is that the generator hard-codes SuiteSparse_long -> Clong, and uses Cvoid instead of nothing. Reworking the rest of the wrapper to use the automatically generated functions would obviously be something which would take a couple of hours at least.

[ViralBShah]

Yes, I think that is pretty much what needs to be done. Going forward then, we auto-generate the wrappers and just check them into SuiteSparse.jl.

[ettersi]

Yes to updating just the types, or also the functions?

[ViralBShah]

Ideally both, so that they are always generated in a consistent way and will be in sync with the generated types. What is your recommendation?

[ViralBShah]

The UMFPACK and SPQR do not use the manually generated C++ wrappers though. So maybe this is a 3 step process, which could be separate PRs:

1. Get the types sorted out - which is the thing that keeps changing the most in CHOLMOD.
2. Update the CHOLMOD functions next.
3. Do the others later.

@Gnimuc would then have to generate the wrappers for the others as well in SuiteSparseGenerator. I suggest we check the generator code into the SuiteSparse stdlib codebase itself with instructions on how to generate the wrappers. This is exactly what we do for Sundials.

[Gnimuc]

@Gnimuc would then have to generate the wrappers for the others as well in SuiteSparseGenerator.

Done!

I suggest we check the generator code into the SuiteSparse stdlib codebase itself with instructions on how to generate the wrappers.

Maybe the first step should be to separate out suitesparse.jl.

[Gnimuc]

Probably better in the long run, but requires a complete overhaul of the CHOLMOD module. So I'd say if we do this, then we should be sure that this is truly what we want and take some time to really get it right. In particular, I guess it would make sense to not only update CHOLMOD, but also the UMFPACK and SPQR wrappers?

Looks like most of the code in the CHOLMOD module are very high level. Maybe they do need to be refactored to some extent, but there should not be too much work to merely replace the old hand-written ccalls with those thin wrapper functions generated by Clang.jl.

[ViralBShah]

@Gnimuc With your recent PRs, is this something we recently addressed?

[Gnimuc]

The last step would be removing SuiteSparse_wrapper.c from SuiteSparse_jll.

[ViralBShah]

Ok - and also from deps.

[ViralBShah]

Removed the wrapper from the JLL: JuliaPackaging/Yggdrasil#3099

[nalimilan]

For the context see JuliaSparse/SuiteSparse.jl#2 and JuliaSparse/SuiteSparse.jl#3.