Performance test about oldimmutablearrays.jl

[dronir]

I can try and put together something quick tonight, similar to the addition test I did earlier.

[dronir]

I ran into a slightly odd thing while writing my performance test. I tried to write just a generic test function, but found that it affects performance really badly in a (to me) unexpected way. Could someone better acquainted with the workings of Julia clarify this to me?

using ImmutableArrays
typealias Vec3 Vector3{Float64}

# Test performance with addition
function test(N::Integer)
    a = Vec3( 0.1, 0.5, 1.0)
    b = Vec3(-0.1, 0.5, 0.0)
    +(a,b)
    @elapsed for i = 1:N
        c = +(a,b)
    end
end

# Test performance with f(a,b)
function test(f::Function, N::Integer)
    a = Vec3( 0.1, 0.5, 1.0)
    b = Vec3(-0.1, 0.5, 0.0)
    f(a,b)
    @elapsed for i = 1:N
        c = f(a,b)
    end
end

Now, the first version, which tests just the + function, takes 0.133 seconds to run with a million iterations, but if I call the second one using the + function, it takes twice as long:

julia> test1(1000000)
0.133707178

julia> test2(+, 1000000)
0.226056539

[dronir]

The actual odd thing is that if I do the exact same test except for Julia vectors, a = [0.1, 0.5, 1.0] etc., there is no difference between those two calls.

[twadleigh]

@dronir : The latest commit generates a definition for addition equivalent the "explicit" version in your benchmark gist.

[dronir]

Ok, I'll give that a spin.

Also, FWIW, I got some feedback on the Julia IRC channel about that timing difference. The idea is that in my first case the compiler can inline the call to +(a,b). For normal Julia Vectors, inlining is not possible because their length is not known a priori.

[dronir]

In my most simplest test (just a loop repeating the same addition of two vectors), we now have a factor of 500 in performance over Julia's Vector. But in most real situations it'll be a lot less. Trying to come up with good realistic tests.

[jayschwa]

I just added a very crude speed test in 3e82d4b that compares ImmutableArrays operations relative to native Julia arrays.

Despite not having changed much over the past couple months, it seems ImmutableArrays is now slower than native arrays. The test blew up on Travis CI for some reason, but locally, I'm seeing the following:

ImmutableArray operation speed relative to native Julia arrays:
+ 6.9845887294827635
* 2.5084415916681846

Either the native arrays got some significant performance gains or there was some kind of regression with respect to structures. Perhaps someone like @JeffBezanson has insight?

[twadleigh]

I'm guessing the performance penalty is in the Matrix4x4() constructor recently added, which is defined in terms of ntuple, and which I have seen to be a big bottleneck in the past. Unfortunately I don't have time to test this hypothesis right now, but I would tweak test/speed.jl to use the older, more verbose, constructor and re-run.

[twadleigh]

The constructor theory would also explain why the + number is bigger than the * number: addition is cheaper, so the effect of the constructor overhead would loom larger.

[jayschwa]

I noticed the slowdown before I added the constructor. Additionally, the
time macro should only be measuring addition or multiplication, not
construction. I acknowledge the constructor is probably sub-optimal, but I
don't think it explains what I'm observing.

[twadleigh]

As you may have noticed, I wrote that note in the middle of the night/super early in the morning. (I was up with some insomnia.) I realized after the fact that maybe I got that wrong.

Matrix4x4 is the biggest structure currently defined. Maybe it is still faster for smaller structures (like Vector3, say)...

[jayschwa]

I just tested with an older version of Julia (a7b6dbd) from about 2 months ago. ImmutableArrays in its current state runs much faster with it. I'll probably open up an issue in the Julia repo.