What to do about big endian architectures' support? about roaring

[lemire]

Most mainstream architectures being little endian, it may make sense to trade off performance in BE if this improves performance for the LE case.

I think that this is absolutely true. BE is a fringe requirement.

I bet that we have no BE user. It might be fine to have a fallback (slow) approach.

[Oppen]

I think it's worth documenting in the README that BE is only supported as a fallback and will suffer degraded performance for some operations. Just to avoid surprises and spurious issues in the unlikely case someone chooses to user roaring there.

[Oppen]

I'm working on a PR to try and test on ppc64. I could add a line in that same PR about it.

[jacksonrnewhouse]

I believe https://github.com/RoaringBitmap/roaring/blob/master/serialization_generic.go does not assume endianness, hence the "generic" monicker. I also believe it to be correct, as that was what was being used on arm64 machines until we updated the build flags. Now, those arm64 machines are little endian, so there might be something else that breaks on BE, but the intent to support both is there, at least. The performance wasn't terrible using serialization_generic, but it did do many more allocations, as it couldn't do unsafe slice manipulations.

[Oppen]

@jacksonrnewhouse that's not the code that runs for little endian. Despite saying it's generic, because it technically is, for little endian this is what gets built: https://github.com/RoaringBitmap/roaring/blob/master/serialization_littleendian.go

This performs better because it simply does unsafe casting of the slices, but is only valid on LE. It doesn't allocate or copy or call generic functions (those get inlined in practice tho, but all the rest is still overhead).

[Oppen]

Regarding whether it is correct: we still need to test it, and we still prefer to update the build flags so LE use the LE-optimized path. We'll inevitably lag behind for some time when new archs come out, but that still is gonna happen and thus we'll still be skipping the generic code from tests.