Summing a list of floating-point numbers about flips

[TysonMN]

That is very clever. Thanks for the explanation and specific example.

I will make sure there is a test that fails if the sort is removed before closing this issue.

[TysonMN]

That's fine. PR #170 is just a proof of concept.

[matthewcrews]

Great question! It's due to the behavior around the addition of floats. If you do not sort ahead of time, values will not cancel that should cancel. For example,

> 1_000_000_000_000.0 + 0.001 - 1_000_000_000_000.0 - 0.001;;
val it : float = -2.34375e-05

It should evaluate to 0.0 but instead it evaluated to -2.34375e-05

If you sort it, you get this...

> 1_000_000_000_000.0 - 1_000_000_000_000.0 + 0.001  - 0.001;;
val it : float = 0.0

This is the correct behavior. The sort protects against some of the oddities of float which are brutally painful to debug in large models.

[TysonMN]

SignInsenstiveComparer orders from smallest to largest magnitude, so this is the order that will be used. I also has the correct sum.

> 0.001  - 0.001 + 1_000_000_000_000.0 - 1_000_000_000_000.0;;
val it : float = 0.0

Is there any reason why smallest to largest magnitude is better than largest to smallest? I can't think of any.

---

While summing, the magnitude of the intermediate sum can become arbitrarily larger than the values being added to it. This can be avoided though.

Instead of direcrectly summing the sequence, a min heap can be used. Obtain the two smallest values (in magnitude) from the heap, sum them, insert this value back into the heap. Repeat until the heap contains one value, which is the sum.

Directly summing the values takes linear (aka n) time while the heap approach takes n log n time. The previous sorting step also takes n log n, so the runtime is not asymptotically worse.

I wonder if it possible for a sequence of floats to have a nonzero sum using the current linear approach and a zero sum using the heap approach. Even if it were possible, I expect it would be extremely rare in practice. I will give it a try.

[matthewcrews]

Is there any reason why smallest to largest magnitude is better than largest to smallest?

I have not observed it making a difference. The key was to get numbers close in magnitude added together before adding numbers in a significantly different order of magnitude.

The min heap approach sounds interesting!

I wonder if it possible for a sequence of floats to have a nonzero sum using the current linear approach and a zero sum using the heap approach. Even if it were possible, I expect it would be extremely rare in practice.

I agree it would be rare but I welcome alternative approaches. They key thing is to get float to behave in the way you would expect from a mathematical sense (things should cancel each other out, no matter the order).

[TysonMN]

People here all agree that smallest to larger magnitude is better. The person in this answer also describes an algorithm essentially the same as the heap one I mentioned.

It seems obvious in hindsight that summing a list of floating-point numbers would be a commonly studied problem. An often mentioned algorithm in this context is Kahan's algorithm.

I will keep reading.

[TysonMN]

I get the impression that summing floats with mixed signs should be achieved by summing the positives and negatives separately.

[TysonMN]

A dependency to consider using is HPCsharp, which implements Kahan's algorithm.

[matthewcrews]

I'll check it out

[TysonMN]

I intended to as well. To help keep life simple, I prefer to one thing per open source project at a time. After PR #161 is complete, I make a PR that refactors the code involving sorting including some additional tests. After that PR is complete, I will look into that library.

[matthewcrews]

So, I am open to adding this dependency provided it doesn't add significantly to the project size. I think I'd rather it go into the dev branch right now as that is where most of development efforts are going and I believe the algorithm in main is sufficient for the moment.

I'm open to thoughts though. Technically it would be fine if it went into main because it's not changing the API. I'm just cognizant of the fact that dev needs to be rebased onto main and that is going to be a little painful 🤣. I'm not a git wizard so hopefully it doesn't get too messy.

[TysonMN]

I'm just cognizant of the fact that dev needs to be rebased onto main and that is going to be a little painful 🤣.

Indeed. That is why I asked if I could make changes in master instead of dev in #138 (comment). It is easier for me to work in master but it will make things harder for you and @smoothdeveloper in dev. I recommend frequent merges of master into dev. (Normally I prefer rebasing instead of merging, but I think merging will be less painful, which is important in this case).

I decided to close issue #138 (which is about a simple improvement to the indentation of the tests) because I don't want to increase the difference with dev without sufficient reason. In contrast, I am not sure that I would have found bug #148 (and bug #153) if not for the code clean up I contributed in PR #147. For that reason, I expect to eventually contribute more code clean up PRs.

I'm not a git wizard so hopefully it doesn't get too messy.

I am willing to help with this. Sometimes at work, we resolve merge conflicts together (like while doing a video chat). I would be up for that as well.

[matthewcrews]

I will definitely reach out to you. I do most of my git work with Visual Studio/ VS Code.

[TysonMN]

I am open to adding this dependency provided it doesn't add significantly to the project size

The difference in program size is easy to check. My goal is to add tests that show the difference between the current behavior and the behavior of alternative approaches like the implementation of Kahan's algorithm in HPCsharp. On the one hand, there will be tests that pass or fail with the current approach or some alternative approach. On the other hand, I will use BenchmarkDotNet to create a report about the differences in runtime.

[TysonMN]

I do most of my git work with Visual Studio/ VS Code.

I highly recommend GitKraken. The merge tool in its pro version is the best.

[matthewcrews]

I am all for great tools! I'll check it out!

…

On Mon, Jan 4, 2021 at 12:22 PM Tyson Williams ***@***.***> wrote: I do most of my git work with Visual Studio/ VS Code. I highly recommend GitKraken <https://www.gitkraken.com/>. The merge tool in its pro version is the best. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#150 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAME677FTD5UEFUAOS2DWEDSYH2L5ANCNFSM4VPDLBEA> .

[TysonMN]

I did some more investigation. A technical read on the subject is Chapter 4: Summation in Accuracy and Stability of Numerical Algorithms. Section 6 describes four general use cases and what summation algorithms best handle those cases. Before reading this, the impression I got is that Kahan's algorithm is typically considered best. My impression is reinforced by this chapter especially because algorithms more advanced than Kahan's were only mentioned briefly and not reconsider in section 6.

Special cases in which other algorithms perform better than Kahan's include when the summands all have the same sign or when large cancellations are expected. I don't think either of these are typically the case in Flips.

For these reasons, I feel very good about using Kahan's algorithm in Flips. At this time, I only know of one NuGet package containing an implemention of Kahan's algorithm, which is HPCsharp. In general, I think it would be better to add a dependency that implements Kahan's algorithm than to directionly implement Kahan's algorithm in Flips.

My goal is to add tests that show the difference between the current behavior and the behavior of alternative approaches like the implementation of Kahan's algorithm in HPCsharp. On the one hand, there will be tests that pass or fail with the current approach or some alternative approach. On the other hand, I will use BenchmarkDotNet to create a report about the differences in runtime.

Now that my confidence in Kahan's algorithm has significantly increased because of that book chapter, I am less concerned with adding accuracy tests and performance tests.

I am open to adding this dependency provided it doesn't add significantly to the project size

@matthewcrews and @smoothdeveloper, other than project size, is there anything else you would like to see before switching to floating-point summation via Kahan's algorithm from a NuGet package?

[matthewcrews]

I was provided a simple implementation of the Kahan Algorithm by a fellow F# dev that I will be using to address this. This will potentially speed up the reduction of LinearExpression which is the hottest piece of code in the library.

[matthewcrews]

I am likely going to close this. I tried the Kahan's algorithm and the accuracy is not as good as what we currently have. It is uncommon for these summations to be made up of many floats so the speed gain from using Kahan's algorithm is minimal. Unless I see a real-world use case where the bottleneck is these summations, I would prefer to keep the current solution in place since it behaves more in line with expectations around the canceling of terms.

You can see this PR and the fact that the tests start to fail due to the summation not being as accurate:
#168

I may try the recommended package and see how that does.

[TysonMN]

I was provided a simple implementation of the Kahan Algorithm by a fellow F# dev that I will be using to address this. This will potentially speed up the reduction of LinearExpression which is the hottest piece of code in the library.
[...]
I tried the Kahan's algorithm and the accuracy is not as good as what we currently have.

I do not recommend trying to write this yourself (even with help from a friend). This kind of code is extremely difficult to get right.

Instead, I think using an existing library is a better choice. The only one I found in my searches is HPCsharp. I forked the branch in your PR to use the most accurate summation from HPCsharp. See PR #170.

[matthewcrews]

Excellent. I was going to try the HPCsharp library and go from there. I'm doing this work on the dev branch right now so I won't be using PR #170 and instead will update #168