`reference/test/solver/bicgstab_kernels.cpp` tests fail when converted to `float` (i.e., single precision) about ginkgo

[gflegar]

The problem here could be one of the following:

• There's something wrong with the implementation of BiCGStab.
• There's a problem with how we test. The number 1e-15 means "reduce the original residual by 15 orders of magnitude", not "obtain accuracy of 1e-15". Thus, if the original residual was to large (which we don't test anywhere), then the single precision version could might as well reduce the original residual by 7 orders of magnitude, and still get a solution which only has 1-3 correct digits. I think it would be more useful if we use a hard limit on residual norm for tests, but we have to implement different stopping criteria to do that.

[gflegar]

Btw, there's an issue on the private repo about implementing tese stopping criteria already: https://gitlab.com/ginkgo-project/ginkgo/issues/60 Maybe we should consider moving it into the public repo.

[venovako]

Sorry if I misunderstood the parameters.
Please also note that the test with one RHS (SolvesDenseSystem) succeeds in float, if that may indicate you something.

[gflegar]

Sorry if I misunderstood the parameters.

Even if you did, it's not an issue here. There's still something wrong with the tests.

Please also note that the test with one RHS (SolvesDenseSystem) succeeds in float, if that may indicate you something.

I noticed, and the explanation is probably this: https://gitlab.com/ginkgo-project/ginkgo/issues/55
TL; DR The convergence test is not exactly the same for multiple RHS as is for one RHS. Another thing we have to fix.

[tcojean]

Yes, actually I remember now that we saw a similar problem for double when I integrated that code, and we had to settle with the current parameters for it to work. But we need to tackle the related issue in the private repo to make this work properly in the general case.

[gflegar]

@tcojean, @hartwiganzt now that we have stopping criteria - maybe we should take another look into this?

[tcojean]

I just checked this again. It looks like we have now fixed this problem in the meantime.

Result:

$ ./reference/test/solver/bicgstab_kernels
Running main() from /home/tcojean/ginkgo-github/build/third_party/gtest/src/googletest/src/gtest_main.cc
[==========] Running 7 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 7 tests from Bicgstab
[ RUN      ] Bicgstab.SolvesDenseSystem
[       OK ] Bicgstab.SolvesDenseSystem (0 ms)
[ RUN      ] Bicgstab.SolvesMultipleDenseSystems
[       OK ] Bicgstab.SolvesMultipleDenseSystems (0 ms)
[ RUN      ] Bicgstab.SolvesDenseSystemUsingAdvancedApply
[       OK ] Bicgstab.SolvesDenseSystemUsingAdvancedApply (1 ms)
[ RUN      ] Bicgstab.SolvesMultipleDenseSystemsUsingAdvancedApply
[       OK ] Bicgstab.SolvesMultipleDenseSystemsUsingAdvancedApply (0 ms)
[ RUN      ] Bicgstab.SolvesBigDenseSystemForDivergenceCheck1
[       OK ] Bicgstab.SolvesBigDenseSystemForDivergenceCheck1 (0 ms)
[ RUN      ] Bicgstab.SolvesBigDenseSystemForDivergenceCheck2
[       OK ] Bicgstab.SolvesBigDenseSystemForDivergenceCheck2 (1 ms)
[ RUN      ] Bicgstab.SolvesMultipleDenseSystemsDivergenceCheck
[       OK ] Bicgstab.SolvesMultipleDenseSystemsDivergenceCheck (0 ms)
[----------] 7 tests from Bicgstab (2 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 1 test case ran. (3 ms total)
[  PASSED  ] 7 tests.

Patch I used:

diff --git a/reference/test/solver/bicgstab_kernels.cpp b/reference/test/solver/bicgstab_kernels.cpp
index bb89d71f..bf37f9b0 100644
--- a/reference/test/solver/bicgstab_kernels.cpp
+++ b/reference/test/solver/bicgstab_kernels.cpp
@@ -52,8 +52,8 @@ namespace {

 class Bicgstab : public ::testing::Test {
 protected:
-    using Mtx = gko::matrix::Dense<>;
-    using Solver = gko::solver::Bicgstab<>;
+    using Mtx = gko::matrix::Dense<float>;
+    using Solver = gko::solver::Bicgstab<float>;

     Bicgstab()
         : exec(gko::ReferenceExecutor::create()),
@@ -66,28 +66,28 @@ protected:
                       gko::stop::Time::build()
                           .with_time_limit(std::chrono::seconds(6))
                           .on(exec),
-                      gko::stop::ResidualNormReduction<>::build()
-                          .with_reduction_factor(1e-15)
+                      gko::stop::ResidualNormReduction<float>::build()
+                          .with_reduction_factor(1e-7)
                           .on(exec))
                   .on(exec)),
           bicgstab_factory_precision(
-              gko::solver::Bicgstab<>::build()
+              gko::solver::Bicgstab<float>::build()
                   .with_criteria(
                       gko::stop::Iteration::build().with_max_iters(50u).on(
                           exec),
                       gko::stop::Time::build()
                           .with_time_limit(std::chrono::seconds(6))
                           .on(exec),
-                      gko::stop::ResidualNormReduction<>::build()
-                          .with_reduction_factor(1e-15)
+                      gko::stop::ResidualNormReduction<float>::build()
+                          .with_reduction_factor(1e-7)
                           .on(exec))
                   .on(exec))
     {}

     std::shared_ptr<const gko::Executor> exec;
     std::shared_ptr<Mtx> mtx;
-    std::unique_ptr<gko::solver::Bicgstab<>::Factory> bicgstab_factory;
-    std::unique_ptr<gko::solver::Bicgstab<>::Factory>
+    std::unique_ptr<gko::solver::Bicgstab<float>::Factory> bicgstab_factory;
+    std::unique_ptr<gko::solver::Bicgstab<float>::Factory>
         bicgstab_factory_precision;
 };

@@ -100,7 +100,7 @@ TEST_F(Bicgstab, SolvesDenseSystem)

     solver->apply(b.get(), x.get());

-    ASSERT_MTX_NEAR(x, l({-4.0, -1.0, 4.0}), 1e-8);
+    ASSERT_MTX_NEAR(x, l({-4.0, -1.0, 4.0}), 1e-4);
 }


@@ -113,7 +113,7 @@ TEST_F(Bicgstab, SolvesMultipleDenseSystems)

     solver->apply(b.get(), x.get());

-    ASSERT_MTX_NEAR(x, l({{-4.0, 1.0}, {-1.0, 2.0}, {4.0, -1.0}}), 1e-8);
+    ASSERT_MTX_NEAR(x, l({{-4.0, 1.0}, {-1.0, 2.0}, {4.0, -1.0}}), 1e-4);
 }


@@ -128,7 +128,7 @@ TEST_F(Bicgstab, SolvesDenseSystemUsingAdvancedApply)
     solver->apply(alpha.get(), b.get(), beta.get(), x.get());


-    ASSERT_MTX_NEAR(x, l({-8.5, -3.0, 6.0}), 1e-8);
+    ASSERT_MTX_NEAR(x, l({-8.5, -3.0, 6.0}), 1e-4);
 }


@@ -144,7 +144,7 @@ TEST_F(Bicgstab, SolvesMultipleDenseSystemsUsingAdvancedApply)
     solver->apply(alpha.get(), b.get(), beta.get(), x.get());


-    ASSERT_MTX_NEAR(x, l({{-8.5, 1.0}, {-3.0, 2.0}, {6.0, -5.0}}), 1e-8);
+    ASSERT_MTX_NEAR(x, l({{-8.5, 1.0}, {-3.0, 2.0}, {6.0, -5.0}}), 1e-4);
 }


@@ -169,7 +169,7 @@ TEST_F(Bicgstab, SolvesBigDenseSystemForDivergenceCheck1)
         x,
         l({0.13853406350816114, -0.08147485210505287, -0.0450299311807042,
            -0.0051264177562865719, 0.11609654300797841, 0.1018688746740561}),
-        1e-9);
+        1e-4);
 }


@@ -193,16 +193,17 @@ TEST_F(Bicgstab, SolvesBigDenseSystemForDivergenceCheck2)
         x,
         l({0.13517641417299162, 0.75117689075221139, 0.47572853185155239,
            -0.50927993095367852, 0.13463333820848167, 0.23126768306576015}),
-        1e-9);
+        1e-4);
 }


-double infNorm(gko::matrix::Dense<> *mat, size_t col = 0)
+template <typename T>
+T infNorm(gko::matrix::Dense<T> *mat, size_t col = 0)
 {
     using std::abs;
-    double norm = 0.0;
+    T norm = 0.0;
     for (size_t i = 0; i < mat->get_size()[0]; ++i) {
-        double absEntry = abs(mat->at(i, col));
+        T absEntry = abs(mat->at(i, col));
         if (norm < absEntry) norm = absEntry;
     }
     return norm;
@@ -273,7 +274,7 @@ TEST_F(Bicgstab, SolvesMultipleDenseSystemsDivergenceCheck)

     // Not sure if this is necessary, the assertions above should cover what is
     // needed.
-    ASSERT_MTX_NEAR(xc, testMtx, 1e-14);
+    ASSERT_MTX_NEAR(xc, testMtx, 1e-4);
 }