basis_checker_columns() from pg/macros/MatrixCheckers.pl is not as dependable as would be desired about pg

[paultpearson]

Hi! Paul Pearson here. I'm glad someone is looking into these issues, as they were known issues when I hurriedly wrote the MatrixCheckers.pl macro and the solution I chose was suboptimal. I found that moderate tolerances seemed to work well enough, but evidently I didn't test that thoroughly enough because you've identified issues with the tolerances. I don't have time right now to work on this. I encourage you to work with Nandor, Davide, and others to fix the issues and merge them into the codebase. Thank you for all of your help with this.

[taniwallach]

@paultpearson - Thanks for the comment about the history, etc.

The issue certainly relates to tolerance issues, and since it seems to be more likely to accept incorrect answers (of a hopefully relatively unusual structure) than to reject incorrect ones, I suspect that few students are likely to have noticed or complained.

Further testing with the test problem from the prior message triggers the problem for the answers: (0,1,-1,-1) and (01,1,-1,-1+0.002), so the problem does not really depend on using an ugly multiple of the first basis vector. (I discovered the issue when testing how code was handling such uglier answers.)

After some additional thought on the issue, I think that the root cause of the bug is how the current code checks that the student's vectors are in the space. The existing code:

      $CTC = ($C->transpose) * $C;
      # S = student, C = correct, X = change of basis matrix
      # Solve S = CX for X using (C^T C)^{-1} C^T S = X.
      $X = ($CTC->inverse) * (($C->transpose) * $S);
      return $S == $C * $X;

calculates a "change of basis matrix $X" and then checks (in the fuzzy sense) that the student's vectors match $C * $X. Given this approach, it seems that any vector sufficiently close to a vector in the space will be able to pass the test. Given the relatively large tolerance used in the existing basis_checker_columns() this is occurring far too easily, in my opinion. It could be that much of the trouble could be avoided if the tolerance, at least for the final comparison, were far smaller.

The proposed code uses "fuzzy" rank calculations with a smaller tolerance to test (in advance) whether each of the student's vectors are in the given space. So far as I can tell, it is far more accurate it detecting vectors which do not belong to the given space.

However, the proposed fuzzy rank code was not sufficiently accurate in detecting linear dependence among the student vectors is some cases, so the proposed code relies on the determinant based test from the existing code as a fall back measure. The fall back code does not seem to be triggered very frequently, but it did catch some things which the rank based code did not. I'm not certain that the fallback code really has much of a practical effect, but I recall one test where without it the number of independent vectors belong to the space was reported incorrectly, as the shift was small enough for the shifted vector to be considered to be in the space while large enough for it to be considered independent of the unshifted vector. This issue led to my decision not to report linear dependence of the submitted answers when some vector in the answer was detected not to be in the space.

Below is a second test problem for a basis of size 3, where the same sort of problem occurs - if the last vector in the answer is shifted a bit (not too much, not too little) from the sum of the first 2 (correct) basis vectors, the incorrect answer will be accepted.

---

test_CustomBasisChecker2.pg follows

# Test problem to test/debug issues I had with basis_checker_columns()
# from pg/macros/MatrixCheckers.pl. 

DOCUMENT();        # This should be the first executable line in the problem.

loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
  "parserMultiAnswer.pl",
  #"MatrixCheckers.pl",
  "rank.pl",
  "CustomBasisChecker1.pl",	# includes a slightly modified version of basis_checker_columns 
				# and a redesigned basis_checker_columns_tani which depends on
				# the new rank.pl macro
);

TEXT(beginproblem());

$showPartialCorrectAnswers = 1;
$showPartialCredit = 1;

Context('Matrix');

$vec1 = Matrix([[ 1,0,0,1 ]])->transpose;
$vec2 = Matrix([[ 0,1,0,1 ]])->transpose;
$vec3 = Matrix([[ 0,0,1,1 ]])->transpose;

$vec4 = Matrix([[ 1.002,0.999,0,2 ]])->transpose;

$multians1 = MultiAnswer($vec1, $vec2, $vec3)->with(
  singleResult => 1,
  separator => ',',
  tex_separator => ',',
  allowBlankAnswers=>0,
  checker => ~~&basis_checker_columns_tani,
#  checker => ~~&basis_checker_columns, # Had issues fixed by the _tani version
);

Context()->texStrings;
BEGIN_TEXT

Try this problem with both of the options for the checker function.
$PAR

The original checker is expected to ${BBOLD}incorrectly${EBOLD} accept \( $vec1, $vec2, $vec4 \).

$PAR

Find a basis \( \mathcal{B} \) of \( \text{Span}\left\lbrace  $vec1, $vec2, $vec3 \right\rbrace \).
$BR


$BCENTER
\( \mathcal{B} = \left\lbrace \;\; \rule{0pt}{50pt} \right. \;\; b_1 = \)
\{ $multians1->ans_array(2) \}
\( , \;\; b_2 = \)
\{ $multians1->ans_array(2) \}
\( , \;\; b_3 = \)
\{ $multians1->ans_array(25) \}
\( \left. \;\; \rule{0pt}{50pt} \;\; \right\rbrace \)
$ECENTER

END_TEXT
Context()->normalStrings;

ANS( $multians1->cmp() );

ENDDOCUMENT();        # This should be the last executable line in the problem.

[taniwallach]

I do hope that I will be able to recruit someone to help test the new code in addition to me before we embark on changing the code in the main pg/macros library, as that will have very wide consequences as soon a people would upgrade pg on their servers.

It should be pretty to use CustomBasisChecker1.pl as a local replacement for MatrixCheckers.pl in any problem currently using basis_checker_columns and to swap between using basis_checker_columns and basis_checker_columns_tani for testing.

On the other hand, not rolling out a fix does leave the problematic code pretty widely used in a number of OPL linear algebra problems, which I think is not an ideal situation.

@paultpearson - Do you think you might be able to help with looking into this during the summer?

@mgage - Can you think of other people who have worked on the linear algebra material who might be willing to help look into this issue and test the proposed code?