UTF-8 trouble with parserPopup.pl in certain settings (may apply to other MathObjects with UTF-8 string answers) about pg

[taniwallach]

Additional notes:

1. When MathJax is used to display equations, the "formatted" answer column contains an MathJax object which has the UTF-8 (Hebrew) text selected in the Popup displayed properly.

2. A possible root cause seems to be in lib/Value/String.pm:

• It seems likely that the problem is that there is either an extra or a missing conversion to UTF-8 being applied to the string being processes before it reaches it's final destination.
• Note also: The quoteTeX subroutine is triggering the use of $self->verb($s) whenever a UTF-8 character (or anything else not in a very short list of allowed characters is detected):

sub quoteTeX {
  my $self = shift; my $s = shift;
  return $self->verb($s) unless $s =~ m/^[-a-z0-9 ,.;:+=?()\[\]]*$/i;
  "\\text{$s}";
}

and it seems that with UTF-8 support added, many more characters could be handled by \text{} without the need for \verb. Finding the correct condition to use may be challenging.

[dpvc]

The image mode issue is due to the fact that md5_hex() (used in EquationCache.pm line 89) can only accept characters with numbers up to 255 (see the last section of the MD5 documentation just above the "See Also".

One solution is to use

my $md5 = md5_hex(utf8::is_utf8($tex) ? Encode::encode_utf8($tex) : $tex);

rather than

my $md5 = md5_hex($tex);

at line 89.

I assume that for the html2xml issue, you need to convert the non-ASCII unicode characters to their &#xXXXX; format, so that U+1234 would become ሴ.

I do not think this is a MathObject problem per se.

[taniwallach]

#466 should be fixing the issue with EquationCache.pm, while openwebwork/webwork2#1089 should have fixed the issue with XMLRPC/html2xml.
Thus this issues seems to be resolved by the merged/in review PRs.

I'm closing this issue.