utf8 match bug about hyperscan

[coytea]

Hi sdeber,

I am trying to reproduce this issue on my machine but I am not seeing the failure for pattern 2. Are you able to attach your test program to the issue (if it is not too big)?

It looks to me if both versions of the pattern end up simply looking for the byte string \xe6\xb5\x8b\xe8\xaf\x95.

I will look further into our code and seeing if I can spot any suspicious interactions between our UTF8 handling and our parsing of \Q\E.

Also, are you using version 4.5 of Hyperscan or an older release?

Thanks,

Alex

[sdeber]

hyperscan.txt

Somehow, it failed to compile the pattern with error code -4. BTW, I am using version 4.5

[coytea]

Thanks. It looks like HS_FLAG_CASELESS is required as well. I will track down what is going on with this.

[sdeber]

@coytea You are right. After I removed HS_FLAG_CASELESS, it actually parsed correctly and the matching also works

[coytea]

We tracked the issue to us incorrectly treating the contents of a \Q \E region as a sequence of bytes rather than a sequence of codepoints when in UTF8 mode. This resulted in the issue that you observed when the caseless flag was used.

We have pushed commits to the develop branch which fix the issue (a185be5 and d317d75). The fix will make it into the next release.

Thanks for spotting and reporting this bug.