SIGILL on convert_utf8_to_utf16le about simdutf

[lemire]

@dagorander Great. We will create a new release soon.

cc @anonrig

[lemire]

Patch release upcoming.

[dagorander]

Impressive turnaround time by literally any metric, thank you all for the effort.

(And as a personal note: this report constitutes my first contribution to a FOSS project in general and also first contribution to making OpenBSD better. So extra thanks for that. :) )

[lemire]

@easyaspi314 Right. So I have brought back your code:

#243

As I stress in the PR, you get your signature on the commit, for fairness.

[lemire]

This has been released.

@dagorander You help was appreciated.

[lemire]

@anonrig is pushing an update to node.js.

[clausecker]

Your CPU does not have AVX512VBMI2 and AVX512BITALG, which are required to run this kernel. Something must be up with our CPU capability detection routines.

This is interesting though; in theory, your CPU should have the required instructions. They don't show up in the dmesg log though.

[clausecker]

The disassembly is unfortunately fairly useless as your gdb is too old to support AVX-512 instructions.

[dagorander]

That is my bad, I accidentally used gdb command (old version 6.3 shipped with the system itself) instead of egdb. Proper dissassembly attached below. I have updated my initial report as well, and updated the core trace to the correct version of gdb as well. Apologies for that.

node_core_disassembled.txt

[lemire]

It is a tiger lake processor which supports AVX-512. We have a recent OpenBSD.

I am puzzled.

[clausecker]

This is quite interesting. The code crashes on the first AVX-512 instruction. Does perhaps OpenBSD not support AVX-512? Or is AVX-512 disabled in the BIOS?

@lemire Do you check if AVX-512 is actually supported by the OS or only if it is present in cpuid? The former can be done by checking with xgetbv if the OS has set up saving/restoring AVX-512 registers on context switch.

[dagorander]

For completeness, I took a dive into the UEFI on the laptop (it is a Framework), looking to see if there might be some setting somewhere that disables certain instructions. I could not find any such.

The current install began as 7.2 Release, then went to 7.2-current, then 7.3-current. So there shouldn't be much baggage in the install.

[lemire]

Your CPU does not have AVX512VBMI2 and AVX512BITALG

The CPU does have this support.

[clausecker]

@lemire I wrote that because the extensions don't show up in the dmesg output. Seems like OpenBSD didn't put the relevant bits in their dmesg parser yet.

[lemire]

@dagorander Can you do something for us? Make sure you have a recent cmake as well as a recent C++ compiler.

git clone https://github.com/simdutf/simdutf
cd simdutf
cmake -B build
cmake --build build
ctest   --test-dir build --verbose

What do you see?

[lemire]

@clausecker Is there any way to run OpenBSD on AWS?

Or to launch OpenBDS inside a Linux server?

[dagorander]

@lemire It looks like the system uses clang v13, so that's from '21. If that's not recent enough for this exercise, that might be an interesting data point itself, perhaps?

As a side note as well, I have sent a notice to the maintainer of the OpenBSD node package, just in case they might have input or relevant domain knowledge.

The steps you indicate result in this:

0% tests passed, 52 tests failed out of 52

Total Test time (real) =   0.47 sec

The following tests FAILED:
	  1 - amalgamation_demo (ILLEGAL)
	  2 - random_fuzzer (ILLEGAL)
	  3 - special_tests (ILLEGAL)
	  4 - validate_ascii_basic_tests (ILLEGAL)
	  5 - validate_ascii_with_errors_tests (ILLEGAL)
	  6 - bele_tests (ILLEGAL)
	  7 - validate_utf8_basic_tests (ILLEGAL)
	  8 - select_implementation (ILLEGAL)
	  9 - validate_utf8_brute_force_tests (ILLEGAL)
	 10 - validate_utf8_puzzler_tests (ILLEGAL)
	 11 - validate_utf8_with_errors_tests (ILLEGAL)
	 12 - validate_utf16le_basic_tests (ILLEGAL)
	 13 - validate_utf16be_basic_tests (ILLEGAL)
	 14 - validate_utf16le_with_errors_tests (ILLEGAL)
	 15 - validate_utf16be_with_errors_tests (ILLEGAL)
	 16 - validate_utf32_basic_tests (ILLEGAL)
	 17 - validate_utf32_with_errors_tests (ILLEGAL)
	 18 - convert_valid_utf8_to_utf16le_tests (ILLEGAL)
	 19 - convert_valid_utf8_to_utf16be_tests (ILLEGAL)
	 20 - convert_valid_utf8_to_utf32_tests (ILLEGAL)
	 21 - convert_utf8_to_utf16le_tests (ILLEGAL)
	 22 - convert_utf8_to_utf16be_tests (ILLEGAL)
	 23 - convert_utf8_to_utf16le_with_errors_tests (ILLEGAL)
	 24 - convert_utf8_to_utf16be_with_errors_tests (ILLEGAL)
	 25 - convert_utf8_to_utf32_tests (ILLEGAL)
	 26 - convert_utf8_to_utf32_with_errors_tests (ILLEGAL)
	 27 - convert_utf16le_to_utf8_tests (ILLEGAL)
	 28 - convert_utf16be_to_utf8_tests (ILLEGAL)
	 29 - convert_utf16le_to_utf8_with_errors_tests (ILLEGAL)
	 30 - convert_utf16be_to_utf8_with_errors_tests (ILLEGAL)
	 31 - convert_utf32_to_utf8_tests (ILLEGAL)
	 32 - convert_utf32_to_utf8_with_errors_tests (ILLEGAL)
	 33 - convert_utf32_to_utf16le_tests (ILLEGAL)
	 34 - convert_utf32_to_utf16be_tests (ILLEGAL)
	 35 - convert_utf32_to_utf16le_with_errors_tests (ILLEGAL)
	 36 - convert_utf32_to_utf16be_with_errors_tests (ILLEGAL)
	 37 - convert_valid_utf16le_to_utf8_tests (ILLEGAL)
	 38 - convert_valid_utf16be_to_utf8_tests (ILLEGAL)
	 39 - convert_valid_utf32_to_utf8_tests (ILLEGAL)
	 40 - convert_valid_utf32_to_utf16le_tests (ILLEGAL)
	 41 - convert_valid_utf32_to_utf16be_tests (ILLEGAL)
	 42 - convert_utf16le_to_utf32_tests (ILLEGAL)
	 43 - convert_utf16be_to_utf32_tests (ILLEGAL)
	 44 - convert_utf16le_to_utf32_with_errors_tests (ILLEGAL)
	 45 - convert_utf16be_to_utf32_with_errors_tests (ILLEGAL)
	 46 - convert_valid_utf16le_to_utf32_tests (ILLEGAL)
	 47 - convert_valid_utf16be_to_utf32_tests (ILLEGAL)
	 48 - count_utf8 (ILLEGAL)
	 49 - count_utf16le (ILLEGAL)
	 50 - count_utf16be (ILLEGAL)
	 51 - detect_encodings_tests (ILLEGAL)
	 52 - basic_fuzzer (ILLEGAL)
Errors while running CTest
Output from these tests are in: /home/daniel/Temp/simdutf/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[lemire]

No. I don't think that it is case where your software is too old.

[clausecker]

@lemire I don't know, I don't use AWS much. It does install just fine in standard VM software; perhaps you can just run it in a Linux box under KVM. There's also this:

https://chrispinnock.com/2021/05/18/openbsd-on-aws/

I think the problem is that OpenBSD doesn't have AVX-512 support and we don't check for this.

[clausecker]

See this answer for how to detect if the OS supports AVX-512: https://stackoverflow.com/a/72523150/417501

[lemire]

@dagorander Can you do one more for me...

git clone https://github.com/easyaspi314/simdutf
cd altsimdutf
git checkout allow_nehalem
cmake -B build
cmake --build build
ctest   --test-dir build --verbose

[lemire]

@clausecker We have a PR from @easyaspi314 that did additional checks. It broke simdutf for some users, but it might contain the necessary fix. If so, I could bring it back minus the components that caused problems.

That's why I am asking @dagorander to run it.

[dagorander]

@lemire I think I only need the last bit of this run. :)

100% tests passed, 0 tests failed out of 52

Total Test time (real) = 148.59 sec

Indeed, it does seem that the PR from @easyaspi314 contains a fix for this issue.

[clausecker]

Here's confirmation that OpenBSD does not support AVX-512: http://cvsweb.openbsd.org/src/sys/arch/amd64/amd64/mds.S?rev=1.4&content-type=text/x-cvsweb-markup

/* we don't support AVX512 yet */

[anonrig]

@dagorander Great. We will create a new release soon.

cc @anonrig

Upon new version, I'll update Node.js as well.

[easyaspi314]

If the only issue is macOS, then removing all the #ifdef __APPLE__ should fix things. However because of how I properly check for AVX-512, macOS will default to AVX2.

However, that will be 10000 times better than netbsd crashing.