Split on multiple characters about stringzilla

[ashvardanian]

Sure, @Zabrane! Here is the C++ code for the benchmark, but it might be a bit convoluted. The function is pretty well documented in the C header, it might be easier to start there.

[ashvardanian]

Hi @luluna02! Thanks for participation! The baseline serial and AVX-512 implementations are present, but there is a lot of space for improvement. All depends on your experience and technical expertise. If you look at the current AVX-512 variant it uses Galois Field operations implemented in hardware and available with GFNI extensions. That helps us emulate multi-shift operation at the level of 8-bit words of the ZMM register. Changing the multiplication argument there can save us several CPU cycles, but it might be a hard place to start.

[ashvardanian]

Yes, this can be a nice feature. What would be a good interface for that? How about this?

text.split(separators='abcd')

I was also considering adding specialized SWAR splitting functionality for CSV rows... I'd have to skip the escape symbols and support separators of different length: ,, ",, ,", ",".

[chris-ha458]

Yeah that is an important concern.
Considering the nature of separators (ascii punctuation, unicode punctuation, ascii whitespace, unicode whitespace, control characters that are often confused by libraries to be either punctuation or whitespace etc)
I think it would be necessary to have a robust set of testing before a feature like this is merged.

One way to help canonicalize the string would be to only accept raw strings r" " but in that case escapes are handled to be literals (so r"\n" is definitely \ and n) which might be difficult to handle as well

[ashvardanian]

@chris-ha458 I've added this functionality, and it will be coming with the v3 major release. With AVX-512 there is a noticeable uplift compared to serial code.

Parsed the file with 8388608 words of 5 mean length!
Benchmarking for whitespaces:
- std::string_view.find_first_of          0.2617 GB/s
- sz_find_from_set_serial          0.2548 GB/s
- sz_find_from_set_avx512          0.4394 GB/s
- std::string_view.find_last_of          0.2861 GB/s
- sz_find_last_from_set_serial          0.2604 GB/s
- sz_find_last_from_set_avx512          0.4402 GB/s
Benchmarking for punctuation marks:
- std::string_view.find_first_of          0.5188 GB/s
- sz_find_from_set_serial          0.5265 GB/s
- sz_find_from_set_avx512          0.6465 GB/s
- std::string_view.find_last_of          0.5308 GB/s
- sz_find_last_from_set_serial          0.5330 GB/s
- sz_find_last_from_set_avx512          0.7384 GB/s

[chris-ha458]

This would be helpful for tokenizers or anything relying on tokenizers, which is an important use case for string manipulation

[chris-ha458]

I would think the best interface would be taking a list.
For instance if we take multi character strings as a list it becomes ambiguous how to handle
"\n"(newline achieved by escape but also technically '' + 'n') vs str(b"0x0A") (actual bit representation for ASCII newline).
Althought the pitfalls still exist for lists like ['\n','','n',str(b'0x0A')], it becomes less ambiguous imo.
(So..many..pitfalls especially how python can silently coerce a string into a list and vice versa)

[ashvardanian]

From UX perspective - you are right. From performance perspective, depending on the time of the day the interpreter may spend a lot of time building those lists… strings might be a cheaper option.

[Zabrane]

@chris-ha458 I've added this functionality, and it will be coming with the v3 major release. With AVX-512 there is a noticeable uplift compared to serial code.

Parsed the file with 8388608 words of 5 mean length!
Benchmarking for whitespaces:
- std::string_view.find_first_of          0.2617 GB/s
- sz_find_from_set_serial          0.2548 GB/s
- sz_find_from_set_avx512          0.4394 GB/s
- std::string_view.find_last_of          0.2861 GB/s
- sz_find_last_from_set_serial          0.2604 GB/s
- sz_find_last_from_set_avx512          0.4402 GB/s
Benchmarking for punctuation marks:
- std::string_view.find_first_of          0.5188 GB/s
- sz_find_from_set_serial          0.5265 GB/s
- sz_find_from_set_avx512          0.6465 GB/s
- std::string_view.find_last_of          0.5308 GB/s
- sz_find_last_from_set_serial          0.5330 GB/s
- sz_find_last_from_set_avx512          0.7384 GB/s

@ashvardanian amazing. I'm also interested on this.
Is there a C example I can look at to learn how to use it please?

[luluna02]

Hello @ashvardanian, I find this project really interesting (good job!!) and I want to contribute.
I'm curious if this issue is closed yet?