[Query] to read a large text file with delimited of size 75 GB about stringzilla

[ashvardanian]

Hi, @crbsram, thanks for reaching out! Everything depends on the complexity of your checks, but I don't see a reason why this wouldn't work with StringZilla.

How long are your rows on average?
Does your CPU support AVX-512?

[crbsram]

Hi Ash
every line is of 3000 characters (max) and for every line i will have to scan the entire 500mil lines
and CPU is Intel Xeon Gold 6330.
Every line is delimited with special character. For every line we will have to iterate it in all 500 million in quicker way.
Henceforth the ask is is stringzilla will help us to achieve this feat by leveraging the capabilities of modern hardware.

Thanks
C.R.Bala

[ashvardanian]

Yes, @crbsram! This sounds like a good case for StringZilla. I'll have to warn you that even with StringZilla 5e6 x 5e6 pairwise string operations won't be very fast. I recommend setting up a smaller benchmark first. Let me know if this works out.

What is the larger goal you are trying to achieve? Is that search?

[crbsram]

Hi @ashvardanian,
Then let me setup a smaller bench mark with 1mil lines and 10 mil lines and provide the result.
THe larger goal is to find string similarity between two strings using trigram method
Hence we don't have choice other than iterating the entire lines in a file.

Thanks
C.R.Bala

[ashvardanian]

Hi @crbsram! How is it going? Any preliminary results?