hash buff size enough? about cpp-fstlib

[yhirose]

@mtxrym, so far, I don't experience any major hash collision with my test data files. Please let me know if you discover a problem.

[mtxrym]

it's not about hash collision; transitions size is larger than 1024, buff is not large enough, segment fault

[yhirose]

@mtxrym, thanks for more detailed information. Fortunately, the transition size never goes beyond 256, because arc is char and the transitions cannot have the same arc entry more than once in fst structure.

But buff also includes id (uint32) and output (value size depends on output_t) and state_output (same as output). It might exceed 1024 when transitions contain many arcs with long string output though, it's a rare case with usual word dictionaries I think. Just in case it happens, I'll make a change to throw an exception when data exceeds the buff size. Thanks for focusing on that.

One question. Did you really experience that situation? If so, what kind of data did you use?

[mtxrym]

just terms with averge 30~40bytes , largest 100 bytes(with chinese character), 400w together to compile, it encounters coredump with buf_len is 1058, exceed 1024

[yhirose]

What does it mean by 400w together to compile? Also in your case, how much buffer size is needed for your data, 2048, 4096?

[mtxrym]

it seems 2048 is enough；
fst::compile<uint32_t>(items, m_term_str, true)
items is vector size ablout 400w

[yhirose]

@mtxrym, thanks for the more information. I'll make a necessary change to fix it soon. Thanks!

[yhirose]

@mtxrym, the current master doesn't have this bug anymore. Thanks for your contribution!