yhirose / cpp-fstlib Goto Github PK

In current condition, state_output will not add up to final output in current frame
I try to fix like this:

        if (ope.data.final && atm.is_match()) {
            if (prefix.empty() || (prefix.size() == 1 && prefix.front() == arc)) {
                if (OutputTraits<output_t>::type() == OutputType::none_t ||
                        OutputTraits<output_t>::empty(state_output)) {
                    accept(word, output);
                } else {
                    accept(word, output + state_output);
                }
            }
        }

I use API named common_prefix_search but the result is not what i want.
I use the example in tutorials.
for example:

// input
const std::vector<std::pair<std::string, std::string>> items = {
  {"hello", "こんにちは!"},
  {"hello world", "こんにちは世界!"}, // incorrect sort order entry...
  {"world", "世界!"},
};
//Omit the initialization process for fstmap; 
//....

//search result with prefix 'h' but no result returned.
auto prefixes = matcher.common_prefix_search("h");

//output=0;
cout << prefixes.size() << endl;  

I expect the result to be “こんにちは!” and “こんにちは世界!”.
Which interface can get the expected result? And what does 'common_prefix_search' mean?

when too many terms（billions of），memory usage is huge（servel GB）； Lucene uses pagecache to lazy load fst bytes，is it possible for fstlib？

Hello, I am diving into let the fstlib support uint64_t type. When I simply change the classes instantiated for std::string and uint32_t types to uint64_t, some error are produced in test file. Can you give some hint that where should I make changes if I want to support uint64_t type?

const std::vector<std::pair<std::string, std::string>> items =
{
{"hello", "a"},
{"hello world", "b"},
{"world", "c"}
};

std::stringstream out;
auto sorted = false; // ask fst::compile to sort entries
auto [result, error_line] = fst::compilestd::string(items, out, sorted);

const auto &byte_code = out.str();
fst::mapstd::string matcher(byte_code.data(), byte_code.size());

//do something
****** here, I have more items to add, how to do that ? ******

hash function use a buffer 1024， is it enough for large of data