tencent / wwsearch Goto Github PK

能提供点python java的sdk嘛

https://my.oschina.net/u/169565/blog/3145404

../librocksdb.so.5.16.6: undefined reference to inflateInit2_' ../librocksdb.so.5.16.6: undefined reference to inflate'
../librocksdb.so.5.16.6: undefined reference to deflateSetDictionary' ../librocksdb.so.5.16.6: undefined reference to deflateInit2_'
../librocksdb.so.5.16.6: undefined reference to deflate' ../librocksdb.so.5.16.6: undefined reference to inflateSetDictionary'
../librocksdb.so.5.16.6: undefined reference to deflateEnd' ../librocksdb.so.5.16.6: undefined reference to inflateEnd'
collect2: error: ld returned 1 exit status
deps/rocksdb/tools/CMakeFiles/sst_dump.dir/build.make:97: recipe for target 'deps/rocksdb/tools/sst_dump' failed
make[2]: *** [deps/rocksdb/tools/sst_dump] Error 1
CMakeFiles/Makefile2:707: recipe for target 'deps/rocksdb/tools/CMakeFiles/sst_dump.dir/all' failed
make[1]: *** [deps/rocksdb/tools/CMakeFiles/sst_dump.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/wwsearch.dir/build.make:638: recipe for target 'CMakeFiles/wwsearch.dir/src/index_config.cpp.o' failed
make[2]: *** [CMakeFiles/wwsearch.dir/src/index_config.cpp.o] Error 1
CMakeFiles/wwsearch.dir/build.make:326: recipe for target 'CMakeFiles/wwsearch.dir/src/bool_weight.cpp.o' failed
make[2]: *** [CMakeFiles/wwsearch.dir/src/bool_weight.cpp.o] Error 1
[ 91%] Linking CXX executable write_stress
../librocksdb.so.5.16.6: undefined reference to inflateInit2_' ../librocksdb.so.5.16.6: undefined reference to inflate'
../librocksdb.so.5.16.6: undefined reference to deflateSetDictionary' ../librocksdb.so.5.16.6: undefined reference to deflateInit2_'
../librocksdb.so.5.16.6: undefined reference to deflate' ../librocksdb.so.5.16.6: undefined reference to inflateSetDictionary'
../librocksdb.so.5.16.6: undefined reference to deflateEnd' ../librocksdb.so.5.16.6: undefined reference to inflateEnd'
collect2: error: ld returned 1 exit status
deps/rocksdb/tools/CMakeFiles/rocksdb_undump.dir/build.make:97: recipe for target 'deps/rocksdb/tools/rocksdb_undump' failed
make[2]: *** [deps/rocksdb/tools/rocksdb_undump] Error 1
CMakeFiles/Makefile2:670: recipe for target 'deps/rocksdb/tools/CMakeFiles/rocksdb_undump.dir/all' failed
make[1]: *** [deps/rocksdb/tools/CMakeFiles/rocksdb_undump.dir/all] Error 2
CMakeFiles/wwsearch.dir/build.make:446: recipe for target 'CMakeFiles/wwsearch.dir/src/index_wrapper.cpp.o' failed
make[2]: *** [CMakeFiles/wwsearch.dir/src/index_wrapper.cpp.o] Error 1
../librocksdb.so.5.16.6: undefined reference to inflateInit2_' ../librocksdb.so.5.16.6: undefined reference to inflate'
../librocksdb.so.5.16.6: undefined reference to deflateSetDictionary' ../librocksdb.so.5.16.6: undefined reference to deflateInit2_'
../librocksdb.so.5.16.6: undefined reference to deflate' ../librocksdb.so.5.16.6: undefined reference to inflateSetDictionary'
../librocksdb.so.5.16.6: undefined reference to deflateEnd' ../librocksdb.so.5.16.6: undefined reference to inflateEnd'
collect2: error: ld returned 1 exit status
deps/rocksdb/tools/CMakeFiles/rocksdb_dump.dir/build.make:97: recipe for target 'deps/rocksdb/tools/rocksdb_dump' failed
make[2]: *** [deps/rocksdb/tools/rocksdb_dump] Error 1
CMakeFiles/Makefile2:633: recipe for target 'deps/rocksdb/tools/CMakeFiles/rocksdb_dump.dir/all' failed
make[1]: *** [deps/rocksdb/tools/CMakeFiles/rocksdb_dump.dir/all] Error 2
CMakeFiles/wwsearch.dir/build.make:758: recipe for target 'CMakeFiles/wwsearch.dir/src/virtual_db_rocks.cpp.o' failed
make[2]: *** [CMakeFiles/wwsearch.dir/src/virtual_db_rocks.cpp.o] Error 1
CMakeFiles/Makefile2:69: recipe for target 'CMakeFiles/wwsearch.dir/all' failed
make[1]: *** [CMakeFiles/wwsearch.dir/all] Error 2
../librocksdb.so.5.16.6: undefined reference to inflateInit2_' ../librocksdb.so.5.16.6: undefined reference to inflate'
../librocksdb.so.5.16.6: undefined reference to deflateSetDictionary' ../librocksdb.so.5.16.6: undefined reference to deflateInit2_'
../librocksdb.so.5.16.6: undefined reference to deflate' ../librocksdb.so.5.16.6: undefined reference to inflateSetDictionary'
../librocksdb.so.5.16.6: undefined reference to deflateEnd' ../librocksdb.so.5.16.6: undefined reference to inflateEnd'
collect2: error: ld returned 1 exit status
deps/rocksdb/tools/CMakeFiles/db_sanity_test.dir/build.make:97: recipe for target 'deps/rocksdb/tools/db_sanity_test' failed
make[2]: *** [deps/rocksdb/tools/db_sanity_test] Error 1
CMakeFiles/Makefile2:490: recipe for target 'deps/rocksdb/tools/CMakeFiles/db_sanity_test.dir/all' failed
make[1]: *** [deps/rocksdb/tools/CMakeFiles/db_sanity_test.dir/all] Error 2

I notice that for each term within any documents to be inserted, all posting list will be updated corresponding to the terms' suffix strings, it's a huge amount of IO operation for the inverted table.
For example:
[key, value] where value is a compressed form of doc id list.
Given huge amount of documents, the value would be very huge, say contain tens of millions of ids.
Suppose each document contains an average of M terms, each term has an average length of L, then for each document, it will need M*L key-value Update operations, and the value might be very huge. Although there exist write batch mechanism in RocksDB to alleviate the IO burden, it's still an expensive mechanism.

Have you made any benchmarks over huge corpus? I wonder how it could perform on both insert and query? Thank you~