elshize / irkit Goto Github PK
View Code? Open in Web Editor NEWInformation Retrieval tools intended for academic research.
Home Page: https://elshize.github.io/irkit/
License: MIT License
Information Retrieval tools intended for academic research.
Home Page: https://elshize.github.io/irkit/
License: MIT License
Note: the list isn't final.
Here's a rough list of the functionalities to implement before releasing 0.1:
Might improve efficiency, not sure how much. Not enough to care right now, but maybe enough when running experiments for a paper.
The integration test fails on terms.txt
, possibly due to some UTF-8/16 characters. I'm not sure how that should affect it since everything is coded byte by byte but that's the only idea I have so far.
Note that hutucker_codec works but haven't tested prefix encoding, that might be the first step.
The implementation of varbyte_codec
looks awful. It needs to be much simpler and readable.
Also, a benchmark should be written to make sure that changes don't slow it down.
Mixing passing ranges by values and references might cause bugs (see #2).
A consistent way of passing argument must be designed to avoid these problems. Passing both containers and views should be seamless, ideally. But it must be efficient. I see a few options but not sure which is the best:
gsl::span
):{
"avg_document_size": 758,
"documents": 328644,
"max_document_size": 24083,
"occurrences": 249345847,
"shards": 123
}
irtl
.irtl
.Investigate and turn on the sanitizers for tests again
By most, I mean anything that processes a single thing, like builds an index, partitions, scores, etc.
Change for irk-query
, irk-threshold
, and others.
For a chosen processing type. For now, only exhaustive:
For a given score or precomputed score.
nonstd::expected
is sometimes cumbersome to use. Whenever you require it to has value, certain boilerplate is involved. But why not replace it with a specific type of assert that aborts the program if the variable has unexpected value.
Should work for any printable unexpected type.
Example:
auto result = something_returning_expected().get_value_or_abort();
For now, index integration test should ensure it's correct. Nevertheless, this should be fixed.
Write irk-taily
tool that extracts Taily scores from an index cluster.
When scoring on-the-fly, it might matter how many look-ups to the size table is made, so implementing a DAAT-specific scoring---which scores entire document at once---might be a good idea.
However, I have doubts it matters much, as the intersection might not be big enough to make a difference.
Another reason for this could be finding out how precision changes if we score with a smoothed language model, and always consider all terms, even those for which a document doesn't exist in the posting list.
It will read/map a file and create a span of T
to it, and perform lookups, print, etc.
The below should be noexcept
but it fails on clang
on Travis (but not locally or moa
). Investigate and make it noexcept
if possible.
disk_memory_source(disk_memory_source&&) = default;
disk_memory_source& operator=(disk_memory_source&&) = default;
As opposed to reverse lookup, which does.
libstreamvbyte.so
There is a much better tool for doing this in bash: split
.
Offsets, such as those to document IDs, frequencies and scores, should ideally follow the following rules:
The question is: can we satisfy all of the above?
Note: ClueWeb09B has over 80 mln terms.
Looks like there's something wrong across block edges. See the integration test.
It's all correct until the first element of the second block.
block: 0
515 ; idx: 515
block: 1
515 ; idx: 516
It performs well for precomputed scores. Also, there is much smaller difference between dynamic and precomputed for TAAT.
Maybe implementing #61 will make difference?
moa:/data/index/irkit/cw09b-nospam
~/irkit/build/bin/irk-score
[2018-11-05 17:30:36.058] [score] [info] Initiating scoring using 8 threads
[2018-11-05 17:30:36.810] [score] [info] Calculating max score
[2018-11-05 17:45:45.419] [score] [info] Max score: 8.48841e+07; Min score: -1.09753e+07
malloc(): memory corruption
Aborted (core dumped)
Write a tool that takes WARC (possibly gzipped) files and print outs the documents.
Title<--->URL<--->size (bytes)<--->term term term...
-s/--stem lang
: use stemming in the selected languageA tool, e.g., irk-scorestats
, should take a parameter score=["bm25", "ql"]
, and compute, for each term, its max, mean, and variance, and store in files:
<score>.max
<score>.mean
<score>.var
For some reason, if I use find_package
to find zlib dependency within conan
packages, it fails despite linking against the library:
/usr/bin/g++-7 -Wall -pedantic -fno-strict-aliasing -march=native -O3 CMakeFiles/irk-warc.dir/irk-warc.cpp.o -o ../bin/irk-warc -Wl,-rpath,/home/travis/.conan/data/zlib/1.2.11/conan/stable/package/5246c0bd84cb3855ffc2a458086a0813344953bf/lib:/home/travis/.conan/data/TBB/2018_U5/conan/stable/package/160b4bac5177b0f9a5f4857d00317fe0862a8a02/lib:/home/travis/.conan/data/streamvbyte/master/elshize/testing/package/6ae331b72e7e265ca2a3d1d8246faf73aa030238/lib:/home/travis/.conan/data/gumbo-parser/1.0/elshize/stable/package/6ae331b72e7e265ca2a3d1d8246faf73aa030238/lib: /home/travis/.conan/data/zlib/1.2.11/conan/stable/package/5246c0bd84cb3855ffc2a458086a0813344953bf/lib/libz.so -lpthread /usr/lib/x86_64-linux-gnu/libboost_iostreams.a /usr/lib/x86_64-linux-gnu/libboost_regex.a /usr/lib/x86_64-linux-gnu/libboost_filesystem.a /usr/lib/x86_64-linux-gnu/libboost_system.a /home/travis/.conan/data/fmt/5.1.0/bincrafters/stable/package/66c5327ebdcecae0a01a863939964495fa019a06/lib/libfmt.a /home/travis/.conan/data/TBB/2018_U5/conan/stable/package/160b4bac5177b0f9a5f4857d00317fe0862a8a02/lib/libtbb.so /home/travis/.conan/data/rax/master/elshize/testing/package/6ae331b72e7e265ca2a3d1d8246faf73aa030238/lib/librax.a /home/travis/.conan/data/streamvbyte/master/elshize/testing/package/6ae331b72e7e265ca2a3d1d8246faf73aa030238/lib/libstreamvbyte.so /home/travis/.conan/data/gumbo-parser/1.0/elshize/stable/package/6ae331b72e7e265ca2a3d1d8246faf73aa030238/lib/libgumbo.so
Check out Travis build for details: https://travis-ci.org/elshize/irkit/builds/449604806
Segmentation fault occurs when compiling with -g
. Investigate and fix.
unit_test_standard_block_list.cpp
needs path fix and to be re-enabled.
Right now, for some reason, rax
has problems with sanitizers. Investigate and fix.
1, 2, and 3 should be about the same.
Same goes for 4 and 5, which will take twice the time (?) of 1.
6 will be slower than 5 but should be reasonable (probably should at some point find out what that means).
Remove include_directories
in favor of the modern approach with target_include_directories
. Figure out how it works and if we can 'inherit' it from irkit
target. If so, then find out what was wrong.
This needs to be further thought through, but I think shared_ptr
is a better way to manage object's lifetime.
Alternative: can it be done with a move semantics? Should it?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.