femto-dev / femto Goto Github PK

Sequence Indexing and Search

License: GNU Lesser General Public License v3.0

Shell 0.12% C++ 38.61% C 32.29% C# 0.13% CSS 0.55% JavaScript 4.72% PHP 0.52% Makefile 0.36% Perl 0.57% Python 2.46% CMake 0.63% Yacc 0.10% Lex 0.23% Batchfile 0.01% HTML 18.55% M4 0.14%

bioinformatics compressed-suffix-array compression fm-index indexing information-retrieval search

femto's People

Contributors

Stargazers

Watchers

femto's Issues

Multiline Queries are going exponential?

Is the following behavior expected?
The index file was generated using a 3MB text file.
As the number of characters in between two words increases the time is increasing exponentially!
I thought RE2 scales linearly.

#ls -lh index.3M 
-rw-r--r-- 1 root root 952K Feb 17 00:57 index.3M

#time ./femto_search index.3M  "world.{1,10}hello"
real    0m0.138s
user    0m0.135s
sys     0m0.001s

#time ./femto_search index.3M  "world.{1,20}hello"
real    0m0.639s
user    0m0.634s
sys     0m0.001s

#time ./femto_search index.3M  "world.{1,40}hello"
real    0m4.319s
user    0m4.300s
sys     0m0.001s

#time ./femto_search index.3M  "world.{1,80}hello"
real    0m35.405s
user    0m35.239s
sys     0m0.015s

#time ./femto_search index.3M  "world.{1,160}hello"
real    5m53.839s
user    5m51.885s
sys     0m0.102s

Indexing a file: What is the worst case memory consumption?

I was trying to index a 5GB file and looks like femto is using all the free memory( about 57GB! )
Amount of memory required is not the problem.
Knowing memory requirements before hand( depending on input file size? ) will help.
Does changing block_size & chunk_size improve speed?

search for "note" returns a syntax error?

# femto_search --matches index "note" -v
syntax error
Could not parse pattern note

Looks like a bug.
May be its getting confused for "not" ?

Does this support multiline search?

before trying to install, I just wanted to know if this supports multiline search?
I know that RE2 supports this.
Ex:

"hello



world"

regex pattern "hello.{1,10}world"
Are there any restrictions due to suffix layer on top of this?

Pthread issue during make check

Hello there,
I have found this issue while compiling on Ubuntu x86_64 Precise.
I am running the make release script: make_release.sh
Everything compiles fine but during make check, there are issues finding pthread functions:

epokh@epokh-work:~/Projects/femto_release/femto/dist-build/femto-1.2.9$ make check > error.txt
obj/dbg/libre2.a(dfa.o): In function `Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:109: undefined reference to `pthread_rwlock_init'
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:109: undefined reference to `pthread_rwlock_init'
obj/dbg/libre2.a(dfa.o): In function `~Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:110: undefined reference to `pthread_rwlock_destroy'
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:110: undefined reference to `pthread_rwlock_destroy'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::ReaderLock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:114: undefined reference to `pthread_rwlock_rdlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::ReaderUnlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:115: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::ReaderUnlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:115: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(dfa.o): In function `~Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:110: undefined reference to `pthread_rwlock_destroy'
obj/dbg/libre2.a(dfa.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(prog.o): In function `Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:109: undefined reference to `pthread_rwlock_init'
obj/dbg/libre2.a(prog.o): In function `~Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:110: undefined reference to `pthread_rwlock_destroy'
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:110: undefined reference to `pthread_rwlock_destroy'
obj/dbg/util/thread.o: In function `Thread::Join()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/util/thread.cc:37: undefined reference to `pthread_join'
obj/dbg/util/thread.o: In function `Thread::Start()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/util/thread.cc:27: undefined reference to `pthread_create'
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/util/thread.cc:30: undefined reference to `pthread_detach'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(re2.o): In function `Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:109: undefined reference to `pthread_rwlock_init'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Lock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:111: undefined reference to `pthread_rwlock_wrlock'
obj/dbg/libre2.a(re2.o): In function `re2::Mutex::Unlock()':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:112: undefined reference to `pthread_rwlock_unlock'
obj/dbg/libre2.a(re2.o): In function `~Mutex':
/home/epokh/Projects/femto_release/femto/dist-build/femto-1.2.9/src/re2/re2/./util/mutex.h:110: undefined reference to `pthread_rwlock_destroy'
collect2: ld returned 1 exit status
make[4]: *** [obj/dbg/test/charclass_test] Error 1
make[3]: *** [check-local] Error 2
make[2]: *** [check-am] Error 2
make[1]: *** [check-recursive] Error 1
make: *** [check-recursive] Error 1

Any idea?

Yacc error

I'm running into following error while doing make install:

I have installed berkley yacc : byacc.x86_64 : Berkeley Yacc, a parser generator

Making install in main
make[2]: Entering directory `/home/manojm321/femto/src/main'
/bin/sh ../../ylwrap posix.bison.y y.tab.c posix.bison.c y.tab.h `echo posix.bison.c | sed -e s/cc$/hh/ -e s/cpp$/hpp/ -e s/cxx$/hxx/ -e s/c++$/h++/ -e s/c$/h/` y.output posix.bison.output -- yacc -d 
yacc: e - line 32 of "/home/manojm321/femto/src/main/posix.bison.y", syntax error
%destructor { string_node_free($$); } T_STRING
^

Apache2 apxs-install issue

Hello,
I have read the comments for the module integration but still if somebody knows how to sort my problem.

epokh@epokh-work:~/femto_release/femto/src/mod_femto$ sudo make apxs-install 
apxs2 -i -a -n femto libmod_femto.la
/usr/share/apache2/build/instdso.sh SH_LIBTOOL='/usr/share/apr-1.0/build/libtool' libmod_femto.la /usr/lib/apache2/modules
/usr/share/apr-1.0/build/libtool --mode=install cp libmod_femto.la /usr/lib/apache2/modules/
libtool: install: `libmod_femto.la' is not a valid libtool archive
libtool: install: Try `libtool --help --mode=install' for more information.
apxs:Error: Command failed with rc=65536
.

I am on ubuntu 12.10 with apache2-prefork-dev.

make fails with "posix.flex.c: No such file or directory" error

I have installed all the required deps . Is there anything missing here?

make[4]: Entering directory '/home/femto/src/main'
/bin/bash ../../ylwrap posix.flex.l .c posix.flex.c -- /bin/bash /home/femto/missing flex  
lex.yyposix.flex.c is unchanged
/bin/bash ../../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../..  -I./../utils   -std=gnu99 -Wall -m64 -D_64_BIT -fopenmp -D_XOPEN_SOURCE=600 -O4 -fomit-frame-pointer -DEXTRA_CHECKS=0 -march=native -mtune=native -MT posix.flex.lo -MD -MP -MF .deps/posix.flex.Tpo -c -o posix.flex.lo posix.flex.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../.. -I./../utils -std=gnu99 -Wall -m64 -D_64_BIT -fopenmp -D_XOPEN_SOURCE=600 -O4 -fomit-frame-pointer -DEXTRA_CHECKS=0 -march=native -mtune=native -MT posix.flex.lo -MD -MP -MF .deps/posix.flex.Tpo -c posix.flex.c  -fPIC -DPIC -o .libs/posix.flex.o
gcc: error: posix.flex.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.

OpenMPI build

I don't know if anyone experienced issues with building openmpi for this project. If you follow the steps in the readme document:

root@ubuntu:~/source/openmpi-1.4.2# ./autogen.sh
root@ubuntu:~/source/openmpi-1.4.2# ./configure --prefix=/opt/openmpi1.4.2 --enable-mpirun-prefix-by-default --enable-mpi-threads --with-threads

It shows a problem that is most likely caused by the newer libtool:

aclocal-1.15: error: ../../config/mca_m4_config_include.m4:8: file 'opal/mca/backtrace/configure.m4' does not exist

I tested with openmpi (1.8.8) which builds fine. Note, the flag --enable-mpi-threads has been changed to --enable-mpi-thread-multiple

  --enable-mpi-thread-multiple
                          Enable MPI_THREAD_MULTIPLE support (default:
                          disabled)

The commands should be:

./configure --prefix=/opt/openmpi1.8.8 --enable-mpirun-prefix-by-default --enable-mpi-thread-multiple --with-threads
make
make install
export PATH=$PATH:/opt/openmpi1.8.8/bin
export LD_LIBRARY_PATH=/opt/openmpi1.8.8/lib

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble