feature request: full text and fact search using bleve about ipfs-cluster

[hsanjuan]

ipfs cluster does not hold data that you can run full text search on. I'm not sure I understand this request... can you ellaborate on what you would like to do?

[joeblew99]

i found a few projects on top of IPFS that do search now.
they are using elastic search, but i will fork and change to use bleve.

Hey @hsanjuan picking this up again :)

I think IPFS does have data you can index and search on. It can index and search anything as long as a mapper is present.

http://www.blevesearch.com/docs/Index-Mapping/

The way it works is really this: https://github.com/RoaringBitmap/roaring

[lanzafame]

@gedw99 Sorry, could you elaborate the use case of full-text search within the context of IPFS Cluster specifically?

Enterprise level search is a combination of free text search and facetted search. Facetted search is like what Amazon looks like where the system automatically created categories you can search by.

So I just think that this sort of search is a great thing for a content system.

There is a nice cluster wrapper of bleve called blast that is a pretty good fit for ipfs cluster.

The link:

https://github.com/mosuka/blast

I am pretty sure the team will need support this usecase. But anyway a way o easily add it as a " Microservice" of sorts or " plugin" might be a suitable option too for the team.

Anyway i am curious to know what others think.

[lanzafame]

@gedw99 we are looking at supporting some very different datasets which will require very different index algorithms, i.e. geographical, genomic, archival. It may be worth looking at specing out some sort of Search interface that will allow for 'plug n play' of different indexing mechanisms, of which bleve would be a candidate.

Well it's good you at least want search. Pilossa I would also look at.great for huge data stores with very fast indexing and retrieval. Bleve interface is very simple. The blast project I referenced is a good basis I think. What serialisation and transports do you want to support ? Do you want to make it work via a bus like NATS. Allows things to buffer and catch-up of course. It's a pretty good way to join up different databases / Microservice.

…

On Thu, 26 Jul 2018, 06:39 Adrian Lanzafame, ***@***.***> wrote: @gedw99 <https://github.com/gedw99> we are looking at supporting some very different datasets which will require very different index algorithms, i.e. geographical, genomic, archival. It may be worth looking at specing out some sort of Search interface that will allow for 'plug n play' of different indexing mechanisms, of which bleve would be a candidate. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwu7oyQbBFoikrBvUY-1KUfxQjuq1ks5uKUfkgaJpZM4McUy6> .

[eocarragain]

I think @vmx is the man to talk to about search on IPFS/IPLD in general (rather than ipfs-cluster specifically)? Agree with @lanzafame that different/configurable indexing strategies will be need for different data sets/use-cases/applications. Presumably more specific cases can be handled higher up the stack too.

[vmx]

If you're interested in search, please also follow/comment on the issue at the IPLD repo: ipld/ipld#43

Hey all Had a look at the repo you suggested. Noise is rust with nodejs API. Don't mean to be a pain but.. nodejs on backend is not right. Ipfs is golang on the backend and core search is a large area and should be in golang. I posted my proposal in the comments section as the readme asked.

…

On Thu, 26 Jul 2018, 13:51 Volker Mische, ***@***.***> wrote: If you're interested in search, please also follow/comment on the issue at the IPLD repo: ipld/ipld#43 <ipld/ipld#43> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwtd3QDU-Zuvpvxwzo6QFRjyfOPB4ks5uKa06gaJpZM4McUy6> .